Re: LC Server UT08 Encode Error
Yes. A0 looks like the middle of a sequence of UTF-8 bytes for one character. Hitting that will cause an error. I wonder why the db complained about CA and 59. > On Sep 13, 2019, at 4:10 PM, Rick Harrison via use-livecode > wrote: > > Hi Bob and Dar, > > Thanks for getting back to me on this. > > After going through the LC list archives, > I came across a helpful tool called > "Unicode Checker” that was originally > suggested by Richmond Mathewson. > (Thanks for that by the way.) > > There is a nice little utility inside of it that > allowed me to compare the string from > Pages that I pasted to the the string that > I had cleaned with BBEdit. The only > differences found was a 000A > and a few 00A0 (NO-BREAK SPACEs). > > I suspect it is the control character really > screwing things up. (I don’t know how > it got into the string.) > > I am not using the accept-charset attribute > in the statement. If not specified > it is supposed to use the default which I’m > guessing is UTF-8. I supposed I could try > a test with the attribute specified to see if > it makes any difference. > > Thanks, > > Rick > > >> On Sep 13, 2019, at 5:36 PM, Dar Scott Consulting via use-livecode >> wrote: >> >> When you Zap Gremlins, are you removing non-ASCII or converting to ASCII? >> Maybe you can do the same thing before saving to the db. >> >> Codes 0xca 0x59 do form an invalid UTF-8 sequence. They are good letters in >> both Latin-1 and CP1252, so I'm not getting a hint. >> >> Wild, wild guess... Are you using an accept-charset attribute in ? >> Maybe fiddling with that will help. >> > > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your subscription > preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode > ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: LC Server UT08 Encode Error
Hi Bob and Dar, Thanks for getting back to me on this. After going through the LC list archives, I came across a helpful tool called "Unicode Checker” that was originally suggested by Richmond Mathewson. (Thanks for that by the way.) There is a nice little utility inside of it that allowed me to compare the string from Pages that I pasted to the the string that I had cleaned with BBEdit. The only differences found was a 000A and a few 00A0 (NO-BREAK SPACEs). I suspect it is the control character really screwing things up. (I don’t know how it got into the string.) I am not using the accept-charset attribute in the statement. If not specified it is supposed to use the default which I’m guessing is UTF-8. I supposed I could try a test with the attribute specified to see if it makes any difference. Thanks, Rick > On Sep 13, 2019, at 5:36 PM, Dar Scott Consulting via use-livecode > wrote: > > When you Zap Gremlins, are you removing non-ASCII or converting to ASCII? > Maybe you can do the same thing before saving to the db. > > Codes 0xca 0x59 do form an invalid UTF-8 sequence. They are good letters in > both Latin-1 and CP1252, so I'm not getting a hint. > > Wild, wild guess... Are you using an accept-charset attribute in ? > Maybe fiddling with that will help. > ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Stop Integer Coercion to Scientific Notation in JSON
On Thu, Sep 12, 2019 at 1:42 PM Sannyasin Brahmanathaswami via use-livecode
wrote:
> Been working Panos off list.
>
> There is a caveat, jsonImport and jsonExport not only "display" but
> "preserve" the scientific notation as a literal string and math will fail.
>
> Panos says: must use, (and include in a standalone) mergeJson to
> use: jsonToArray and arrayToJSON
>
While ArrayToJSON does fix the problem you are experiencing, it does
introduce additional problems. Try the following code:
```
on mouseUp
put 1 is 1 into tValueA["root"]["boolean_1"]
put 1 is 0 into tValueA["root"]["boolean_2"]
put null into tValueA["root"]["null_1"]
put "100" into tValueA["root"]["text"]
put 100+0 into tValueA["root"]["number"]
put ArrayToJSON(tValueA)
end mouseUp
```
The output is as follows:
```
{
"root": {
"boolean_2": false,
"text": 100,
"null_1": "",
"boolean_1": true,
"number": 100
}
}
```
ArrayToJSON won't ever quote numbers, even if the variable in LiveCode has
the number stored as a string (and not an integer or a real). Imagine a
user enters the string "100" into a text field and then your app sends that
value to a JSON API. If you send an integer to a JSON API that is expecting
a string you will most likely get an error.
The output would more appropriately be rendered this way:
```
{
"root": {
"boolean_2": false,
"text": "100",
"null_1": null,
"boolean_1": true,
"number": 100
}
}
```
Here is a variation on ArrayToJSON that renders the more appropriate output:
```
function ArrayToJSON pArray,pForceRootType,pPretty
local tKey, e
repeat for each key tKey in pArray
if pArray[tKey] is an array then
put "}"(pArray[tKey]) into pArray[tKey]
else if pArray[tKey] is NULL then
put "null" into pArray[tKey]
else if pArray[tKey] is strictly a boolean \
or pArray[tKey] is strictly an integer \
or pArray[tKey] is strictly a real then
put pArray[tKey] into pArray[tKey]
else
# treat as a string
put "}}"[tKey] into pArray[tKey]
end if
end repeat
return(mergJSONEncode("pArray",pForceRootType,pPretty))
end ArrayToJSON
```
In my projects I use this modified version of ArrayToJSON to export JSON
and a wrapper around JsonImport to import JSON. In my experience that gets
me the closest to expected values. Unfortunately JsonImport is rather slow,
but it is more accurate. I don't recall all of the nuances as to why though.
Here is my wrapper around JsonImmport:
```
function _importJSON pJSON
# zero width joiner breaks JSON Importer
# http://quality.livecode.com/show_bug.cgi?id=19691
replace numtoCodePoint("0x200D") with empty in pJSON
return JsonImport(pJSON)
end _importJSON
```
--
Trevor DeVore
ScreenSteps
www.screensteps.com
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Re: LC Server UT08 Encode Error
When you Zap Gremlins, are you removing non-ASCII or converting to ASCII? Maybe you can do the same thing before saving to the db. Codes 0xca 0x59 do form an invalid UTF-8 sequence. They are good letters in both Latin-1 and CP1252, so I'm not getting a hint. Wild, wild guess... Are you using an accept-charset attribute in ? Maybe fiddling with that will help. > On Sep 12, 2019, at 7:21 PM, Rick Harrison via use-livecode > wrote: > > I am having a problem with LC Server > posting data to my Postgresql database. > (LC Server version 9.0.4) > > If I type into a form field such as for > a field named “description”, and submit > the field, it goes into the database without > any problems whatsoever. > > If I compose a description in an application > such as Apple’s Pages or in Text Edit, and > then copy and paste the text into the > description field. It throws an error. > > ERROR: invalid byte sequence for > encoding "UTF8": 0xca 0x59 (0) > > Of course due to this error the > record never gets written out to > the database. > > I know this error is due to gremlins > in the text because if I zap gremlins > with BBEdit and then copy and > paste into the field, everything > works fine! > > I’m sure that other users will > probably use a copy and paste > method at some point, so I need > to scrub the data somehow to > prevent the error from occurring. > > Or is this an encoding/decoding > problem requiring some other > solution? > > Suggestions? > > Rick > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your subscription > preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode > ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: LC Server UT08 Encode Error
Based on the source of the data, I'm going to say it's an encoding issue. I cannot imagine that pages is embedding non-printing characters that get past the clipboard. Bob S > On Sep 12, 2019, at 18:21 , Rick Harrison via use-livecode > wrote: > > I am having a problem with LC Server > posting data to my Postgresql database. > (LC Server version 9.0.4) > > If I type into a form field such as for > a field named “description”, and submit > the field, it goes into the database without > any problems whatsoever. > > If I compose a description in an application > such as Apple’s Pages or in Text Edit, and > then copy and paste the text into the > description field. It throws an error. > > ERROR: invalid byte sequence for > encoding "UTF8": 0xca 0x59 (0) > > Of course due to this error the > record never gets written out to > the database. > > I know this error is due to gremlins > in the text because if I zap gremlins > with BBEdit and then copy and > paste into the field, everything > works fine! > > I’m sure that other users will > probably use a copy and paste > method at some point, so I need > to scrub the data somehow to > prevent the error from occurring. > > Or is this an encoding/decoding > problem requiring some other > solution? > > Suggestions? > > Rick > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your subscription > preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: BasicGeoLib_v100
Danke Hermann. I am still planning to use something like this. I have database with all dutch postcode with housenr's and longitudes and latitudes. This is now public information ordered some years ago by a Dutch Federal Judge as the post delivery appealed against this. But it is not easy to find the download, they hide it a bit. Anyway i downloaded it as almost 1GB of csv data which turned into almost 8GB of database data. Took days to convert and upload it. It's around 6000 tables with also each around 6000 to 13000 entries. Right now just used to find a place via postcode (LeitZahl) entry. And there some php examples on the web to calculate what you now have created, these calculations can be tricky. (i know there are also online services for this with json data as reply, some for free until so many calls, but it's nice to be able to do it yourself) Will check it out in time, and hopefully to be used when i expand my application. Thanks a lot! Jerry Op 13-9-2019 om 07:23 schreef William Prothero via use-livecode: Nice contribution. Thanks! Bill On Sep 12, 2019, at 1:16 PM, hh via use-livecode wrote: Some specialists may be interested in the following, please report if you have any problem with the stack. BasicGeoLib is a collection of very basic functions for using geoPoints. ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
