This makes sense to me (I think) because if I am not mistaken, UTF16 is Unicode, and UTF8 is simple ASCII. The slowdown from 6.7 to 7.0 was precicely the support for Unicode text. Someone will correct me if I am wrong about this. As a hobbyist, I try and stay away from localization issues. But I am interested in the idea that all text incoming should be text decoded and outgoing the inverse. (Did I get that right??)
Bob S > On Sep 3, 2021, at 17:29 , Alex Tweedly via use-livecode > <use-livecode@lists.runrev.com> wrote: > > I went back and re-did the tests, checking on the results. > > The file *is* UTF8, so I need to textDecode() it; if I don't, the result are > simply wrong, and so the times are irrelevant. > > 1. Once it has been textDecoded(), i.e. is in internal format, and I run my > algorithm it gets the correct results, taking 115.1 seconds. > > 2. BUT, if just before the algorithm is run, I do a textEncode(tStr, "UTF8") > , it gets the correct results (identical to the above), but in only 3.3 > seconds. > > The code, in a zip file containing the test stack, SpellCheck Library, and > the 'bible' and "war&peace" sample textfiles, can be downloaded from > > https://www.tweedly.org/Downloads/SpellLib.gz > > if anyone wants to look at it. > > Alex. > > > > On 03/09/2021 13:38, Alex Tweedly via use-livecode wrote: >> >> On 03/09/2021 11:07, David V Glasgow via use-livecode wrote: >> >>> Alex states that put textEncode(tWHoleText, "UTF8") into tWholeText speeds >>> replace up, but David B says LC internal format is UTF16. Doesn’t the 8 vs >>> 16 difference matter? Or matters less than other encodings? >> >> I would regard that timing comparison with much suspicion. I was >> textEncoding() it inappropriately - I had just read it in from a file, so I >> *should* have been textDecoding() it. Therefore it is unclear whether the >> times I was seeing then are meaningful. >> >> Alex. >> >> >> _______________________________________________ >> use-livecode mailing list >> use-livecode@lists.runrev.com >> Please visit this url to subscribe, unsubscribe and manage your subscription >> preferences: >> http://lists.runrev.com/mailman/listinfo/use-livecode > > _______________________________________________ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your subscription > preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode _______________________________________________ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode