Actually, Unicode / ISO 10646 is a 21-bit encoding, with values from 0 to 0x10FFFF.
On Wed, Oct 8, 2014 at 10:13 AM, J Decker <d3c...@gmail.com> wrote: > On Tue, Oct 7, 2014 at 2:20 PM, jose isaias cabrera < > jic...@cinops.xerox.com > > wrote: > > > > > "J Decker" wrote... > > > > > > So, I guess it is technically not allowed to encode 11 bit unicode > >> characters as 16. > >> the greek characters are 0x3XX which is 10 bits... I checked what > >> WideCharToMultiByte was doing and found it was using 11 bit encodings... > >> fixed my encoder to use an appropriate size for what's required, added > 11 > >> bit decoding, and now in and out works for that and some chinese > >> characters > >> which are more than 11 bits. > >> > >> The 'unrecognized token' is 0xE0 ? ... although a thing could be 12 > bits > >> exactly... so is it checking ( char[0] == 0xe0 ) && ( ( char[1] & 0xE0 ) > >> == > >> 0x80 )? > >> > >> as a side note.. using visual studio to mouse over the resulting char * > >> string with 11 bit encodings it shows bad characters, if encoded as > (valid > >> but illegal) 16 bit it browses correctly. > >> > > > > J, > > > > My suggestion is for you to read about ANSI, ASCII, UTF7, UTF8, UTF16 and > > UTF32 and understand the ins and outs of the various encoding. You may > need > > to create your own wrapper to get things to work correctly. > > > > Right; I did, and have, but missed the part 'must be encoded in least bits' > (and I'm not sure it is there, and visual studio sees it as a valid thing > to do; to use a encoding larger than the number of bits required) and > unicode only uses 20 bits max so the extended 5 and 6 byte utf-8 encodings > never get used Have a custom wrapper for systems that are not windows; and > now it's more robust. > > Still think it's something of a bug, but has been worked around so for me > it won't be an issue again. > > > > > > > What happens if you create a text file using notepad and make sure that > > you save it as UTF8 and then read that file with the content that you > want? > > Then write it to SQLite and get it back and write it back to another > file? > > Does that work? > > _______________________________________________ > > sqlite-users mailing list > > sqlite-users@sqlite.org > > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > > > _______________________________________________ > sqlite-users mailing list > sqlite-users@sqlite.org > http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users > -- Christopher Vance _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
