Re: [sqlite] Conversion failure

[Igor Korot]

On Sat, Jun 25, 2016 at 11:26 PM, Simon Slavin  wrote:
>
> On 26 Jun 2016, at 2:52am, Igor Korot  wrote:
>
>> Now we need to see the actual German version test result with old and
>> new version of the shell tool.
>
> No need.  Given your testing and the bugfix Keith pointed to I can predict it 
> will work fine for the new version.  The problem was that the old version of 
> the shell tool running on Windows could create bad database files.  The fix 
> was in the shell tool and not the API.  And versions released after 4/4/2016 
> have this fix.
>
> Worth noting that other programs which use the API can have the same bug that 
> the shell tool had.  They will have this if they pass non-unicode strings to 
> API calls.  And this is particularly easy in operating systems and 
> programming languages which don't use Unicode as their standard fpr strings.  
> So we all need to know this.

Thank you.

>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Simon Slavin]

On 26 Jun 2016, at 2:52am, Igor Korot  wrote:

> Now we need to see the actual German version test result with old and
> new version of the shell tool.

No need.  Given your testing and the bugfix Keith pointed to I can predict it 
will work fine for the new version.  The problem was that the old version of 
the shell tool running on Windows could create bad database files.  The fix was 
in the shell tool and not the API.  And versions released after 4/4/2016 have 
this fix.

Worth noting that other programs which use the API can have the same bug that 
the shell tool had.  They will have this if they pass non-unicode strings to 
API calls.  And this is particularly easy in operating systems and programming 
languages which don't use Unicode as their standard fpr strings.  So we all 
need to know this.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Keith Medcalf]

> Good news - it looks like the API was fixed as well. I don't have a
> crash anymore.
> However I have a bad news - the API was fixed in the backward-incompatible
> way.
> Meaning that running the new DLL against the old database will still
> crash. But I guess that is kind of expected.
 
> So with the 3.13 version of the SQLite everything works as expected
> with this test.
 
> Now we need to see the actual German version test result with old and
> new version of the shell tool.

https://www.sqlite.org/src/timeline?r=altShellFix






___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Igor Korot]

Simon,

On Sat, Jun 25, 2016 at 9:11 PM, Igor Korot  wrote:
> Simon,
>
> On Sat, Jun 25, 2016 at 8:32 PM, Simon Slavin  wrote:
>>
>> On 26 Jun 2016, at 1:29am, Igor Korot  wrote:
>>
>>> Now it would be interesting to know what is the result on German
>>> version of Windows with German
>>> keyboard.
>>
>> Agreed.
>
> Anyone from Germany around who has German version of Windows with
> German keyboard?
>
>>
>>> Also, for my purposes I should be using that version of the SQLite,
>>> right? And everything will just work.
>>
>> That version or later.
>
> OK, I will download it and try it.
>
>>
>> But what I want to know is what was fixed.  The change may have been inside 
>> the API or merely in the shell tool itself.  If the change was in the shell 
>> tool then I think the API from your version is fine.  And presumably you 
>> were using the shell tool only for testing and what you really care about is 
>> the API.
>
> Yes, shell was used for testing purposes.
> Let me get the code and re-compile. Then we will know what was fixed.

Good news - it looks like the API was fixed as well. I don't have a
crash anymore.
However I have a bad news - the API was fixed in the backward-incompatible way.
Meaning that running the new DLL against the old database will still
crash. But I guess that is kind of expected.

So with the 3.13 version of the SQLite everything works as expected
with this test.

Now we need to see the actual German version test result with old and
new version of the shell tool.

Thank you.

>
> Thank you.
>
>>
>> Simon.
>> ___
>> sqlite-users mailing list
>> sqlite-users@mailinglists.sqlite.org
>> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Igor Korot]

Simon,

On Sat, Jun 25, 2016 at 8:32 PM, Simon Slavin  wrote:
>
> On 26 Jun 2016, at 1:29am, Igor Korot  wrote:
>
>> Now it would be interesting to know what is the result on German
>> version of Windows with German
>> keyboard.
>
> Agreed.

Anyone from Germany around who has German version of Windows with
German keyboard?

>
>> Also, for my purposes I should be using that version of the SQLite,
>> right? And everything will just work.
>
> That version or later.

OK, I will download it and try it.

>
> But what I want to know is what was fixed.  The change may have been inside 
> the API or merely in the shell tool itself.  If the change was in the shell 
> tool then I think the API from your version is fine.  And presumably you were 
> using the shell tool only for testing and what you really care about is the 
> API.

Yes, shell was used for testing purposes.
Let me get the code and re-compile. Then we will know what was fixed.

Thank you.

>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Simon Slavin]

On 26 Jun 2016, at 1:29am, Igor Korot  wrote:

> Now it would be interesting to know what is the result on German
> version of Windows with German
> keyboard.

Agreed.

> Also, for my purposes I should be using that version of the SQLite,
> right? And everything will just work.

That version or later.

But what I want to know is what was fixed.  The change may have been inside the 
API or merely in the shell tool itself.  If the change was in the shell tool 
then I think the API from your version is fine.  And presumably you were using 
the shell tool only for testing and what you really care about is the API.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Igor Korot]

Simon,

On Sat, Jun 25, 2016 at 8:18 PM, Simon Slavin  wrote:
>
> On 26 Jun 2016, at 1:14am, Igor Korot  wrote:
>
>> That would be 313, correct?
>
> Yes.  The idea is to see if the bug still occurs in the current version.  If 
> it doesn't, it has probably been fixed already.

It looks like I have the same results:

SQLite version 3.13.0 2016-05-18 10:57:30
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> CREATE TABLE abcß▀(id integer primary key, α▐ string);
sqlite> SELECT name, hex(name) FROM sqlite_master;
abcß▀|616263C39FE29680
sqlite> PRAGMA encoding;
UTF-8
sqlite>

Now it would be interesting to know what is the result on German
version of Windows with German
keyboard.

Also, for my purposes I should be using that version of the SQLite,
right? And everything will just work.

Thank you.

>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Simon Slavin]

On 26 Jun 2016, at 1:14am, Igor Korot  wrote:

> That would be 313, correct?

Yes.  The idea is to see if the bug still occurs in the current version.  If it 
doesn't, it has probably been fixed already.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Simon Slavin]

For comparison, here is what I get on my Mac:

SQLite version 3.13.0 2016-05-02 15:00:23
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> CREATE TABLE abcß▀(id integer primary key, αΓ string);
sqlite> SELECT name, hex(name) FROM sqlite_master;
abcß▀|616263C39FE29680
sqlite> PRAGMA encoding;
UTF-8

I entered all the command text by pasting in the text from your post.  As you 
can see, the eszett character has been correctly encoded into UTF-8 as '0xC3 
0x9F'.  For completeness' sake I tried a couple of different ways of typing 
eszett and they both yielded the correct result.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Igor Korot]

Simon,

On Sat, Jun 25, 2016 at 8:08 PM, Simon Slavin  wrote:
>
> On 25 Jun 2016, at 5:51am, Igor Korot  wrote:
>
>> This are the results of me trying:
>>
>> SQLite version 3.9.2 2015-11-02 18:31:45
>> Enter ".help" for usage hints.
>> Connected to a transient in-memory database.
>> Use ".open FILENAME" to reopen on a persistent database.
>> sqlite> CREATE TABLE abcß▀(id integer primary key, αΓ string); first
>> value was ALT+225, second - ALT+223.
>> sqlite> SELECT name, hex(name) FROM sqlite_master;
>> abcß▀|616263E1DF
>> sqlite>
>
> The above output indicates a bug.  The correct UTF-8 encoding for eszett is 
> "0xC3 0x9F".  Congratulations for finding a short sequence which clearly 
> shows something that needs to be fixed.
>
> Can you please download an up-to-date version of SQLite shell tool from 
> 'Precompiled binaries' in the download page:
>
> 

That would be 313, correct?

Thank you.

>
> and try exactly the same thing again ?  Can you please add the command and 
> output from
>
> PRAGMA encoding
>
> to your sequence ?
>
> Simon.
>
> PS: Apologies to German speakers.  Apparently 'esset' is an English version 
> of 'eszett'.  I speak very poor German.  I will try to use the correct form 
> in future.
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Simon Slavin]

On 25 Jun 2016, at 5:51am, Igor Korot  wrote:

> This are the results of me trying:
> 
> SQLite version 3.9.2 2015-11-02 18:31:45
> Enter ".help" for usage hints.
> Connected to a transient in-memory database.
> Use ".open FILENAME" to reopen on a persistent database.
> sqlite> CREATE TABLE abcß▀(id integer primary key, αΓ string); first
> value was ALT+225, second - ALT+223.
> sqlite> SELECT name, hex(name) FROM sqlite_master;
> abcß▀|616263E1DF
> sqlite>

The above output indicates a bug.  The correct UTF-8 encoding for eszett is 
"0xC3 0x9F".  Congratulations for finding a short sequence which clearly shows 
something that needs to be fixed.  

Can you please download an up-to-date version of SQLite shell tool from 
'Precompiled binaries' in the download page:



and try exactly the same thing again ?  Can you please add the command and 
output from

PRAGMA encoding

to your sequence ?

Simon.

PS: Apologies to German speakers.  Apparently 'esset' is an English version of 
'eszett'.  I speak very poor German.  I will try to use the correct form in 
future.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Scott Robison]

> This are the results of me trying:
>
> SQLite version 3.9.2 2015-11-02 18:31:45
> Enter ".help" for usage hints.
> Connected to a transient in-memory database.
> Use ".open FILENAME" to reopen on a persistent database.
> sqlite> CREATE TABLE abcß▀(id integer primary key, αΓ string); first
> value was ALT+225, second - ALT+223.
> sqlite> SELECT name, hex(name) FROM sqlite_master;
> abcß▀|616263E1DF
> sqlite>
>
> So now the question is - what encoding is that value, so that it can
> be successfully converted to wstring?

That encoding is raw bytes. Either you need to be using 3.13 as someone
else said, or the sqlite_master table stores names as whatever raw code
page / encoding was used to create them. I would bet money changing that
would be bad (backward incompatible) and that there is no portable way to
do what you are trying, as it would involve knowing what code page was in
use at the time of object creation. This would be even worse if multiple
machines created objects over time.

But I'm driving across country and can't check source to confirm this.

>
> It is not UTF-8 and it is not UTF-16 and it's definitely not ASCII.
>
> Thank you.
>
> >
> > --
> > Scott Robison
> > ___
> > sqlite-users mailing list
> > sqlite-users@mailinglists.sqlite.org
> > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Igor Korot]

Keith,

On Sat, Jun 25, 2016 at 10:14 AM, Keith Medcalf  wrote:
>> > And look in the registry under
>> HKLM\CurrentControlSet\Control\NLS\CodePage
>
>> On my Windows 8.1 machine I don't have HK_LOCAL_MACHINE\CurrentControlSet.
>
> Ooops.  My typo ...
>
> It is HKLM\SYSTEM\CurrentControlSet\NLS\CodePage

...and there is no HKLM\SYSTEM\CurrentControlSet\NLS

Thank you.

>
>> > and report the value for ACP and OEMCP ...
>
>> > and also the value returned by the CHCP command at the cmd prompt.
>
>> C:\Users\Igor\Desktop>chcp
>> Active code page: 437
>
>> Thank you.
>
> The updates to improve the handling of UTF8 console I/O on Windows were only 
> merged to trunk here:
>
> (2016-04-04) https://www.sqlite.org/src/info/4534ebf15fbcd6fe
>
> and the first release with these changes was 3.13.0.
>
>
>
>
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Keith Medcalf]

> > And look in the registry under
> HKLM\CurrentControlSet\Control\NLS\CodePage
 
> On my Windows 8.1 machine I don't have HK_LOCAL_MACHINE\CurrentControlSet.

Ooops.  My typo ...

It is HKLM\SYSTEM\CurrentControlSet\NLS\CodePage
 
> > and report the value for ACP and OEMCP ...

> > and also the value returned by the CHCP command at the cmd prompt.
 
> C:\Users\Igor\Desktop>chcp
> Active code page: 437
 
> Thank you.

The updates to improve the handling of UTF8 console I/O on Windows were only 
merged to trunk here:

(2016-04-04) https://www.sqlite.org/src/info/4534ebf15fbcd6fe

and the first release with these changes was 3.13.0.




___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Igor Korot]

Keith,

On Sat, Jun 25, 2016 at 8:43 AM, Keith Medcalf <kmedc...@dessus.com> wrote:
>
> And look in the registry under HKLM\CurrentControlSet\Control\NLS\CodePage

On my Windows 8.1 machine I don't have HK_LOCAL_MACHINE\CurrentControlSet.

>
> and report the value for ACP and OEMCP ...
>
> and also the value returned by the CHCP command at the cmd prompt.

C:\Users\Igor\Desktop>chcp
Active code page: 437

Thank you.

>
>> -Original Message-
>> From: sqlite-users-boun...@mailinglists.sqlite.org [mailto:sqlite-users-
>> boun...@mailinglists.sqlite.org] On Behalf Of Simon Slavin
>> Sent: Saturday, 25 June, 2016 02:40
>> To: SQLite mailing list
>> Subject: Re: [sqlite] Conversion failure
>>
>>
>> On 25 Jun 2016, at 5:51am, Igor Korot <ikoro...@gmail.com> wrote:
>>
>> > So now the question is - what encoding is that value, so that it can
>> > be successfully converted to wstring?
>>
>> Please execute
>>
>> PRAGMA encoding
>>
>> https://www.sqlite.org/pragma.html#pragma_encoding
>>
>> Simon.
>> ___
>> sqlite-users mailing list
>> sqlite-users@mailinglists.sqlite.org
>> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
>
>
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Igor Korot]

Simon,

On Sat, Jun 25, 2016 at 4:39 AM, Simon Slavin  wrote:
>
> On 25 Jun 2016, at 5:51am, Igor Korot  wrote:
>
>> So now the question is - what encoding is that value, so that it can
>> be successfully converted to wstring?
>
> Please execute
>
> PRAGMA encoding

It gives UTF-8.

Thank you.

>
> https://www.sqlite.org/pragma.html#pragma_encoding
>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Keith Medcalf]

And look in the registry under HKLM\CurrentControlSet\Control\NLS\CodePage

and report the value for ACP and OEMCP ...

and also the value returned by the CHCP command at the cmd prompt.

> -Original Message-
> From: sqlite-users-boun...@mailinglists.sqlite.org [mailto:sqlite-users-
> boun...@mailinglists.sqlite.org] On Behalf Of Simon Slavin
> Sent: Saturday, 25 June, 2016 02:40
> To: SQLite mailing list
> Subject: Re: [sqlite] Conversion failure
> 
> 
> On 25 Jun 2016, at 5:51am, Igor Korot <ikoro...@gmail.com> wrote:
> 
> > So now the question is - what encoding is that value, so that it can
> > be successfully converted to wstring?
> 
> Please execute
> 
> PRAGMA encoding
> 
> https://www.sqlite.org/pragma.html#pragma_encoding
> 
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users



___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Simon Slavin]

On 25 Jun 2016, at 5:51am, Igor Korot  wrote:

> So now the question is - what encoding is that value, so that it can
> be successfully converted to wstring?

Please execute

PRAGMA encoding

https://www.sqlite.org/pragma.html#pragma_encoding

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Igor Korot]

Scott,

On Fri, Jun 24, 2016 at 2:16 PM, Scott Robison  wrote:
> On Fri, Jun 24, 2016 at 12:03 PM, Scott Robison 
> wrote:
>
>> On Windows, when you get a string of characters, you either get an ANSI
>> string using some code page, or you get a wide character string.
>>
>> When you get an ANSI string, it is just a sequence of 8 bit bytes. UTF-8
>> is also a sequence of 8 bit bytes. The meaning / encoding of those 8 bit
>> bytes are very different.
>>
>> SQLite will allow you to write any 8 bit byte sequence you want as a
>> string. It does not attempt to validate the bytes. It will read the bytes
>> back exactly as written. So if you wrote an ANSI string to the database
>> instead of a UTF-8 string, you will get back the ANSI string.
>>
>> This all assumes you're using the UTF-8 functions, which might be more
>> accurately described as byte functions. SQLite databases have an encoding.
>> They store either UTF-8 text or UTF-16 text. If your database is UTF-8 and
>> you use the char/byte based interface, SQLite won't interpret the bytes. If
>> your database is UTF-16 and you use the wide character based interface,
>> SQLite won't interpret the wide characters. It assumes you've given it
>> valid data and will use it as is. This is particularly convenient when
>> dealing with variant columns.
>>
>> If, however, your database is UTF-8 and you use the UTF-16 interface
>> functions, SQLite will attempt to convert the data between UTF-8 & UTF-16.
>> If your database is UTF-16 and you use the UTF-8 interface functions,
>> SQLite will attempt to convert the data. In these cases, it is important to
>> have valid UTF-whatever in the database.
>>
>> It looks to me like, in your case, some program wrote a byte sequence to
>> the database that was not UTF-8. You later read that string back out of the
>> database, and attempt to convert it to a wstring with your C++ code. The
>> byte sequence was not UTF-8, hence the failure.
>>
>> I seem to recall a recent discussion on the list about the shell and
>> console input / output and it not being treated 100% accurately as
>> UTF-whatever. Library internals are, but the IO layer in the shell, not so
>> much.
>>
>> Thus you cannot depend on the shell to translate non-ASCII characters on
>> Windows and write them as UTF-whatever. If using the shell is essential to
>> your process, you can't currently get there from here.
>>
>> Though maybe ... instead of typing ALT+225, try typing ALT+195 ALT+159. In
>> your windows console, that would give you the equivalent byte sequence for
>> that character, compensating for the fact that SQLite doesn't (I believe)
>> transform console input to UTF-8. If I am mistaken on that point, I
>> apologize.
>>
>> If the two alt-code byte sequences create data your C++ code can then
>> process (because it's valid UTF-8), you'll know for certain that the SQLite
>> shell on Windows does not process UTF-8 for console IO, just internally to
>> the database layer.
>>
>
> Okay, rather than guessing, I just did a test from a Windows 10 command
> prompt. I am getting appropriate UTF-8 sequences. Here is my experiment:
>
> I opened a memory database and issued the following commands:
>
> create table test(a text);
> insert into test values('ß'),('▀'),('á'),('ß'); -- for the first value I
> typed ALT+225, then ALT+223, then ALT+0225, then ALT+0223
> select a, hex(a) from test;
>
> Which resulted in four rows of output:
>
> ß|C3A1
> ▀|C39F
> á|C2A0
> ß|C3A1
>
> I'm hoping all these extended characters are handled properly by gmail and
> whatever email program you use.
>
> Windows supports legacy ALT+### codes that map to the legacy code page. It
> also supports ALT+0### which map to Unicode code points. This allows people
> who're accustomed to the ALT+### format to still see the character they
> expect, but translated to the equivalent Unicode code point.
>
> Again, this is with Windows 10. Perhaps you could try a similar sequence to
> what I typed above on your SQLite shell and Windows command prompt version
> and see what you get back.

This are the results of me trying:

SQLite version 3.9.2 2015-11-02 18:31:45
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> CREATE TABLE abcß▀(id integer primary key, αΓ string); first
value was ALT+225, second - ALT+223.
sqlite> SELECT name, hex(name) FROM sqlite_master;
abcß▀|616263E1DF
sqlite>

So now the question is - what encoding is that value, so that it can
be successfully converted to wstring?

It is not UTF-8 and it is not UTF-16 and it's definitely not ASCII.

Thank you.

>
> --
> Scott Robison
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list

Re: [sqlite] Conversion failure

[Scott Robison]

On Fri, Jun 24, 2016 at 2:02 PM, Keith Medcalf  wrote:

>
> > No, I'm using consolas, but am using whatever default codepage settings
> > come with the system (I installed Windows 10 1511 February update on a
> new
> > computer a few months ago). ACP=1252, MACCP=1, OEMCP=437. chcp
> reports
> > my console is using code page 437.
>
> Fascinating!
> This means that changing the default OEMCP in the registry is not the same
> as changing the codepage on the console with chcp.
>
> > In any case: the keyboard input with the SQLite shell on Windows 10
> > default configuration (with consolas as the font) clearly does translate
> code
> > points to UTF-8 sequences.
>
> I wonder if the difference between your ALT+223 and ALT+0223 is because
> the OEMCP is being used to translate the "input" in the former case (but
> not when using ALT+0223, which forces a unicode code point), and since I
> have changed the OEMCP to be the same as the ACP, I get Unicode code points
> even without the leading 0's.
>

That's what I said, or at least intended to say, a couple messages ago.
However, I am sleep deprived and anxiously awaiting a vacation to begin in
a few hours so it may not have been clear. :) In fact, I may just bug out
early. It's largely a ghost town around here at the moment anyway.

-- 
Scott Robison
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Scott Robison]

On Fri, Jun 24, 2016 at 12:54 PM, Keith Medcalf  wrote:

>
> > On Friday, 24 June, 2016 12:17 -0600, Scott Robison said:
>
> > Okay, rather than guessing, I just did a test from a Windows 10 command
> > prompt. I am getting appropriate UTF-8 sequences. Here is my experiment:
> >
> > I opened a memory database and issued the following commands:
> >
> > create table test(a text);
> > insert into test values('ß'),('▀'),('á'),('ß'); -- for the first value I
> > typed ALT+225, then ALT+223, then ALT+0225, then ALT+0223
> > select a, hex(a) from test;
> >
> > Which resulted in four rows of output:
> >
> > ß|C3A1
> > ▀|C39F
> > á|C2A0
> > ß|C3A1
>

{snipped}


> And I get this, on Windows 10 1511, Consolas font, codepage 437 (IBM ANSI
> codepage with drawing characters in upper 127 characters):
>
> Windows supports legacy ALT+### codes that map to the legacy code page. It
> > also supports ALT+0### which map to Unicode code points. This allows
> > people who're accustomed to the ALT+### format to still see the
> character they
> > expect, but translated to the equivalent Unicode code point.
>
> This is claimed, but not quite true.  I don't know what font and codepage
> you have set for the console but they affect both the input and output
> conversions.  Generally, the ACP codepage(s) work correctly, OEM ones do
> not.  You also have to match the font to the codepage or the input
> translation is based on the output translation, resulting in incorrectness.
>
> I suspect you are using an OEM font, and not a unicode font ...
>

No, I'm using consolas, but am using whatever default codepage settings
come with the system (I installed Windows 10 1511 February update on a new
computer a few months ago). ACP=1252, MACCP=1, OEMCP=437. chcp reports
my console is using code page 437.

In any case: the keyboard input with the SQLite shell on Windows 10 default
configuration (with consolas as the font) clearly does translate code
points to UTF-8 sequences.

-- 
Scott Robison
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Scott Robison]

On Fri, Jun 24, 2016 at 12:03 PM, Scott Robison 
wrote:

> On Windows, when you get a string of characters, you either get an ANSI
> string using some code page, or you get a wide character string.
>
> When you get an ANSI string, it is just a sequence of 8 bit bytes. UTF-8
> is also a sequence of 8 bit bytes. The meaning / encoding of those 8 bit
> bytes are very different.
>
> SQLite will allow you to write any 8 bit byte sequence you want as a
> string. It does not attempt to validate the bytes. It will read the bytes
> back exactly as written. So if you wrote an ANSI string to the database
> instead of a UTF-8 string, you will get back the ANSI string.
>
> This all assumes you're using the UTF-8 functions, which might be more
> accurately described as byte functions. SQLite databases have an encoding.
> They store either UTF-8 text or UTF-16 text. If your database is UTF-8 and
> you use the char/byte based interface, SQLite won't interpret the bytes. If
> your database is UTF-16 and you use the wide character based interface,
> SQLite won't interpret the wide characters. It assumes you've given it
> valid data and will use it as is. This is particularly convenient when
> dealing with variant columns.
>
> If, however, your database is UTF-8 and you use the UTF-16 interface
> functions, SQLite will attempt to convert the data between UTF-8 & UTF-16.
> If your database is UTF-16 and you use the UTF-8 interface functions,
> SQLite will attempt to convert the data. In these cases, it is important to
> have valid UTF-whatever in the database.
>
> It looks to me like, in your case, some program wrote a byte sequence to
> the database that was not UTF-8. You later read that string back out of the
> database, and attempt to convert it to a wstring with your C++ code. The
> byte sequence was not UTF-8, hence the failure.
>
> I seem to recall a recent discussion on the list about the shell and
> console input / output and it not being treated 100% accurately as
> UTF-whatever. Library internals are, but the IO layer in the shell, not so
> much.
>
> Thus you cannot depend on the shell to translate non-ASCII characters on
> Windows and write them as UTF-whatever. If using the shell is essential to
> your process, you can't currently get there from here.
>
> Though maybe ... instead of typing ALT+225, try typing ALT+195 ALT+159. In
> your windows console, that would give you the equivalent byte sequence for
> that character, compensating for the fact that SQLite doesn't (I believe)
> transform console input to UTF-8. If I am mistaken on that point, I
> apologize.
>
> If the two alt-code byte sequences create data your C++ code can then
> process (because it's valid UTF-8), you'll know for certain that the SQLite
> shell on Windows does not process UTF-8 for console IO, just internally to
> the database layer.
>

Okay, rather than guessing, I just did a test from a Windows 10 command
prompt. I am getting appropriate UTF-8 sequences. Here is my experiment:

I opened a memory database and issued the following commands:

create table test(a text);
insert into test values('ß'),('▀'),('á'),('ß'); -- for the first value I
typed ALT+225, then ALT+223, then ALT+0225, then ALT+0223
select a, hex(a) from test;

Which resulted in four rows of output:

ß|C3A1
▀|C39F
á|C2A0
ß|C3A1

I'm hoping all these extended characters are handled properly by gmail and
whatever email program you use.

Windows supports legacy ALT+### codes that map to the legacy code page. It
also supports ALT+0### which map to Unicode code points. This allows people
who're accustomed to the ALT+### format to still see the character they
expect, but translated to the equivalent Unicode code point.

Again, this is with Windows 10. Perhaps you could try a similar sequence to
what I typed above on your SQLite shell and Windows command prompt version
and see what you get back.

-- 
Scott Robison
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Keith Medcalf]

And, by the way, do not forget to set your console font to a UNICODE font.  The 
raster fonts are designed for OEM code pages (eg, 437/850) and cannot display 
the unicode character set properly.  You need to use a TrueType font (both 
Consolas and Lucida Console will work -- Consolas being the less hideous of the 
two).  And set the console code page to either the ACP value (1252 for Western 
European) or the UTF8 codepage (65001).  Using an OEM code page will not work 
(437, 850, etc).

> -Original Message-
> From: sqlite-users-boun...@mailinglists.sqlite.org [mailto:sqlite-users-
> boun...@mailinglists.sqlite.org] On Behalf Of Keith Medcalf
> Sent: Friday, 24 June, 2016 11:56
> To: SQLite mailing list
> Subject: Re: [sqlite] Conversion failure
> 
> 
> > I just did ALT+225 in the SQLite shell during the CREATE TABLE command.
> 
> that should be ALT+223, which is ß (UTF8 = C3 9F)
> ALT+225 is a different character á (UTF8 = C3 A1)
> 
> 
> 
> 
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users



___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Scott Robison]

On Fri, Jun 24, 2016 at 11:08 AM, Igor Korot  wrote:

> > To answer your main question: Will a DB in SQLite produce the same
> > characters/encoding in Germany as in the US or China and can data be
> safely
> > sent from one to the other?:
> > The answer is: It should... if the Apps you use convert values correctly
> > between UTF-8 and local encodings (Steps: INPUT-TRANS-OUTPUT above), then
> > the SQLite API will do its part correctly (STORE-RETRIEVE) and you should
> > see the same things everywhere. if this isn't the case, then the Apps you
> > use are broken.
>
> OK, so all in all.
> What I gather from all your replies is that however I enter the data -
> table name, table fields -
> whether it will be with "ALT+num", or directly typing it on the
> keyboard, and independently
> on where the input is produced - US, Germany or China - querying such
> database will work.
> correctly around the world.
>
> Am I understanding this right?
>
> If yes, than going back to my original post:
>
> struct Table
> {
> std::wstring name;
> std::vector fields;
> std::vector foreign_keys;
> };
>
> std::wstring_convert myconv;
> const unsigned char *tableName = sqlite3_column_text( stmt, 0 );
> pimpl->m_tables[m_catalog].push_back( Table( myconv.from_bytes( (const
> char *) tableName ), fields, foreign_keys ) );
>
> This code compiled with MSVC 2010 as C++11 snippet will crash
> on the "ALT+225" symbol inside myconv.from_bytes().
> And if my understanding above is correct - it shouldn't crash.
>
> So what can I do to fix it?
>
> Thank you.
>

On Windows, when you get a string of characters, you either get an ANSI
string using some code page, or you get a wide character string.

When you get an ANSI string, it is just a sequence of 8 bit bytes. UTF-8 is
also a sequence of 8 bit bytes. The meaning / encoding of those 8 bit bytes
are very different.

SQLite will allow you to write any 8 bit byte sequence you want as a
string. It does not attempt to validate the bytes. It will read the bytes
back exactly as written. So if you wrote an ANSI string to the database
instead of a UTF-8 string, you will get back the ANSI string.

This all assumes you're using the UTF-8 functions, which might be more
accurately described as byte functions. SQLite databases have an encoding.
They store either UTF-8 text or UTF-16 text. If your database is UTF-8 and
you use the char/byte based interface, SQLite won't interpret the bytes. If
your database is UTF-16 and you use the wide character based interface,
SQLite won't interpret the wide characters. It assumes you've given it
valid data and will use it as is. This is particularly convenient when
dealing with variant columns.

If, however, your database is UTF-8 and you use the UTF-16 interface
functions, SQLite will attempt to convert the data between UTF-8 & UTF-16.
If your database is UTF-16 and you use the UTF-8 interface functions,
SQLite will attempt to convert the data. In these cases, it is important to
have valid UTF-whatever in the database.

It looks to me like, in your case, some program wrote a byte sequence to
the database that was not UTF-8. You later read that string back out of the
database, and attempt to convert it to a wstring with your C++ code. The
byte sequence was not UTF-8, hence the failure.

I seem to recall a recent discussion on the list about the shell and
console input / output and it not being treated 100% accurately as
UTF-whatever. Library internals are, but the IO layer in the shell, not so
much.

Thus you cannot depend on the shell to translate non-ASCII characters on
Windows and write them as UTF-whatever. If using the shell is essential to
your process, you can't currently get there from here.

Though maybe ... instead of typing ALT+225, try typing ALT+195 ALT+159. In
your windows console, that would give you the equivalent byte sequence for
that character, compensating for the fact that SQLite doesn't (I believe)
transform console input to UTF-8. If I am mistaken on that point, I
apologize.

If the two alt-code byte sequences create data your C++ code can then
process (because it's valid UTF-8), you'll know for certain that the SQLite
shell on Windows does not process UTF-8 for console IO, just internally to
the database layer.

-- 
Scott Robison
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Keith Medcalf]

> I just did ALT+225 in the SQLite shell during the CREATE TABLE command.

that should be ALT+223, which is ß (UTF8 = C3 9F)
ALT+225 is a different character á (UTF8 = C3 A1)




___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[R Smith]

On 2016/06/24 7:08 PM, Igor Korot wrote:


OK, so all in all.
What I gather from all your replies is that however I enter the data -
table name, table fields -
whether it will be with "ALT+num", or directly typing it on the
keyboard, and independently
on where the input is produced - US, Germany or China - querying such
database will work.
correctly around the world.

Am I understanding this right?

If yes, than going back to my original post:

struct Table
{
 std::wstring name;
 std::vector fields;
 std::vector foreign_keys;
};

std::wstring_convert myconv;
const unsigned char *tableName = sqlite3_column_text( stmt, 0 );
pimpl->m_tables[m_catalog].push_back( Table( myconv.from_bytes( (const
char *) tableName ), fields, foreign_keys ) );

This code compiled with MSVC 2010 as C++11 snippet will crash
on the "ALT+225" symbol inside myconv.from_bytes().
And if my understanding above is correct - it shouldn't crash.

So what can I do to fix it?

Thank you.


It's hard to tell - That structure is not valid SQL, it must go through 
some connector in MSVC (which I am not familiar with) and in that thing 
gets translated. That is where it will break too when you add esset or 
other weird characters, but that is not SQLite that can help you, 
perhaps there is a MSVC forum with people who know why the translation 
doesn't work?  If the value gets to SQLite correctly, there will be no 
problems and it will be the same data anyone else in the rest of the 
World will see.



___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Igor Korot]

Hi,

On Fri, Jun 24, 2016 at 12:51 PM, R Smith  wrote:
>
>
> On 2016/06/24 6:11 PM, Simon Slavin wrote:
>>
>> The ALT+num system for entering unusual characters is a Windows thing.  On
>> a Mac you do it by picking the character from a virtual keyboard shown on
>> the display, or by holding a key down for a second to see variations on it.
>> For instance, if I hold down the 's' key I get three variations on 's', one
>> of which is esset.  Whichever way I do it, I get the same Unicode character,
>> which I can prove using hexdump.  The computer can't tell how the character
>> was entered.
>>
>> Sorry, but whatever your problem is is going to have to be investigated by
>> a Windows user.  Worse still, it might be different for different versions
>> of Windows.
>
>
> I'm not sure it will be different between Windows versions (talking
> different flavours of the same generation at least), but it may definitely
> be different for different pieces of software on the very same computer
> even.
>
> Igor, to try and explain very simply what needs to happen for one of your
> scenarios, I will try a box diagram (I hope this prints correctly):
>
> +-- INPUT --+ +--- TRANS ---++-- STORE --+
> | Get Input from| | Convert whatever | | |
> | some console in   | | we got from the | | Push the UTF-8 |
> | whatever encoding | --> | input into UTF-8 | --> | translated value |
> | or codepage etc.  | | specifically | | into an SQLite |
> |(like std:wstring) | || | DB |
> \---/\-/\---/
>
> Now Send the DB to your friend in Germany or China.
>
> + RETRIEVE -++--- TRANS ---+ +-- OUTPUT -+
> | | | Convert the UTF-8 | | |
> | Read the UTF-8 | | into whatever | |  Display to user |
> | from the DB | --> | encoding/code-page | --> |  in console or |
> | | | the target machine | |  GUI in correct |
> | | | or user likes. | |  format. |
> \---/\-/\---/
>
>
> SQLite API is only ever involved in the STORE and RETRIEVE steps. In those
> steps, it expects UTF-8 and gives back whatever it was given, assuming it to
> be UTF-8.
> Everything else is up to the App / CLi or whatever other program is tasked
> with getting information from and to users.
>
> The problem you described about the command line interface seems that you
> might get the wrong information back because you entered esset into the
> console. Problem is, the console in windows using Alt-225 is not UTF-8 (as
> other posts explained), it stores a byte-sequence that is not UTF-8, but
> that shouldn't matter, because on return the same byte sequence should
> display the same in the same console. if it doesn't, then we have a bug (as
> Simon mentioned), but the bug is in the console app, not in SQLite.
> SQlite will definitely STORE and RETRIEVE stuff the same way.
>
> Whether the SQLite Cli works as expected in Windows I do not know yet and
> cannot test right now, but will do when I get home and someone else did not
> do it already.
>
> To answer your main question: Will a DB in SQLite produce the same
> characters/encoding in Germany as in the US or China and can data be safely
> sent from one to the other?:
> The answer is: It should... if the Apps you use convert values correctly
> between UTF-8 and local encodings (Steps: INPUT-TRANS-OUTPUT above), then
> the SQLite API will do its part correctly (STORE-RETRIEVE) and you should
> see the same things everywhere. if this isn't the case, then the Apps you
> use are broken.

OK, so all in all.
What I gather from all your replies is that however I enter the data -
table name, table fields -
whether it will be with "ALT+num", or directly typing it on the
keyboard, and independently
on where the input is produced - US, Germany or China - querying such
database will work.
correctly around the world.

Am I understanding this right?

If yes, than going back to my original post:

struct Table
{
std::wstring name;
std::vector fields;
std::vector foreign_keys;
};

std::wstring_convert myconv;
const unsigned char *tableName = sqlite3_column_text( stmt, 0 );
pimpl->m_tables[m_catalog].push_back( Table( myconv.from_bytes( (const
char *) tableName ), fields, foreign_keys ) );

This code compiled with MSVC 2010 as C++11 snippet will crash
on the "ALT+225" symbol inside myconv.from_bytes().
And if my understanding above is correct - it shouldn't crash.

So what can I do to fix it?

Thank you.

>
> I hope this makes sense :)
> Ryan
>
>
>
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org

Re: [sqlite] Conversion failure

[R Smith]

Well, that broke spectacularly.
Please bear with me for one more try...

On 2016/06/24 6:51 PM, R Smith wrote:


Igor, to try and explain very simply what needs to happen for one of 
your scenarios, I will try a box diagram (I hope this prints correctly):


+-- INPUT --+ +--- TRANS ---++-- STORE
--+
| Get Input from| | Convert whatever|
|   |
| some console in   | | we got from the | | Push the
UTF-8|
| whatever encoding | --> | input into UTF-8| --> | translated
value  |
| or codepage etc.  | | specifically| | into an
SQLite|
|(like std:wstring) | | | |
DB|
\---/ \-/
\---/

Now Send the DB to your friend in Germany or China.

+ RETRIEVE -+ +--- TRANS ---+ +-- OUTPUT
-+
|   | | Convert the UTF-8   |
|   |
| Read the UTF-8| | into whatever   | | Display to
user  |
| from the DB   | --> | encoding/code-page  | --> |  in
console or|
|   | | the target machine  | |  GUI in
correct   |
|   | | or user likes.  | | format.  
   |

\---/ \-/
\---/



SQLite API is only ever involved in the STORE and RETRIEVE steps. In 
those steps, it expects UTF-8 and gives back whatever it was given, 
assuming it to be UTF-8.
Everything else is up to the App / CLi or whatever other program is 
tasked with getting information from and to users.


The problem you described about the command line interface seems that 
you might get the wrong information back because you entered esset 
into the console. Problem is, the console in windows using Alt-225 is 
not UTF-8 (as other posts explained), it stores a byte-sequence that 
is not UTF-8, but that shouldn't matter, because on return the same 
byte sequence should display the same in the same console. if it 
doesn't, then we have a bug (as Simon mentioned), but the bug is in 
the console app, not in SQLite.

SQlite will definitely STORE and RETRIEVE stuff the same way.

Whether the SQLite Cli works as expected in Windows I do not know yet 
and cannot test right now, but will do when I get home and someone 
else did not do it already.


To answer your main question: Will a DB in SQLite produce the same 
characters/encoding in Germany as in the US or China and can data be 
safely sent from one to the other?:
The answer is: It should... if the Apps you use convert values 
correctly between UTF-8 and local encodings (Steps: INPUT-TRANS-OUTPUT 
above), then the SQLite API will do its part correctly 
(STORE-RETRIEVE) and you should see the same things everywhere. if 
this isn't the case, then the Apps you use are broken.


I hope this makes sense :)
Ryan


___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[R Smith]

On 2016/06/24 6:11 PM, Simon Slavin wrote:

The ALT+num system for entering unusual characters is a Windows thing.  On a 
Mac you do it by picking the character from a virtual keyboard shown on the 
display, or by holding a key down for a second to see variations on it.  For 
instance, if I hold down the 's' key I get three variations on 's', one of 
which is esset.  Whichever way I do it, I get the same Unicode character, which 
I can prove using hexdump.  The computer can't tell how the character was 
entered.

Sorry, but whatever your problem is is going to have to be investigated by a 
Windows user.  Worse still, it might be different for different versions of 
Windows.


I'm not sure it will be different between Windows versions (talking 
different flavours of the same generation at least), but it may 
definitely be different for different pieces of software on the very 
same computer even.


Igor, to try and explain very simply what needs to happen for one of 
your scenarios, I will try a box diagram (I hope this prints correctly):


+-- INPUT --+ +--- TRANS ---++-- STORE --+
| Get Input from| | Convert whatever | | |
| some console in   | | we got from the | | Push the UTF-8 |
| whatever encoding | --> | input into UTF-8 | --> | translated value |
| or codepage etc.  | | specifically | | into an SQLite |
|(like std:wstring) | || | DB |
\---/\-/\---/

Now Send the DB to your friend in Germany or China.

+ RETRIEVE -++--- TRANS ---+ +-- OUTPUT -+
| | | Convert the UTF-8 | | |
| Read the UTF-8 | | into whatever | |  Display to user |
| from the DB | --> | encoding/code-page | --> |  in console or |
| | | the target machine | |  GUI in correct |
| | | or user likes. | |  format. |
\---/\-/\---/


SQLite API is only ever involved in the STORE and RETRIEVE steps. In 
those steps, it expects UTF-8 and gives back whatever it was given, 
assuming it to be UTF-8.
Everything else is up to the App / CLi or whatever other program is 
tasked with getting information from and to users.


The problem you described about the command line interface seems that 
you might get the wrong information back because you entered esset into 
the console. Problem is, the console in windows using Alt-225 is not 
UTF-8 (as other posts explained), it stores a byte-sequence that is not 
UTF-8, but that shouldn't matter, because on return the same byte 
sequence should display the same in the same console. if it doesn't, 
then we have a bug (as Simon mentioned), but the bug is in the console 
app, not in SQLite.

SQlite will definitely STORE and RETRIEVE stuff the same way.

Whether the SQLite Cli works as expected in Windows I do not know yet 
and cannot test right now, but will do when I get home and someone else 
did not do it already.


To answer your main question: Will a DB in SQLite produce the same 
characters/encoding in Germany as in the US or China and can data be 
safely sent from one to the other?:
The answer is: It should... if the Apps you use convert values correctly 
between UTF-8 and local encodings (Steps: INPUT-TRANS-OUTPUT above), 
then the SQLite API will do its part correctly (STORE-RETRIEVE) and you 
should see the same things everywhere. if this isn't the case, then the 
Apps you use are broken.


I hope this makes sense :)
Ryan


___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Simon Slavin]

On 24 Jun 2016, at 4:56pm, Igor Korot  wrote:

> Do you have an international version of OSX or an English one?

There is no such thing.  All copies of OS X work with all languages.  You tell 
it which languages you can read and any programs which support those languages 
start displaying in them.  You pick whatever input language you want and the 
computer will act as if you're typing on that keyboard.  Unfortunately the keys 
on your physical keyboard don't magically change to show the correct characters.

> Can you check if there is a difference between using  method

The ALT+num system for entering unusual characters is a Windows thing.  On a 
Mac you do it by picking the character from a virtual keyboard shown on the 
display, or by holding a key down for a second to see variations on it.  For 
instance, if I hold down the 's' key I get three variations on 's', one of 
which is esset.  Whichever way I do it, I get the same Unicode character, which 
I can prove using hexdump.  The computer can't tell how the character was 
entered.

Sorry, but whatever your problem is is going to have to be investigated by a 
Windows user.  Worse still, it might be different for different versions of 
Windows.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Igor Korot]

 Simon,

On Fri, Jun 24, 2016 at 11:44 AM, Simon Slavin  wrote:
>
> On 24 Jun 2016, at 4:37pm, Igor Korot  wrote:
>
>> Now are all those scenarios correct?
>> Will me and my German friend be able to open each other db and work with 
>> them?
>
> Yes.  Unless there's a bug somewhere.  Programs which call the SQLite API 
> should not be doing anything different just because they have a different 
> code page selected.  Nor should they interpret a character different 
> depending on whether it was typed on a keyboard or entered using the ALT-key 
> method.  Nor should a character be interpreted differently because you're 
> running a German version of the OS instead of a USA version of the OS.
>
> Esset is esset.  The character that looks like that should not have different 
> values.

Do you have an international version of OSX or an English one?
Can you check if there is a difference between using  method
and just entering the symbol
from the keyboard method from the SQLite byte sequence POV?

As you can see from the log session I posted the SELECT SQLite_master
correctly displays the table name.
Which probably means that if I right now install the German keyboard
and try to type that character
I will get the same byte sequence.

Or not and I'm completely off here?

Thank you.

>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Simon Slavin]

On 24 Jun 2016, at 4:37pm, Igor Korot  wrote:

> Now are all those scenarios correct?
> Will me and my German friend be able to open each other db and work with them?

Yes.  Unless there's a bug somewhere.  Programs which call the SQLite API 
should not be doing anything different just because they have a different code 
page selected.  Nor should they interpret a character different depending on 
whether it was typed on a keyboard or entered using the ALT-key method.  Nor 
should a character be interpreted differently because you're running a German 
version of the OS instead of a USA version of the OS.

Esset is esset.  The character that looks like that should not have different 
values.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Igor Korot]

On Fri, Jun 24, 2016 at 11:36 AM, Simon Slavin  wrote:
>
> On 24 Jun 2016, at 4:34pm, Igor Korot  wrote:
>
>> What do you mean?
>> I'm talking here about the SQLite shell tool downloaded from the
>> official web site (the executable).
>> I'm NOT talking about self-compiled tool and neither modified self-compiled 
>> one.
>
> I'm saying that there may be a bug in this tool.  After all, it's just 
> another program which calls the SQLite API.  It has had other bugs fixed over 
> the years.

OK, got it.
No, I'm talking about the situation when there is no bugs in the conversion.

So those scenarios are correct, right?

Now, the question is - will the code I posted originally work
correctly in all of them?

Thank you.

>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Igor Korot]

Simon,

On Fri, Jun 24, 2016 at 11:34 AM, Igor Korot  wrote:
> Simon,
>
> On Fri, Jun 24, 2016 at 11:13 AM, Simon Slavin  wrote:
>>
>> On 24 Jun 2016, at 3:55pm, Igor Korot  wrote:
>>
>>> Are those 3 scenarios correct?
>>
>> They are if the shell tool (or any other SQLite software) works the way it 
>> should do.  If you have found a situation where the selected code page 
>> changes what makes it into a SQLite database then there is a bug somewhere.  
>> The bug is probably in the program which calls the SQLite API, not in the 
>> API itself.
>
> What do you mean?
> I'm talking here about the SQLite shell tool downloaded from the
> official web site (the executable).
> I'm NOT talking about self-compiled tool and neither modified self-compiled 
> one.
>
>>
>> I cannot test the SQLite shell tool under Windows myself: I don't have a 
>> Windows computer.
>>
>>> If the answer is yes, will the code I posted in the original post work
>>> for all 3 of them?
>>> Or maybe just for the first one?
>>> Because if I'm in Germany, have the German version of Windows and the
>>> German keyboard,
>>> I don't need to type  to get that symbol. I will just type it
>>> on the keyboard.
>>
>> The program you're using should translate any strings into Unicode before it 
>> calls the SQLite API.  If it is passing strings encoded any other way (e.g. 
>> ANSI, ASCII, or a local code page) then it has a bug.
>
> Well if you look at the code I provided, I'm retrieving the table name
> from the database and then
> convert it to the std::wstring for processing by my software.
> This is what failed and so this is what I'm concern at the moment.
>
>>
>> Don't forget that you're meant to be able to send me your database.  I have 
>> a Mac.  Macs don't use code pages at all.  But I should be able to type your 
>> German or Chinese table name on my keyboard and get the right table.  And if 
>> you think Mac users don't matter, consider all the people using iPhones all 
>> over the world.  They don't use code pages either.  All Apple equipment uses 
>> Unicode.
>
> Well porting to Mac will open a completely different set of worms as
> Apple is huge beast itself. ;-)
> For now though I'm looking for the simplest case.

Sorry meant to add this but hit "Send".
It should also work between 2 Macs not just Windows <-> Mac.

Now are all those scenarios correct?
Will me and my German friend be able to open each other db and work with them?

Thank you.

>
> Thank you.
>
>>
>> Simon.
>> ___
>> sqlite-users mailing list
>> sqlite-users@mailinglists.sqlite.org
>> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Simon Slavin]

On 24 Jun 2016, at 4:34pm, Igor Korot  wrote:

> What do you mean?
> I'm talking here about the SQLite shell tool downloaded from the
> official web site (the executable).
> I'm NOT talking about self-compiled tool and neither modified self-compiled 
> one.

I'm saying that there may be a bug in this tool.  After all, it's just another 
program which calls the SQLite API.  It has had other bugs fixed over the years.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Igor Korot]

Simon,

On Fri, Jun 24, 2016 at 11:13 AM, Simon Slavin  wrote:
>
> On 24 Jun 2016, at 3:55pm, Igor Korot  wrote:
>
>> Are those 3 scenarios correct?
>
> They are if the shell tool (or any other SQLite software) works the way it 
> should do.  If you have found a situation where the selected code page 
> changes what makes it into a SQLite database then there is a bug somewhere.  
> The bug is probably in the program which calls the SQLite API, not in the API 
> itself.

What do you mean?
I'm talking here about the SQLite shell tool downloaded from the
official web site (the executable).
I'm NOT talking about self-compiled tool and neither modified self-compiled one.

>
> I cannot test the SQLite shell tool under Windows myself: I don't have a 
> Windows computer.
>
>> If the answer is yes, will the code I posted in the original post work
>> for all 3 of them?
>> Or maybe just for the first one?
>> Because if I'm in Germany, have the German version of Windows and the
>> German keyboard,
>> I don't need to type  to get that symbol. I will just type it
>> on the keyboard.
>
> The program you're using should translate any strings into Unicode before it 
> calls the SQLite API.  If it is passing strings encoded any other way (e.g. 
> ANSI, ASCII, or a local code page) then it has a bug.

Well if you look at the code I provided, I'm retrieving the table name
from the database and then
convert it to the std::wstring for processing by my software.
This is what failed and so this is what I'm concern at the moment.

>
> Don't forget that you're meant to be able to send me your database.  I have a 
> Mac.  Macs don't use code pages at all.  But I should be able to type your 
> German or Chinese table name on my keyboard and get the right table.  And if 
> you think Mac users don't matter, consider all the people using iPhones all 
> over the world.  They don't use code pages either.  All Apple equipment uses 
> Unicode.

Well porting to Mac will open a completely different set of worms as
Apple is huge beast itself. ;-)
For now though I'm looking for the simplest case.

Thank you.

>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Simon Slavin]

On 24 Jun 2016, at 3:55pm, Igor Korot  wrote:

> Are those 3 scenarios correct?

They are if the shell tool (or any other SQLite software) works the way it 
should do.  If you have found a situation where the selected code page changes 
what makes it into a SQLite database then there is a bug somewhere.  The bug is 
probably in the program which calls the SQLite API, not in the API itself.

I cannot test the SQLite shell tool under Windows myself: I don't have a 
Windows computer.

> If the answer is yes, will the code I posted in the original post work
> for all 3 of them?
> Or maybe just for the first one?
> Because if I'm in Germany, have the German version of Windows and the
> German keyboard,
> I don't need to type  to get that symbol. I will just type it
> on the keyboard.

The program you're using should translate any strings into Unicode before it 
calls the SQLite API.  If it is passing strings encoded any other way (e.g. 
ANSI, ASCII, or a local code page) then it has a bug.

Don't forget that you're meant to be able to send me your database.  I have a 
Mac.  Macs don't use code pages at all.  But I should be able to type your 
German or Chinese table name on my keyboard and get the right table.  And if 
you think Mac users don't matter, consider all the people using iPhones all 
over the world.  They don't use code pages either.  All Apple equipment uses 
Unicode.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Igor Korot]

Rowan et al,

On Fri, Jun 24, 2016 at 4:34 AM, Rowan Worth  wrote:
> On 24 June 2016 at 16:13, Simon Slavin  wrote:
>
>> On 24 Jun 2016, at 5:04am, Igor Korot  wrote:
>>
>> > But everything should work independently of what code page is being used?
>>
>> The SQLite shell tool should work independently of the code page you have
>> set.  It should be translating strings from your current code page into
>> Unicode.  From your report it appears that it is not doing this, which
>> means there's a bug in it.
>>
>
> Thank you Simon for your voice of reason!
>
> For everyone else having a go at Igor, running the sqlite shell and typing
> some characters into it does not constitute any claim about how those
> characters are encoded. That's the operating system/application's concern.

I'm here in US and have only US_English version of Windows.
IIUC here are couple of scenarios:

Some guy in Germany (China, Japan) create SQLite db in the SQLite shell tool.
(S)he also have German (Chinese, Japanese) version of Windows (localized).
For whatever reason, (s)he decides to name all tables and fields in
German (Chinese, Japanese).

Now (s)he can inside the same tool do whatever - run queries (SELECT,
INSERT, UPDATE),
create triggers/stored procedures, etc. Because his/her
OS/keyboard/drivers allows straight
typing of native characters.

Now this guy can send his/hers db file to me alone with some scripts to run.
I should be able to open the db file on my US-English based Windows machine with
SQLite command line tool withouit any issues and either load the
script and execute it
or open the script in notepad and copy/paste the queries into the
SQLite shell tool.

And here is the scenario #3.
Here in US with US-English based Windows machine, I can add German
language and start
creating the database, tables and fields just like in the scenario 1.
And I will also be able to
run everything on the database.
And I should be able to send my db file to my German friend and he
will be able to also
open such db and run queries.

Are those 3 scenarios correct?

If the answer is yes, will the code I posted in the original post work
for all 3 of them?
Or maybe just for the first one?
Because if I'm in Germany, have the German version of Windows and the
German keyboard,
I don't need to type  to get that symbol. I will just type it
on the keyboard.
And if I have a German keyboard installed I also don't need to type
 to get
to this symbol. It might take me a while to find it on my English
keyboard, but that's
different story. ;-)

The thing is - all I care about are those 3 scenarios, because only
those are normal. Unfortunately I don't have an international version
of Windows to test the database created with international characters
appropriately and
I don't want to add another keyboard layout.
.
So I tried to emulate it and it looks like I failed to create a proper
test case.

So, can someone please answer those questions.

Thank you.

P.S.: I may support the  scenario, but not right now.

>
> -Rowan
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Rowan Worth]

On 24 June 2016 at 16:13, Simon Slavin  wrote:

> On 24 Jun 2016, at 5:04am, Igor Korot  wrote:
>
> > But everything should work independently of what code page is being used?
>
> The SQLite shell tool should work independently of the code page you have
> set.  It should be translating strings from your current code page into
> Unicode.  From your report it appears that it is not doing this, which
> means there's a bug in it.
>

Thank you Simon for your voice of reason!

For everyone else having a go at Igor, running the sqlite shell and typing
some characters into it does not constitute any claim about how those
characters are encoded. That's the operating system/application's concern.

-Rowan
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Simon Slavin]

On 24 Jun 2016, at 5:04am, Igor Korot  wrote:

> But everything should work independently of what code page is being used?

The SQLite shell tool should work independently of the code page you have set.  
It should be translating strings from your current code page into Unicode.  
From your report it appears that it is not doing this, which means there's a 
bug in it.

> My interface to the DB is based on the std::wstring. So, if someone
> will create the DB
> inside the shell and then create tables and name them using Chinese,
> why I should change
> something before doing it, just to satisfy one software?

You already have a database created using a code page.

Try setting your code page to 65001 (UTF-8 Unicode) and using the identical 
commands to create another database.

Do you get an identical file ?

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[R Smith]

On 2016/06/24 9:15 AM, Hick Gunter wrote:

I have a friend who owns an automated dog hotel. To check in your dog, it needs 
to go inside a standard dog box which is then put on a conveyor belt. To check 
out your dog, you present your receipt to a scanner, which causes the box to be 
retrieved and presetned on the conveyor belt. The boxes are usually never 
opened. Some folks have successfully managed to have their cats stay there. He 
recently added an automated trimming service, which would open the box, scan 
the dog, determine if nails or coat need trimming and perform the trimming. 
This service crashed when a cat owner inadvertently selected this option...


But what if he wrote "DOG Inside" on the side of the box? Wouldn't that 
fix the system?


___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[R Smith]

On 2016/06/24 6:04 AM, Igor Korot wrote:


No, I didn't.
But everything should work independently of what code page is being used?
My interface to the DB is based on the std::wstring. So, if someone
will create the DB
inside the shell and then create tables and name them using Chinese,
why I should change
something before doing it, just to satisfy one software?

Thank you.


The suggestions from people here are not to try and make SQLite work, it 
works fine. They are trying to help you to enter/insert/produce valid 
UTF-8, which you are clearly not doing.


SQLite reads UTF-8.
You are not giving it UTF-8.
It gives back what you gave it.
It still isn't UTF-8.

There is no mystery.

std::wstring is not guaranteed to function natively with UTF-8. here are 
some clues why and how to fix it:


http://stackoverflow.com/questions/4358870/convert-wstring-to-string-encoded-in-utf-8
http://stackoverflow.com/questions/2527720/confused-about-cs-stdwstring-utf-16-utf-8-and-displaying-strings-in-a-win

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Hick Gunter]

The hex dump shows the sz-ligature encoded as (hex) e1. That is NOT a valid UTF 
sequence.

Sqlite ASSUMES and EXPECTS all input to be UTF encoded. Passing ISO/ANSI 
encoded special characters in strings is ok as long as the presentation layer 
on both sides EXPECTS that same encoding. Using ISO/ANSI encoded characters in 
SQL Statements is not supported. It is up to your front end to properly convert 
between encodings in the presentation layer and in the data storage layer.

I have a friend who owns an automated dog hotel. To check in your dog, it needs 
to go inside a standard dog box which is then put on a conveyor belt. To check 
out your dog, you present your receipt to a scanner, which causes the box to be 
retrieved and presetned on the conveyor belt. The boxes are usually never 
opened. Some folks have successfully managed to have their cats stay there. He 
recently added an automated trimming service, which would open the box, scan 
the dog, determine if nails or coat need trimming and perform the trimming. 
This service crashed when a cat owner inadvertently selected this option...

-Ursprüngliche Nachricht-
Von: sqlite-users-boun...@mailinglists.sqlite.org 
[mailto:sqlite-users-boun...@mailinglists.sqlite.org] Im Auftrag von Igor Korot
Gesendet: Freitag, 24. Juni 2016 04:14
An: SQLite mailing list <sqlite-users@mailinglists.sqlite.org>
Betreff: Re: [sqlite] Conversion failure

Simon,

On Thu, Jun 23, 2016 at 8:44 PM, Simon Slavin <slav...@bigfraud.org> wrote:
>
> On 23 Jun 2016, at 11:42pm, Igor Korot <ikoro...@gmail.com> wrote:
>
>> OK, so are they UTF-8, UTF-16 or something else?
>> And I'm talking the default one - the one which is set when I just
>> open the shell tool and say CREATE TABLE() on the empty database.
>> Knowing this will help with the exception I am seeing.
>
> You can find out which encoding is being used by using
>
> PRAGMA encoding
>
> For further details please see
>
> <https://www.sqlite.org/pragma.html#pragma_encoding>

SQLite version 3.9.2 2015-11-02 18:31:45 Enter ".help" for usage hints.
sqlite> PRAGMA encoding;
UTF-8
sqlite> SELECT name FROM sqlite_master WHERE type="table";
leagues
owners
positions
positionsforleague
scorehits
scorepitch
scorehitsforleague
scorepitchforleague
teams
teamsforleague
playerposition
playerscorehits
playerscorepitches
targetshits
targetspitches
playerdraft
playerpositioninleague
players
playersinleague
leaguescorehitter
leaguescorepitcher
abcß
sqlite_sequence
sqlite>

The code from the initial post works for all tables but "abcß".

And here is the relevant piece of bytecodes: http://imgur.com/Trzylo4.

Thank you.

>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___
 Gunter Hick
Software Engineer
Scientific Games International GmbH
FN 157284 a, HG Wien
Klitschgasse 2-4, A-1130 Vienna, Austria
Tel: +43 1 80100 0
E-Mail: h...@scigames.at

This communication (including any attachments) is intended for the use of the 
intended recipient(s) only and may contain information that is confidential, 
privileged or legally protected. Any unauthorized use or dissemination of this 
communication is strictly prohibited. If you have received this communication 
in error, please immediately notify the sender by return e-mail message and 
delete all copies of the original communication. Thank you for your cooperation.


___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Rowan Worth]

The sqlite shell, at least historically, has I think not accounted for text
encoding and simply passed whatever it reads from the console into the
database.

There has been recent changes in this area since your last email on the
subject, for sqlite 3.12.0. What version are you using, Igor?

-Rowan

On 24 June 2016 at 12:04, Igor Korot  wrote:

> Kevin,
>
> On Thu, Jun 23, 2016 at 11:39 PM, Kevin Benson 
> wrote:
> > --
> >--
> >   --
> >  --Ô¿Ô--
> > K e V i N
> >
> > On Thu, Jun 23, 2016 at 10:38 AM, Igor Korot  wrote:
> >
> >> Hi, Clemens,
> >>
> >> On Thu, Jun 23, 2016 at 10:33 AM, Clemens Ladisch 
> >> wrote:
> >> > Igor Korot wrote:
> >> >> I am trying to find out why the following code fails to do proper
> >> conversion.
> >> >> It works if the tableName have "abcd", but fails if it has "abcß"
> (the
> >> >> German letter for the "ss" (looks like Greek letter beta)).
> >> >>
> >> >> const unsigned char *tableName = sqlite3_column_text( stmt, 0 );
> >> >> pimpl->m_tables[m_catalog].push_back( Table( myconv.from_bytes(
> (const
> >> char *) tableName ), fields, foreign_keys ) );
> >> >
> >> > What are the byte values in tableName?
> >>
> >> I have no idea.
> >> I just did ALT+225 in the SQLite shell during the CREATE TABLE command.
> >>
> >> Thank you.
> >>
> >>
> > Have you tried changing to Windows UTF-8 codepage by issuing
> >>chcp 65001
> > at the Windows command prompt, before starting the SQLite shell and doing
> > ALT+225 during the CREATE TABLE command ?
>
> No, I didn't.
> But everything should work independently of what code page is being used?
> My interface to the DB is based on the std::wstring. So, if someone
> will create the DB
> inside the shell and then create tables and name them using Chinese,
> why I should change
> something before doing it, just to satisfy one software?
>
> Thank you.
> > ___
> > sqlite-users mailing list
> > sqlite-users@mailinglists.sqlite.org
> > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Igor Korot]

Kevin,

On Thu, Jun 23, 2016 at 11:39 PM, Kevin Benson  wrote:
> --
>--
>   --
>  --Ô¿Ô--
> K e V i N
>
> On Thu, Jun 23, 2016 at 10:38 AM, Igor Korot  wrote:
>
>> Hi, Clemens,
>>
>> On Thu, Jun 23, 2016 at 10:33 AM, Clemens Ladisch 
>> wrote:
>> > Igor Korot wrote:
>> >> I am trying to find out why the following code fails to do proper
>> conversion.
>> >> It works if the tableName have "abcd", but fails if it has "abcß" (the
>> >> German letter for the "ss" (looks like Greek letter beta)).
>> >>
>> >> const unsigned char *tableName = sqlite3_column_text( stmt, 0 );
>> >> pimpl->m_tables[m_catalog].push_back( Table( myconv.from_bytes( (const
>> char *) tableName ), fields, foreign_keys ) );
>> >
>> > What are the byte values in tableName?
>>
>> I have no idea.
>> I just did ALT+225 in the SQLite shell during the CREATE TABLE command.
>>
>> Thank you.
>>
>>
> Have you tried changing to Windows UTF-8 codepage by issuing
>>chcp 65001
> at the Windows command prompt, before starting the SQLite shell and doing
> ALT+225 during the CREATE TABLE command ?

No, I didn't.
But everything should work independently of what code page is being used?
My interface to the DB is based on the std::wstring. So, if someone
will create the DB
inside the shell and then create tables and name them using Chinese,
why I should change
something before doing it, just to satisfy one software?

Thank you.
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Kevin Benson]

--
   --
  --
 --Ô¿Ô--
K e V i N

On Thu, Jun 23, 2016 at 10:38 AM, Igor Korot  wrote:

> Hi, Clemens,
>
> On Thu, Jun 23, 2016 at 10:33 AM, Clemens Ladisch 
> wrote:
> > Igor Korot wrote:
> >> I am trying to find out why the following code fails to do proper
> conversion.
> >> It works if the tableName have "abcd", but fails if it has "abcß" (the
> >> German letter for the "ss" (looks like Greek letter beta)).
> >>
> >> const unsigned char *tableName = sqlite3_column_text( stmt, 0 );
> >> pimpl->m_tables[m_catalog].push_back( Table( myconv.from_bytes( (const
> char *) tableName ), fields, foreign_keys ) );
> >
> > What are the byte values in tableName?
>
> I have no idea.
> I just did ALT+225 in the SQLite shell during the CREATE TABLE command.
>
> Thank you.
>
>
Have you tried changing to Windows UTF-8 codepage by issuing
>chcp 65001
at the Windows command prompt, before starting the SQLite shell and doing
ALT+225 during the CREATE TABLE command ?
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Igor Korot]

Simon,

On Thu, Jun 23, 2016 at 8:44 PM, Simon Slavin  wrote:
>
> On 23 Jun 2016, at 11:42pm, Igor Korot  wrote:
>
>> OK, so are they UTF-8, UTF-16 or something else?
>> And I'm talking the default one - the one which is set when I just
>> open the shell tool and say CREATE TABLE() on the
>> empty database.
>> Knowing this will help with the exception I am seeing.
>
> You can find out which encoding is being used by using
>
> PRAGMA encoding
>
> For further details please see
>
> 

SQLite version 3.9.2 2015-11-02 18:31:45
Enter ".help" for usage hints.
sqlite> PRAGMA encoding;
UTF-8
sqlite> SELECT name FROM sqlite_master WHERE type="table";
leagues
owners
positions
positionsforleague
scorehits
scorepitch
scorehitsforleague
scorepitchforleague
teams
teamsforleague
playerposition
playerscorehits
playerscorepitches
targetshits
targetspitches
playerdraft
playerpositioninleague
players
playersinleague
leaguescorehitter
leaguescorepitcher
abcß
sqlite_sequence
sqlite>

The code from the initial post works for all tables but "abcß".

And here is the relevant piece of bytecodes: http://imgur.com/Trzylo4.

Thank you.

>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Simon Slavin]

On 23 Jun 2016, at 11:42pm, Igor Korot  wrote:

> OK, so are they UTF-8, UTF-16 or something else?
> And I'm talking the default one - the one which is set when I just
> open the shell tool and say CREATE TABLE() on the
> empty database.
> Knowing this will help with the exception I am seeing.

You can find out which encoding is being used by using

PRAGMA encoding

For further details please see



Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Kees Nuyt]

On Thu, 23 Jun 2016 17:49:04 -0400, Igor Korot
 wrote:

> Yes, it is a PC (a laptop with Windows).
> OK, I understand and its unfortunate, but that's life.
>
> Now the question is: what is the best way to fix it?
> From the link I posted it sounds like there is a byte sequence before
> the string, which identifies the encoding.

The unicode character for "LATIN SMALL LETTER SHARP S" (the
es-zet ligature) is U+00DF, which is UTF-8 sequence c3 9f (hex).

Ref: 
http://www.utf8-chartable.de/
http://www.fileformat.info/info/charset/UTF-8/list.htm
http://dev.networkerror.org/utf8/

> Will this help?

Hope this helps.

-- 
Regards,

Kees Nuyt

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[R Smith]

On 2016/06/24 12:42 AM, Igor Korot wrote:


Your locale should not have any effect on what goes into a SQLite 
database. All strings must be translated into Unicode before they are 
passed to the SQLite API. And by 'All strings' I include SQL commands 
like the ones which create tables and define field names.

If your software works correctly then

* Your locale has no effect on entity names in a SQLite database
* Your code page has no effect on entity names in a SQLite database

If either of these are not true then there is a fault with your programming and 
it should be fixed.

And it is not.
As you can see I presume that whatever I got as a table name is UTF-8
and then I am converting it
to std::wstring.

Thank you.


Hi Igor,
I think the thing that is hard to explain is that when you set SQLite to 
use UTF-8 (the default in recent versions) it basically means that it 
/understands/ UTF-8 strings that you feed to it. It does NOT mean that 
it magically converts strings to UTF-8 or that it ever does any 
conversion, it only means that when you type UTF-8 or read UTF-8 text 
from files into it, it will grasp the contents in terms of reading the 
queries and other things you may have in the file as UTF-8.


What you specify as a table name, or put into the Database as values for 
fields, does not in any way get altered. Whatever you put into it is 
what will come out. If you /think/ you have put UTF-8 in there, but 
something else comes out, it means you did not put UTF-8 in there - and 
this is why people asked you for are the byte-sequences of the strings 
you put in. Sometimes it will look like UTF-8 but it may simply be ANSI 
or such. By the way, there is nothing precluding another code-page to 
have characters like the German SS - it can (and does) happily exist on 
many code pages. The fact that you can conjure it up in a CLI or some 
editor doesn't guarantee UTF-8-ness. [btw: AFAIK Windows console 
Alt-225, or anything above Alt-127, specifically isn't UTF-8]


I also understand that the way you connect to SQLite might make it hard 
to control or even be able to read the exact byte-sequences of the 
strings that go in and out of it... but I'm afraid you have to find a 
way for us to see what bytes those strings are really made of to help 
you figure out where it goes wrong (if indeed it does).


Best of luck!
Ryan

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Igor Korot]

Simon,

On Thu, Jun 23, 2016 at 6:14 PM, Simon Slavin  wrote:
>
> On 23 Jun 2016, at 10:53pm, Igor Korot  wrote:
>
>> So what I'm trying to do is to see what will happen if I get a
>> database with the German character(s)
>> or Asian characters in the table/field name and they will be entered
>> the same way.
>
> All table/field names in a SQLite database are Unicode.  They can have 
> characters in any language (or several languages) in and the will all work 
> fine.  Unless the program which created those databases had a bug in.

OK, so are they UTF-8, UTF-16 or something else?
And I'm talking the default one - the one which is set when I just
open the shell tool and say CREATE TABLE() on the
empty database.
Knowing this will help with the exception I am seeing.

>
>> My program will probably try to open on the same machine or at least
>> on the machine with the same locale,
>> but nevertheless, it should be possible to get the information I want.
>
> Your locale should not have any effect on what goes into a SQLite database.  
> All strings must be translated into Unicode before they are passed to the 
> SQLite API.  And by 'All strings' I include SQL commands like the ones which 
> create tables and define field names.
>
> If your software works correctly then
>
> * Your locale has no effect on entity names in a SQLite database
> * Your code page has no effect on entity names in a SQLite database
>
> If either of these are not true then there is a fault with your programming 
> and it should be fixed.

And it is not.
As you can see I presume that whatever I got as a table name is UTF-8
and then I am converting it
to std::wstring.

Thank you.

>
> Simon.
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Simon Slavin]

On 23 Jun 2016, at 10:53pm, Igor Korot  wrote:

> So what I'm trying to do is to see what will happen if I get a
> database with the German character(s)
> or Asian characters in the table/field name and they will be entered
> the same way.

All table/field names in a SQLite database are Unicode.  They can have 
characters in any language (or several languages) in and the will all work 
fine.  Unless the program which created those databases had a bug in.

> My program will probably try to open on the same machine or at least
> on the machine with the same locale,
> but nevertheless, it should be possible to get the information I want.

Your locale should not have any effect on what goes into a SQLite database.  
All strings must be translated into Unicode before they are passed to the 
SQLite API.  And by 'All strings' I include SQL commands like the ones which 
create tables and define field names.

If your software works correctly then

* Your locale has no effect on entity names in a SQLite database
* Your code page has no effect on entity names in a SQLite database

If either of these are not true then there is a fault with your programming and 
it should be fixed.

Simon.
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Igor Korot]

Hi, Ralf,

On Thu, Jun 23, 2016 at 12:38 PM, Ralf Junker  wrote:
> If you are on Windows, you can use SQLiteSpy to correct such wrongly entered
> ANSI text to Unicode throughout an entire database:

It is not wrongly entered text.
I am using Windows with English locale only.

So what I'm trying to do is to see what will happen if I get a
database with the German character(s)
or Asian characters in the table/field name and they will be entered
the same way.

My program will probably try to open on the same machine or at least
on the machine with the same locale,
but nevertheless, it should be possible to get the information I want.

Thank you for the suggestion though.

>
>   http://yunqa.de/delphi/products/sqlitespy/index
>
> Open the database and from the menu pick
>
>   Execute -> Text to Unicode Convertsion ...
>
> A dialog opens where you can check and uncheck the tables and columns to
> convert. Next choose the "Current Database Codepage". This should be the
> codepage of the wrongly entered text and defaults to your system codepage.
> If unsure, use an educated guess or try different codepages.
>
> Press OK to execute the conversion. For safety, all changes are by default
> performed within a transaction. The dialog closes when done and can browse
> the tables to check the changes. Depending on the outcome, at last manually
> COMMIT or ROLLBACK.
>
> Ralf
>
> On 23.06.2016 18:16, Hick Gunter wrote:
>
>> Your data entry device (I guess a PC running a flavor of windows) is
>> generating a certain sequence of bytes when you press ALT+225. This
>> sequence is probably ISO/ANSI encoded instead of UTF-8 encoded. It
>> has nothing to do with sqlite itself. Sqlite will faithfully
>> reproduce whatever byte sequence you gave to it.
>
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Igor Korot]

Hi, Gunter,

On Thu, Jun 23, 2016 at 12:16 PM, Hick Gunter <h...@scigames.at> wrote:
> Your data entry device (I guess a PC running a flavor of windows) is 
> generating a certain sequence of bytes when you press ALT+225. This sequence 
> is probably ISO/ANSI encoded instead of UTF-8 encoded. It has nothing to do 
> with sqlite itself. Sqlite will faithfully reproduce whatever byte sequence 
> you gave to it.

Yes, it is a PC (a laptop with Windows).
OK, I understand and its unfortunate, but that's life.

Now the question is: what is the best way to fix it?
From the link I posted it sounds like there is a byte sequence before
the string, which identifies the encoding.
Will this help?

Thank you.
>
> -Ursprüngliche Nachricht-
> Von: sqlite-users-boun...@mailinglists.sqlite.org 
> [mailto:sqlite-users-boun...@mailinglists.sqlite.org] Im Auftrag von Igor 
> Korot
> Gesendet: Donnerstag, 23. Juni 2016 17:58
> An: SQLite mailing list <sqlite-users@mailinglists.sqlite.org>
> Betreff: Re: [sqlite] Conversion failure
>
> Hi, Gunter,
>
> On Thu, Jun 23, 2016 at 10:59 AM, Hick Gunter <h...@scigames.at> wrote:
>> Open the editor application, type in your command, save to file and the view 
>> with a hex editor. I suspect it will be in ISO encoding.
>
> According to https://sqlite.org/src4/doc/trunk/www/data_encoding.wiki,
> all this data should be in some kind of
> UTF encoding (UTF-8 or UTF16).
> And I didn't explicitly changed the encoding at all.
>
> Thank you.
>
>>
>> -Ursprüngliche Nachricht-
>> Von: sqlite-users-boun...@mailinglists.sqlite.org
>> [mailto:sqlite-users-boun...@mailinglists.sqlite.org] Im Auftrag von
>> Igor Korot
>> Gesendet: Donnerstag, 23. Juni 2016 16:39
>> An: SQLite mailing list <sqlite-users@mailinglists.sqlite.org>
>> Betreff: Re: [sqlite] Conversion failure
>>
>> Hi, Clemens,
>>
>> On Thu, Jun 23, 2016 at 10:33 AM, Clemens Ladisch <clem...@ladisch.de> wrote:
>>> Igor Korot wrote:
>>>> I am trying to find out why the following code fails to do proper 
>>>> conversion.
>>>> It works if the tableName have "abcd", but fails if it has "abcß"
>>>> (the German letter for the "ss" (looks like Greek letter beta)).
>>>>
>>>> const unsigned char *tableName = sqlite3_column_text( stmt, 0 );
>>>> pimpl->m_tables[m_catalog].push_back( Table( myconv.from_bytes(
>>>> pimpl->(const char *) tableName ), fields, foreign_keys ) );
>>>
>>> What are the byte values in tableName?
>>
>> I have no idea.
>> I just did ALT+225 in the SQLite shell during the CREATE TABLE command.
>>
>> Thank you.
>>
>>>
>>>
>>> Regards,
>>> Clemens
>>> ___
>>> sqlite-users mailing list
>>> sqlite-users@mailinglists.sqlite.org
>>> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>> ___
>> sqlite-users mailing list
>> sqlite-users@mailinglists.sqlite.org
>> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>>
>>
>> ___
>>  Gunter Hick
>> Software Engineer
>> Scientific Games International GmbH
>> FN 157284 a, HG Wien
>> Klitschgasse 2-4, A-1130 Vienna, Austria
>> Tel: +43 1 80100 0
>> E-Mail: h...@scigames.at
>>
>> This communication (including any attachments) is intended for the use of 
>> the intended recipient(s) only and may contain information that is 
>> confidential, privileged or legally protected. Any unauthorized use or 
>> dissemination of this communication is strictly prohibited. If you have 
>> received this communication in error, please immediately notify the sender 
>> by return e-mail message and delete all copies of the original 
>> communication. Thank you for your cooperation.
>>
>>
>> ___
>> sqlite-users mailing list
>> sqlite-users@mailinglists.sqlite.org
>> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
>
> ___
>  Gunter Hick
> Software Engineer
> Scientific Games International GmbH
> FN 157284 a, HG Wien
> Klitschgasse 2-4, A-1130 Vienna, Austria
> Tel: +43 1 80100 0
> E-Mail: h..

Re: [sqlite] Conversion failure

[Ralf Junker]

If you are on Windows, you can use SQLiteSpy to correct such wrongly 
entered ANSI text to Unicode throughout an entire database:


  http://yunqa.de/delphi/products/sqlitespy/index

Open the database and from the menu pick

  Execute -> Text to Unicode Convertsion ...

A dialog opens where you can check and uncheck the tables and columns to 
convert. Next choose the "Current Database Codepage". This should be the 
codepage of the wrongly entered text and defaults to your system 
codepage. If unsure, use an educated guess or try different codepages.


Press OK to execute the conversion. For safety, all changes are by 
default performed within a transaction. The dialog closes when done and 
can browse the tables to check the changes. Depending on the outcome, at 
last manually COMMIT or ROLLBACK.


Ralf

On 23.06.2016 18:16, Hick Gunter wrote:


Your data entry device (I guess a PC running a flavor of windows) is
generating a certain sequence of bytes when you press ALT+225. This
sequence is probably ISO/ANSI encoded instead of UTF-8 encoded. It
has nothing to do with sqlite itself. Sqlite will faithfully
reproduce whatever byte sequence you gave to it.

___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Hick Gunter]

Your data entry device (I guess a PC running a flavor of windows) is generating 
a certain sequence of bytes when you press ALT+225. This sequence is probably 
ISO/ANSI encoded instead of UTF-8 encoded. It has nothing to do with sqlite 
itself. Sqlite will faithfully reproduce whatever byte sequence you gave to it.

-Ursprüngliche Nachricht-
Von: sqlite-users-boun...@mailinglists.sqlite.org 
[mailto:sqlite-users-boun...@mailinglists.sqlite.org] Im Auftrag von Igor Korot
Gesendet: Donnerstag, 23. Juni 2016 17:58
An: SQLite mailing list <sqlite-users@mailinglists.sqlite.org>
Betreff: Re: [sqlite] Conversion failure

Hi, Gunter,

On Thu, Jun 23, 2016 at 10:59 AM, Hick Gunter <h...@scigames.at> wrote:
> Open the editor application, type in your command, save to file and the view 
> with a hex editor. I suspect it will be in ISO encoding.

According to https://sqlite.org/src4/doc/trunk/www/data_encoding.wiki,
all this data should be in some kind of
UTF encoding (UTF-8 or UTF16).
And I didn't explicitly changed the encoding at all.

Thank you.

>
> -Ursprüngliche Nachricht-
> Von: sqlite-users-boun...@mailinglists.sqlite.org
> [mailto:sqlite-users-boun...@mailinglists.sqlite.org] Im Auftrag von
> Igor Korot
> Gesendet: Donnerstag, 23. Juni 2016 16:39
> An: SQLite mailing list <sqlite-users@mailinglists.sqlite.org>
> Betreff: Re: [sqlite] Conversion failure
>
> Hi, Clemens,
>
> On Thu, Jun 23, 2016 at 10:33 AM, Clemens Ladisch <clem...@ladisch.de> wrote:
>> Igor Korot wrote:
>>> I am trying to find out why the following code fails to do proper 
>>> conversion.
>>> It works if the tableName have "abcd", but fails if it has "abcß"
>>> (the German letter for the "ss" (looks like Greek letter beta)).
>>>
>>> const unsigned char *tableName = sqlite3_column_text( stmt, 0 );
>>> pimpl->m_tables[m_catalog].push_back( Table( myconv.from_bytes(
>>> pimpl->(const char *) tableName ), fields, foreign_keys ) );
>>
>> What are the byte values in tableName?
>
> I have no idea.
> I just did ALT+225 in the SQLite shell during the CREATE TABLE command.
>
> Thank you.
>
>>
>>
>> Regards,
>> Clemens
>> ___
>> sqlite-users mailing list
>> sqlite-users@mailinglists.sqlite.org
>> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
>
> ___
>  Gunter Hick
> Software Engineer
> Scientific Games International GmbH
> FN 157284 a, HG Wien
> Klitschgasse 2-4, A-1130 Vienna, Austria
> Tel: +43 1 80100 0
> E-Mail: h...@scigames.at
>
> This communication (including any attachments) is intended for the use of the 
> intended recipient(s) only and may contain information that is confidential, 
> privileged or legally protected. Any unauthorized use or dissemination of 
> this communication is strictly prohibited. If you have received this 
> communication in error, please immediately notify the sender by return e-mail 
> message and delete all copies of the original communication. Thank you for 
> your cooperation.
>
>
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___
 Gunter Hick
Software Engineer
Scientific Games International GmbH
FN 157284 a, HG Wien
Klitschgasse 2-4, A-1130 Vienna, Austria
Tel: +43 1 80100 0
E-Mail: h...@scigames.at

This communication (including any attachments) is intended for the use of the 
intended recipient(s) only and may contain information that is confidential, 
privileged or legally protected. Any unauthorized use or dissemination of this 
communication is strictly prohibited. If you have received this communication 
in error, please immediately notify the sender by return e-mail message and 
delete all copies of the original communication. Thank you for your cooperation.


___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Igor Korot]

Hi, Gunter,

On Thu, Jun 23, 2016 at 10:59 AM, Hick Gunter <h...@scigames.at> wrote:
> Open the editor application, type in your command, save to file and the view 
> with a hex editor. I suspect it will be in ISO encoding.

According to https://sqlite.org/src4/doc/trunk/www/data_encoding.wiki,
all this data should be in some kind of
UTF encoding (UTF-8 or UTF16).
And I didn't explicitly changed the encoding at all.

Thank you.

>
> -Ursprüngliche Nachricht-
> Von: sqlite-users-boun...@mailinglists.sqlite.org 
> [mailto:sqlite-users-boun...@mailinglists.sqlite.org] Im Auftrag von Igor 
> Korot
> Gesendet: Donnerstag, 23. Juni 2016 16:39
> An: SQLite mailing list <sqlite-users@mailinglists.sqlite.org>
> Betreff: Re: [sqlite] Conversion failure
>
> Hi, Clemens,
>
> On Thu, Jun 23, 2016 at 10:33 AM, Clemens Ladisch <clem...@ladisch.de> wrote:
>> Igor Korot wrote:
>>> I am trying to find out why the following code fails to do proper 
>>> conversion.
>>> It works if the tableName have "abcd", but fails if it has "abcß"
>>> (the German letter for the "ss" (looks like Greek letter beta)).
>>>
>>> const unsigned char *tableName = sqlite3_column_text( stmt, 0 );
>>> pimpl->m_tables[m_catalog].push_back( Table( myconv.from_bytes(
>>> pimpl->(const char *) tableName ), fields, foreign_keys ) );
>>
>> What are the byte values in tableName?
>
> I have no idea.
> I just did ALT+225 in the SQLite shell during the CREATE TABLE command.
>
> Thank you.
>
>>
>>
>> Regards,
>> Clemens
>> ___
>> sqlite-users mailing list
>> sqlite-users@mailinglists.sqlite.org
>> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
>
>
> ___
>  Gunter Hick
> Software Engineer
> Scientific Games International GmbH
> FN 157284 a, HG Wien
> Klitschgasse 2-4, A-1130 Vienna, Austria
> Tel: +43 1 80100 0
> E-Mail: h...@scigames.at
>
> This communication (including any attachments) is intended for the use of the 
> intended recipient(s) only and may contain information that is confidential, 
> privileged or legally protected. Any unauthorized use or dissemination of 
> this communication is strictly prohibited. If you have received this 
> communication in error, please immediately notify the sender by return e-mail 
> message and delete all copies of the original communication. Thank you for 
> your cooperation.
>
>
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Hick Gunter]

Open the editor application, type in your command, save to file and the view 
with a hex editor. I suspect it will be in ISO encoding.

-Ursprüngliche Nachricht-
Von: sqlite-users-boun...@mailinglists.sqlite.org 
[mailto:sqlite-users-boun...@mailinglists.sqlite.org] Im Auftrag von Igor Korot
Gesendet: Donnerstag, 23. Juni 2016 16:39
An: SQLite mailing list <sqlite-users@mailinglists.sqlite.org>
Betreff: Re: [sqlite] Conversion failure

Hi, Clemens,

On Thu, Jun 23, 2016 at 10:33 AM, Clemens Ladisch <clem...@ladisch.de> wrote:
> Igor Korot wrote:
>> I am trying to find out why the following code fails to do proper conversion.
>> It works if the tableName have "abcd", but fails if it has "abcß"
>> (the German letter for the "ss" (looks like Greek letter beta)).
>>
>> const unsigned char *tableName = sqlite3_column_text( stmt, 0 );
>> pimpl->m_tables[m_catalog].push_back( Table( myconv.from_bytes(
>> pimpl->(const char *) tableName ), fields, foreign_keys ) );
>
> What are the byte values in tableName?

I have no idea.
I just did ALT+225 in the SQLite shell during the CREATE TABLE command.

Thank you.

>
>
> Regards,
> Clemens
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users


___
 Gunter Hick
Software Engineer
Scientific Games International GmbH
FN 157284 a, HG Wien
Klitschgasse 2-4, A-1130 Vienna, Austria
Tel: +43 1 80100 0
E-Mail: h...@scigames.at

This communication (including any attachments) is intended for the use of the 
intended recipient(s) only and may contain information that is confidential, 
privileged or legally protected. Any unauthorized use or dissemination of this 
communication is strictly prohibited. If you have received this communication 
in error, please immediately notify the sender by return e-mail message and 
delete all copies of the original communication. Thank you for your cooperation.


___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Igor Korot]

Hi, Clemens,

On Thu, Jun 23, 2016 at 10:33 AM, Clemens Ladisch  wrote:
> Igor Korot wrote:
>> I am trying to find out why the following code fails to do proper conversion.
>> It works if the tableName have "abcd", but fails if it has "abcß" (the
>> German letter for the "ss" (looks like Greek letter beta)).
>>
>> const unsigned char *tableName = sqlite3_column_text( stmt, 0 );
>> pimpl->m_tables[m_catalog].push_back( Table( myconv.from_bytes( (const char 
>> *) tableName ), fields, foreign_keys ) );
>
> What are the byte values in tableName?

I have no idea.
I just did ALT+225 in the SQLite shell during the CREATE TABLE command.

Thank you.

>
>
> Regards,
> Clemens
> ___
> sqlite-users mailing list
> sqlite-users@mailinglists.sqlite.org
> http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Conversion failure

[Clemens Ladisch]

Igor Korot wrote:
> I am trying to find out why the following code fails to do proper conversion.
> It works if the tableName have "abcd", but fails if it has "abcß" (the
> German letter for the "ss" (looks like Greek letter beta)).
>
> const unsigned char *tableName = sqlite3_column_text( stmt, 0 );
> pimpl->m_tables[m_catalog].push_back( Table( myconv.from_bytes( (const char 
> *) tableName ), fields, foreign_keys ) );

What are the byte values in tableName?


Regards,
Clemens
___
sqlite-users mailing list
sqlite-users@mailinglists.sqlite.org
http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users