Re: [sqlite] Accented characters and ODBC - Fixed

[Greg Bryant]

I had the right idea, but the wrong code page.

So, going from MBCS project in visual c++, there is a magic ANSI to UTF8
conversion happening when the data is stored, but not one coming back out.
I could not find a direct UTF8 to ANSI code page conversion, presumably
since they are both "multibyte", but I found a function online that would
take the string from UTF8 to back to ANSI - by routing it through an extra
conversion. (First do a MultiByteToWideChar using the UTF8 code page 65001,
then WideCharToMultiByte using the ANSI code page, 1252).  Algorithm found
at http://www.chilkatsoft.com/p/p_348.asp.

Greg

On Thu, Oct 7, 2010 at 11:22 AM, Greg Bryant  wrote:

> Still hoping someone can help me with this.  I dug into it some with a hex
> editor, and it may be the app mode, which is multi-byte (and makes extensive
> use of libraries, so is not about to change).
>
> If I put the string Andé in, it is (41 6e 64 e9) in the debugger, but
> looking at the sqlite database file, it is (41 6e 64 c3 a9), with the c3
> appearing to be a UTF-8 "trigger" indicating a 2 byte characters.  So,
> somewhere along the way it went from mb to utf8, and it's just not being
> converted back out for the SELECT.  I tried a quick wide-to-multibyte
> conversion, but got real garbage then.  Next step is a manual mb to utf8
> conversion going in, but if anyone has any pointers, I'd sure love to have
> them, since I'm just guessing.
>
> Greg
>
>
> On Fri, Oct 1, 2010 at 12:51 PM, Greg Bryant  wrote:
>
>> Not sure if ODBC questions belong here, feel free to point me to a better
>> forum.
>>
>> I'm using current SQLite (3.7.2) via a visual c++ app.  We're connecting a
>> sqlite3 database via ODBC (driver from , also current version - 0.87).  If I
>> do either an insert or update using an accented character (e.g. André), it
>> works fine, and I can check it using anything that directly connects to the
>> database (I've been using SQLiteStudio and SQLiteAdmin) and it looks fine.
>> When I go to do a select via the ODBC connection, however, I get back
>> garbage for the accented character (specifically, I get André).
>>
>> I don't think the app settings are a problem, since the accent works fine
>> through the insert, and step into doesn't get me past the SQLFetch from
>> Microsoft's odbc layer, and I don't think it's sqlite, since it appears to
>> be correct when viewed through a direct connection, which just leaves the
>> odbc layer.  Does anyone have any pointers on where I can look to figure
>> this out?
>>
>> Thanks,
>> Greg
>>
>
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Accented characters and ODBC

[Greg Bryant]

Still hoping someone can help me with this.  I dug into it some with a hex
editor, and it may be the app mode, which is multi-byte (and makes extensive
use of libraries, so is not about to change).

If I put the string Andé in, it is (41 6e 64 e9) in the debugger, but
looking at the sqlite database file, it is (41 6e 64 c3 a9), with the c3
appearing to be a UTF-8 "trigger" indicating a 2 byte characters.  So,
somewhere along the way it went from mb to utf8, and it's just not being
converted back out for the SELECT.  I tried a quick wide-to-multibyte
conversion, but got real garbage then.  Next step is a manual mb to utf8
conversion going in, but if anyone has any pointers, I'd sure love to have
them, since I'm just guessing.

Greg

On Fri, Oct 1, 2010 at 12:51 PM, Greg Bryant  wrote:

> Not sure if ODBC questions belong here, feel free to point me to a better
> forum.
>
> I'm using current SQLite (3.7.2) via a visual c++ app.  We're connecting a
> sqlite3 database via ODBC (driver from , also current version - 0.87).  If I
> do either an insert or update using an accented character (e.g. André), it
> works fine, and I can check it using anything that directly connects to the
> database (I've been using SQLiteStudio and SQLiteAdmin) and it looks fine.
> When I go to do a select via the ODBC connection, however, I get back
> garbage for the accented character (specifically, I get André).
>
> I don't think the app settings are a problem, since the accent works fine
> through the insert, and step into doesn't get me past the SQLFetch from
> Microsoft's odbc layer, and I don't think it's sqlite, since it appears to
> be correct when viewed through a direct connection, which just leaves the
> odbc layer.  Does anyone have any pointers on where I can look to figure
> this out?
>
> Thanks,
> Greg
>
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Accented characters and ODBC

[Christian Werner]

Greg Bryant wrote:
> 
> Not sure if ODBC questions belong here, feel free to point me to a better
> forum.
> 
> I'm using current SQLite (3.7.2) via a visual c++ app.  We're connecting a
> sqlite3 database via ODBC (driver from , also current version - 0.87).  If I
> do either an insert or update using an accented character (e.g. André), it
> works fine, and I can check it using anything that directly connects to the
> database (I've been using SQLiteStudio and SQLiteAdmin) and it looks fine.
> When I go to do a select via the ODBC connection, however, I get back
> garbage for the accented character (specifically, I get AndrÃfÆ?©).
> 
> I don't think the app settings are a problem, since the accent works fine
> through the insert, and step into doesn't get me past the SQLFetch from
> Microsoft's odbc layer, and I don't think it's sqlite, since it appears to
> be correct when viewed through a direct connection, which just leaves the
> odbc layer.  Does anyone have any pointers on where I can look to figure
> this out?

How do you retrieve character data? When using SQL_C_CHAR the 0.87 version
of the driver outputs UTF-8 strings whereas your application deals most
likely with codepage 1251 (Windows Latin-1). When you use SQL_C_WCHAR
everything should be fine. You might also try out a pre-release which has
UTF-8 to/from CP_OEMCP/CP_ACP built into the SQLite3 driver, it is on

 http://www.ch-werner.de/sqliteodbc/sqliteodbc-088pre2.exe

OTOH, I'm not familiar with SQLiteStudio and SQLiteAdmin, do these
insert/update UTF-8 encoded strings?

Regards,
Christian
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Accented characters and ODBC

[Oliver Peters]

Greg Bryant  writes:

just saw that you usually should install 3 different ODBC-drivers and 1 of them
is called SQLite ODBC (UTF-8) Driver.

Maybe you should give it a try before asking the programmer

greetings




___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Accented characters and ODBC

[Oliver Peters]

Greg Bryant  writes:

[...]

> Does anyone have any pointers on where I can look to figure
> this out?

[...]

maybe the friendly driver programmer is willing to help you
if you send a bug report

http://www.ch-werner.de/sqliteodbc/html/index.html

greetings
Oliver

___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Accented characters and ODBC

[Drake Wilson]

Quoth Greg Bryant , on 2010-10-01 12:51:10 -0500:
> I'm using current SQLite (3.7.2) via a visual c++ app.  We're connecting a
> sqlite3 database via ODBC (driver from

Driver from where again?

> When I go to do a select via the ODBC connection, however, I get back
> garbage for the accented character (specifically, I get André).

You're getting back UTF-8 and then (somewhere) interpreting it as
Latin-1.  I'm not too familiar with ODBC; is there a way you can
specify the connection's text encoding?  SQLite uses UTF-8 or UTF-16
for text.  (You can choose which only at database creation time.)
Similarly, make sure that you actually give it Unicode strings in the
target encoding; there may be some autocorrection going on if you try
to feed it Latin-1 characters, but I wouldn't rely on it.

> Thanks,
> Greg

   ---> Drake Wilson
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[sqlite] Accented characters and ODBC

[Greg Bryant]

Not sure if ODBC questions belong here, feel free to point me to a better
forum.

I'm using current SQLite (3.7.2) via a visual c++ app.  We're connecting a
sqlite3 database via ODBC (driver from , also current version - 0.87).  If I
do either an insert or update using an accented character (e.g. André), it
works fine, and I can check it using anything that directly connects to the
database (I've been using SQLiteStudio and SQLiteAdmin) and it looks fine.
When I go to do a select via the ODBC connection, however, I get back
garbage for the accented character (specifically, I get André).

I don't think the app settings are a problem, since the accent works fine
through the insert, and step into doesn't get me past the SQLFetch from
Microsoft's odbc layer, and I don't think it's sqlite, since it appears to
be correct when viewed through a direct connection, which just leaves the
odbc layer.  Does anyone have any pointers on where I can look to figure
this out?

Thanks,
Greg
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users