Re: [sqlite] UTF16 - sqlite3_prepare16_v2
Oops -- you're right...I was paying attention to comments and didn't notice the 16 statement that followed it. Minor documentation boo-boo I guess... It appears that the "L" produces UTF-32 in Unix and UTF-16 in Windows. So the code in sqlite3VdbeMemSetStr only copies the "C" in Unix but does the whole string in Windows. I put a couple of debug statements in and can see the diff between the two. I did this: for(nByte=0; nByte<=iLimit && z[nByte]; nByte++){} printf("nbyte8=%d\n",nByte); }else{ for(nByte=0; nByte<=iLimit && (z[nByte] | z[nByte+1]); nByte+=2){} printf("nbyte!8=%d\n",nByte); And I get this output from Unix: not an error nbyte!8=2 nbyte8=22 near "C": syntax error And this from Windows: not an error nbyte!8=202 nbyte8=4 nbyte8=8 nbyte8=3 nbyte8=4 nbyte8=7 nbyte8=4 not an error nbyte8=5 nbyte8=4 nbyte8=4 nbyte8=100 nbyte8=4 nbyte8=8 nbyte8=3 nbyte8=4 nbyte8=7 nbyte8=4 nbyte8=4 nbyte8=45 #include #include "sqlite3.h" int main() { sqlite3 *Database; sqlite3_stmt *pStatement; sqlite3_open_v2("test.sqlite", &Database, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, 0); sqlite3_exec(Database, "PRAGMA encoding = \"UTF-16\"", 0, 0, 0); std::cout << sqlite3_errmsg(Database) << std::endl; sqlite3_prepare16_v2(Database, L"CREATE TABLE user (userID INTEGER NOT NULL PRIM ARY KEY, lastName VARCHAR(50), firstName VARCHAR(50));", -1,&pStatement, 0); std::cout << sqlite3_errmsg(Database) << std::endl; sqlite3_step(pStatement); sqlite3_close(Database); return 0; } Michael D. Black Senior Scientist Northrop Grumman Mission Systems ________ From: sqlite-users-boun...@sqlite.org on behalf of Jay A. Kreibich Sent: Sat 5/22/2010 9:43 AM To: General Discussion of SQLite Database Subject: Re: [sqlite] UTF16 - sqlite3_prepare16_v2 On Sat, May 22, 2010 at 09:23:06AM -0500, Black, Michael (IS) scratched on the wall: > I think you're making the mistake of thinking that the entire SQL > string is UTF-16. It is. That's the whole point of the "16" interfaces. > Look at the API for sqlite3_prepare16_v2 > SQLITE_API int sqlite3_prepare16_v2( > sqlite3 *db, /* Database handle. */ > const void *zSql, /* UTF-8 encoded SQL statement. */ Yes, I know it says that in the source next to the function, but that's wrong. Following the source confirms that. A set of copy-paste errors, no doubt. The website (and the prototypes in the source) are correct: http://www.sqlite.org/c3ref/prepare.html int sqlite3_prepare16_v2( sqlite3 *db,/* Database handle */ const void *zSql, /* SQL statement, UTF-16 encoded */ int nByte, /* Maximum length of zSql in bytes. */ sqlite3_stmt **ppStmt, /* OUT: Statement handle */ const void **pzTail /* OUT: Pointer to unused portion of zSql */ ); > It's your data fields that get encoded. Not exactly. The database has a single encoding for all text values. Any text values that are recorded in the database are converted to that encoding. Consider bind values. You can specify input values in whatever encoding you want. SQLite will convert them to whatever the database encoding is before recording them. It works the same on output, where you can request a column in whatever encoding you want, regardless of the database encoding. sqlite3_open() and sqlite3_open16() are a bit different. In addition to the filename, using sqlite3_open16() to create a database will set the global database encoding to UTF-16. But that has a whole different set of issues, which is why there is no sqlite3_open16_v2(). -j -- Jay A. Kreibich < J A Y @ K R E I B I.C H > "Intelligence is like underwear: it is important that you have it, but showing it to the wrong people has the tendency to make them feel uncomfortable." -- Angela Johnson ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF16 - sqlite3_prepare16_v2
That was it. I didn't know that the wchar_t has different bit sizes on Linux and Windows. Thank you very much. Jay A. Kreibich-2 wrote: > >> I know that the SQL syntax is correct and it is possible to execute that >> source under windows. >> I don't understand why it doesn't work with Linux. >> Does somebody has an idea what's going wrong? > > Wide characters are 32 bits on most non-Windows platforms. When > passed to a 16 bit system, the first character of L"CREATE..." is > seen as ['C', '\0']. > > http://en.wikipedia.org/wiki/C_syntax#Wide_character_strings > > This is part of the reason SQLite3 uses void* rather than wchar_t* > for UTF-16 strings. > -- View this message in context: http://old.nabble.com/UTF16---sqlite3_prepare16_v2-tp28643020p28644005.html Sent from the SQLite mailing list archive at Nabble.com. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF16 - sqlite3_prepare16_v2
If i take a look at the declaration of sqlite3_prepare16_v2() it looks like this: SQLITE_API int sqlite3_prepare16_v2( sqlite3 *db,/* Database handle */ const void *zSql, /* SQL statement, UTF-16 encoded */ There is a UTF-16 encoded string needed. Black, Michael (IS) wrote: > > I think you're making the mistake of thinking that the entire SQL string > is UTF-16. > > Look at the API for sqlite3_prepare16_v2 > SQLITE_API int sqlite3_prepare16_v2( > sqlite3 *db, /* Database handle. */ > const void *zSql, /* UTF-8 encoded SQL statement. */ > > It's your data fields that get encoded. > -- View this message in context: http://old.nabble.com/UTF16---sqlite3_prepare16_v2-tp28643020p28643888.html Sent from the SQLite mailing list archive at Nabble.com. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF16 - sqlite3_prepare16_v2
On Sat, May 22, 2010 at 09:23:06AM -0500, Black, Michael (IS) scratched on the wall: > I think you're making the mistake of thinking that the entire SQL > string is UTF-16. It is. That's the whole point of the "16" interfaces. > Look at the API for sqlite3_prepare16_v2 > SQLITE_API int sqlite3_prepare16_v2( > sqlite3 *db, /* Database handle. */ > const void *zSql, /* UTF-8 encoded SQL statement. */ Yes, I know it says that in the source next to the function, but that's wrong. Following the source confirms that. A set of copy-paste errors, no doubt. The website (and the prototypes in the source) are correct: http://www.sqlite.org/c3ref/prepare.html int sqlite3_prepare16_v2( sqlite3 *db,/* Database handle */ const void *zSql, /* SQL statement, UTF-16 encoded */ int nByte, /* Maximum length of zSql in bytes. */ sqlite3_stmt **ppStmt, /* OUT: Statement handle */ const void **pzTail /* OUT: Pointer to unused portion of zSql */ ); > It's your data fields that get encoded. Not exactly. The database has a single encoding for all text values. Any text values that are recorded in the database are converted to that encoding. Consider bind values. You can specify input values in whatever encoding you want. SQLite will convert them to whatever the database encoding is before recording them. It works the same on output, where you can request a column in whatever encoding you want, regardless of the database encoding. sqlite3_open() and sqlite3_open16() are a bit different. In addition to the filename, using sqlite3_open16() to create a database will set the global database encoding to UTF-16. But that has a whole different set of issues, which is why there is no sqlite3_open16_v2(). -j -- Jay A. Kreibich < J A Y @ K R E I B I.C H > "Intelligence is like underwear: it is important that you have it, but showing it to the wrong people has the tendency to make them feel uncomfortable." -- Angela Johnson ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF16 - sqlite3_prepare16_v2
I think you're making the mistake of thinking that the entire SQL string is UTF-16. Look at the API for sqlite3_prepare16_v2 SQLITE_API int sqlite3_prepare16_v2( sqlite3 *db, /* Database handle. */ const void *zSql, /* UTF-8 encoded SQL statement. */ It's your data fields that get encoded. I'm not a UTF-16 expert by any means but that's how it appears to me. I'm surprised it works in windows -- I did confirm your error on Linux. Michael D. Black Senior Scientist Northrop Grumman Mission Systems From: sqlite-users-boun...@sqlite.org on behalf of shion Sent: Sat 5/22/2010 7:27 AM To: sqlite-users@sqlite.org Subject: [sqlite] UTF16 - sqlite3_prepare16_v2 Hello, I already found a lot of threads about the UTF16 topic, but I couldn't solve my problem. I have the following source code: sqlite3_open_v2("test.sqlite", &Database, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, 0); sqlite3_exec(Database, "PRAGMA encoding = \"UTF-16\"", 0, 0, 0); sqlite3_prepare16_v2(Database, L"CREATE TABLE user (userID INTEGER NOT NULL PRIMARY KEY, lastName VARCHAR(50), firstName VARCHAR(50));", -1, &pStatement, 0); std::cout << sqlite3_errmsg(Database) << std::endl; The result is: near "C": syntax error I know that the SQL syntax is correct and it is possible to execute that source under windows. I don't understand why it doesn't work with Linux. Does somebody has an idea what's going wrong? Operating system: 2.6.31-19-generic #56-Ubuntu SMP i686 GNU/Linux IDE: NetBeans Regards, shion -- View this message in context: http://old.nabble.com/UTF16---sqlite3_prepare16_v2-tp28643020p28643020.html Sent from the SQLite mailing list archive at Nabble.com. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF16 - sqlite3_prepare16_v2
On Sat, May 22, 2010 at 05:27:24AM -0700, shion scratched on the wall: > > Hello, > > I already found a lot of threads about the UTF16 topic, but I couldn't solve > my problem. > I have the following source code: > > sqlite3_open_v2("test.sqlite", &Database, SQLITE_OPEN_READWRITE | > SQLITE_OPEN_CREATE, 0); > sqlite3_exec(Database, "PRAGMA encoding = \"UTF-16\"", 0, 0, 0); > sqlite3_prepare16_v2(Database, L"CREATE TABLE user (userID INTEGER NOT NULL > PRIMARY KEY, lastName VARCHAR(50), firstName VARCHAR(50));", -1, > &pStatement, 0); > std::cout << sqlite3_errmsg(Database) << std::endl; > > The result is: > near "C": syntax error Compile or runtime? Runtime, I assume. > I know that the SQL syntax is correct and it is possible to execute that > source under windows. > I don't understand why it doesn't work with Linux. > Does somebody has an idea what's going wrong? Wide characters are 32 bits on most non-Windows platforms. When passed to a 16 bit system, the first character of L"CREATE..." is seen as ['C', '\0']. http://en.wikipedia.org/wiki/C_syntax#Wide_character_strings This is part of the reason SQLite3 uses void* rather than wchar_t* for UTF-16 strings. -j -- Jay A. Kreibich < J A Y @ K R E I B I.C H > "Intelligence is like underwear: it is important that you have it, but showing it to the wrong people has the tendency to make them feel uncomfortable." -- Angela Johnson ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] UTF16 - sqlite3_prepare16_v2
Hello, I already found a lot of threads about the UTF16 topic, but I couldn't solve my problem. I have the following source code: sqlite3_open_v2("test.sqlite", &Database, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, 0); sqlite3_exec(Database, "PRAGMA encoding = \"UTF-16\"", 0, 0, 0); sqlite3_prepare16_v2(Database, L"CREATE TABLE user (userID INTEGER NOT NULL PRIMARY KEY, lastName VARCHAR(50), firstName VARCHAR(50));", -1, &pStatement, 0); std::cout << sqlite3_errmsg(Database) << std::endl; The result is: near "C": syntax error I know that the SQL syntax is correct and it is possible to execute that source under windows. I don't understand why it doesn't work with Linux. Does somebody has an idea what's going wrong? Operating system: 2.6.31-19-generic #56-Ubuntu SMP i686 GNU/Linux IDE: NetBeans Regards, shion -- View this message in context: http://old.nabble.com/UTF16---sqlite3_prepare16_v2-tp28643020p28643020.html Sent from the SQLite mailing list archive at Nabble.com. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users