Re: [sqlite] UTF16 - sqlite3_prepare16_v2
Oops -- you're right...I was paying attention to comments and didn't notice the
16 statement that followed it.
Minor documentation boo-boo I guess...
It appears that the "L" produces UTF-32 in Unix and UTF-16 in Windows.
So the code in sqlite3VdbeMemSetStr only copies the "C" in Unix but does the
whole string in Windows.
I put a couple of debug statements in and can see the diff between the two.
I did this:
for(nByte=0; nByte<=iLimit && z[nByte]; nByte++){}
printf("nbyte8=%d\n",nByte);
}else{
for(nByte=0; nByte<=iLimit && (z[nByte] | z[nByte+1]); nByte+=2){}
printf("nbyte!8=%d\n",nByte);
And I get this output from Unix:
not an error
nbyte!8=2
nbyte8=22
near "C": syntax error
And this from Windows:
not an error
nbyte!8=202
nbyte8=4
nbyte8=8
nbyte8=3
nbyte8=4
nbyte8=7
nbyte8=4
not an error
nbyte8=5
nbyte8=4
nbyte8=4
nbyte8=100
nbyte8=4
nbyte8=8
nbyte8=3
nbyte8=4
nbyte8=7
nbyte8=4
nbyte8=4
nbyte8=45
#include
#include "sqlite3.h"
int main()
{
sqlite3 *Database;
sqlite3_stmt *pStatement;
sqlite3_open_v2("test.sqlite", , SQLITE_OPEN_READWRITE |
SQLITE_OPEN_CREATE, 0);
sqlite3_exec(Database, "PRAGMA encoding = \"UTF-16\"", 0, 0, 0);
std::cout << sqlite3_errmsg(Database) << std::endl;
sqlite3_prepare16_v2(Database, L"CREATE TABLE user (userID INTEGER NOT NULL PRIM
ARY KEY, lastName VARCHAR(50), firstName VARCHAR(50));", -1,, 0);
std::cout << sqlite3_errmsg(Database) << std::endl;
sqlite3_step(pStatement);
sqlite3_close(Database);
return 0;
}
Michael D. Black
Senior Scientist
Northrop Grumman Mission Systems
From: sqlite-users-boun...@sqlite.org on behalf of Jay A. Kreibich
Sent: Sat 5/22/2010 9:43 AM
To: General Discussion of SQLite Database
Subject: Re: [sqlite] UTF16 - sqlite3_prepare16_v2
On Sat, May 22, 2010 at 09:23:06AM -0500, Black, Michael (IS) scratched on the
wall:
> I think you're making the mistake of thinking that the entire SQL
> string is UTF-16.
It is. That's the whole point of the "16" interfaces.
> Look at the API for sqlite3_prepare16_v2
> SQLITE_API int sqlite3_prepare16_v2(
> sqlite3 *db, /* Database handle. */
> const void *zSql, /* UTF-8 encoded SQL statement. */
Yes, I know it says that in the source next to the function, but
that's wrong. Following the source confirms that. A set of
copy-paste errors, no doubt. The website (and the prototypes in the
source) are correct:
http://www.sqlite.org/c3ref/prepare.html
int sqlite3_prepare16_v2(
sqlite3 *db,/* Database handle */
const void *zSql, /* SQL statement, UTF-16 encoded */
int nByte, /* Maximum length of zSql in bytes. */
sqlite3_stmt **ppStmt, /* OUT: Statement handle */
const void **pzTail /* OUT: Pointer to unused portion of zSql */
);
> It's your data fields that get encoded.
Not exactly. The database has a single encoding for all text values.
Any text values that are recorded in the database are converted to that
encoding.
Consider bind values. You can specify input values in whatever
encoding you want. SQLite will convert them to whatever the
database encoding is before recording them. It works the same on
output, where you can request a column in whatever encoding you want,
regardless of the database encoding.
sqlite3_open() and sqlite3_open16() are a bit different. In addition
to the filename, using sqlite3_open16() to create a database will set
the global database encoding to UTF-16. But that has a whole different
set of issues, which is why there is no sqlite3_open16_v2().
-j
--
Jay A. Kreibich < J A Y @ K R E I B I.C H >
"Intelligence is like underwear: it is important that you have it,
but showing it to the wrong people has the tendency to make them
feel uncomfortable." -- Angela Johnson
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF16 - sqlite3_prepare16_v2
That was it. I didn't know that the wchar_t has different bit sizes on Linux and Windows. Thank you very much. Jay A. Kreibich-2 wrote: > >> I know that the SQL syntax is correct and it is possible to execute that >> source under windows. >> I don't understand why it doesn't work with Linux. >> Does somebody has an idea what's going wrong? > > Wide characters are 32 bits on most non-Windows platforms. When > passed to a 16 bit system, the first character of L"CREATE..." is > seen as ['C', '\0']. > > http://en.wikipedia.org/wiki/C_syntax#Wide_character_strings > > This is part of the reason SQLite3 uses void* rather than wchar_t* > for UTF-16 strings. > -- View this message in context: http://old.nabble.com/UTF16---sqlite3_prepare16_v2-tp28643020p28644005.html Sent from the SQLite mailing list archive at Nabble.com. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF16 - sqlite3_prepare16_v2
If i take a look at the declaration of sqlite3_prepare16_v2() it looks like this: SQLITE_API int sqlite3_prepare16_v2( sqlite3 *db,/* Database handle */ const void *zSql, /* SQL statement, UTF-16 encoded */ There is a UTF-16 encoded string needed. Black, Michael (IS) wrote: > > I think you're making the mistake of thinking that the entire SQL string > is UTF-16. > > Look at the API for sqlite3_prepare16_v2 > SQLITE_API int sqlite3_prepare16_v2( > sqlite3 *db, /* Database handle. */ > const void *zSql, /* UTF-8 encoded SQL statement. */ > > It's your data fields that get encoded. > -- View this message in context: http://old.nabble.com/UTF16---sqlite3_prepare16_v2-tp28643020p28643888.html Sent from the SQLite mailing list archive at Nabble.com. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF16 - sqlite3_prepare16_v2
On Sat, May 22, 2010 at 09:23:06AM -0500, Black, Michael (IS) scratched on the wall: > I think you're making the mistake of thinking that the entire SQL > string is UTF-16. It is. That's the whole point of the "16" interfaces. > Look at the API for sqlite3_prepare16_v2 > SQLITE_API int sqlite3_prepare16_v2( > sqlite3 *db, /* Database handle. */ > const void *zSql, /* UTF-8 encoded SQL statement. */ Yes, I know it says that in the source next to the function, but that's wrong. Following the source confirms that. A set of copy-paste errors, no doubt. The website (and the prototypes in the source) are correct: http://www.sqlite.org/c3ref/prepare.html int sqlite3_prepare16_v2( sqlite3 *db,/* Database handle */ const void *zSql, /* SQL statement, UTF-16 encoded */ int nByte, /* Maximum length of zSql in bytes. */ sqlite3_stmt **ppStmt, /* OUT: Statement handle */ const void **pzTail /* OUT: Pointer to unused portion of zSql */ ); > It's your data fields that get encoded. Not exactly. The database has a single encoding for all text values. Any text values that are recorded in the database are converted to that encoding. Consider bind values. You can specify input values in whatever encoding you want. SQLite will convert them to whatever the database encoding is before recording them. It works the same on output, where you can request a column in whatever encoding you want, regardless of the database encoding. sqlite3_open() and sqlite3_open16() are a bit different. In addition to the filename, using sqlite3_open16() to create a database will set the global database encoding to UTF-16. But that has a whole different set of issues, which is why there is no sqlite3_open16_v2(). -j -- Jay A. Kreibich < J A Y @ K R E I B I.C H > "Intelligence is like underwear: it is important that you have it, but showing it to the wrong people has the tendency to make them feel uncomfortable." -- Angela Johnson ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF16 - sqlite3_prepare16_v2
I think you're making the mistake of thinking that the entire SQL string is
UTF-16.
Look at the API for sqlite3_prepare16_v2
SQLITE_API int sqlite3_prepare16_v2(
sqlite3 *db, /* Database handle. */
const void *zSql, /* UTF-8 encoded SQL statement. */
It's your data fields that get encoded.
I'm not a UTF-16 expert by any means but that's how it appears to me.
I'm surprised it works in windows -- I did confirm your error on Linux.
Michael D. Black
Senior Scientist
Northrop Grumman Mission Systems
From: sqlite-users-boun...@sqlite.org on behalf of shion
Sent: Sat 5/22/2010 7:27 AM
To: sqlite-users@sqlite.org
Subject: [sqlite] UTF16 - sqlite3_prepare16_v2
Hello,
I already found a lot of threads about the UTF16 topic, but I couldn't solve
my problem.
I have the following source code:
sqlite3_open_v2("test.sqlite", , SQLITE_OPEN_READWRITE |
SQLITE_OPEN_CREATE, 0);
sqlite3_exec(Database, "PRAGMA encoding = \"UTF-16\"", 0, 0, 0);
sqlite3_prepare16_v2(Database, L"CREATE TABLE user (userID INTEGER NOT NULL
PRIMARY KEY, lastName VARCHAR(50), firstName VARCHAR(50));", -1,
, 0);
std::cout << sqlite3_errmsg(Database) << std::endl;
The result is:
near "C": syntax error
I know that the SQL syntax is correct and it is possible to execute that
source under windows.
I don't understand why it doesn't work with Linux.
Does somebody has an idea what's going wrong?
Operating system: 2.6.31-19-generic #56-Ubuntu SMP i686 GNU/Linux
IDE: NetBeans
Regards,
shion
--
View this message in context:
http://old.nabble.com/UTF16---sqlite3_prepare16_v2-tp28643020p28643020.html
Sent from the SQLite mailing list archive at Nabble.com.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] UTF16 - sqlite3_prepare16_v2
On Sat, May 22, 2010 at 05:27:24AM -0700, shion scratched on the wall:
>
> Hello,
>
> I already found a lot of threads about the UTF16 topic, but I couldn't solve
> my problem.
> I have the following source code:
>
> sqlite3_open_v2("test.sqlite", , SQLITE_OPEN_READWRITE |
> SQLITE_OPEN_CREATE, 0);
> sqlite3_exec(Database, "PRAGMA encoding = \"UTF-16\"", 0, 0, 0);
> sqlite3_prepare16_v2(Database, L"CREATE TABLE user (userID INTEGER NOT NULL
> PRIMARY KEY, lastName VARCHAR(50), firstName VARCHAR(50));", -1,
> , 0);
> std::cout << sqlite3_errmsg(Database) << std::endl;
>
> The result is:
> near "C": syntax error
Compile or runtime? Runtime, I assume.
> I know that the SQL syntax is correct and it is possible to execute that
> source under windows.
> I don't understand why it doesn't work with Linux.
> Does somebody has an idea what's going wrong?
Wide characters are 32 bits on most non-Windows platforms. When
passed to a 16 bit system, the first character of L"CREATE..." is
seen as ['C', '\0'].
http://en.wikipedia.org/wiki/C_syntax#Wide_character_strings
This is part of the reason SQLite3 uses void* rather than wchar_t*
for UTF-16 strings.
-j
--
Jay A. Kreibich < J A Y @ K R E I B I.C H >
"Intelligence is like underwear: it is important that you have it,
but showing it to the wrong people has the tendency to make them
feel uncomfortable." -- Angela Johnson
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] UTF16 - sqlite3_prepare16_v2
Hello,
I already found a lot of threads about the UTF16 topic, but I couldn't solve
my problem.
I have the following source code:
sqlite3_open_v2("test.sqlite", , SQLITE_OPEN_READWRITE |
SQLITE_OPEN_CREATE, 0);
sqlite3_exec(Database, "PRAGMA encoding = \"UTF-16\"", 0, 0, 0);
sqlite3_prepare16_v2(Database, L"CREATE TABLE user (userID INTEGER NOT NULL
PRIMARY KEY, lastName VARCHAR(50), firstName VARCHAR(50));", -1,
, 0);
std::cout << sqlite3_errmsg(Database) << std::endl;
The result is:
near "C": syntax error
I know that the SQL syntax is correct and it is possible to execute that
source under windows.
I don't understand why it doesn't work with Linux.
Does somebody has an idea what's going wrong?
Operating system: 2.6.31-19-generic #56-Ubuntu SMP i686 GNU/Linux
IDE: NetBeans
Regards,
shion
--
View this message in context:
http://old.nabble.com/UTF16---sqlite3_prepare16_v2-tp28643020p28643020.html
Sent from the SQLite mailing list archive at Nabble.com.
___
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
