Re: [sqlite] Using sqlite3_open or sqlite3_open16?
Windows (NT, 2000, XP etc..) unicode strings are in UCS-2 (!= UTF-16) You should be calling SHGetSpecialFolderPathW (note the W at the end) with a 'wide' buffer for the "My Documents" directory, before converting this to UTF-8 and passing it to sqlite_open(). HTH. On 24/04/06, COS <[EMAIL PROTECTED]> wrote: > Thanks for the clarification. This information got me to the right > direction. I have found the "MultiByteToWideChar" function. It does what I > need. But instead of making conversions I decided to use the "IsTextUnicode" > function (I didn't know that one either) which provides far more information > about the string. If it returns true then I can safely (I suppose) use the > sqlite3_open16 to open the database. In my tests it worked perfectly, but I > guess the best way to really test this is asking some of our users to do so. > ;-) > > Thanks a lot for your time and help on this. You really helped me a lot. > > Best Regards, > > COS > >
Re: [sqlite] Using sqlite3_open or sqlite3_open16?
Hi Jens, > If you're coding against the 'standard' Win32 APIs, Windows will send > you the strings in the character encoding specified by the either > the System or the current user settings. This has nothing to do with > unicode. However, for many roman characters, the standard Windows > code pages and UTF-8 have a fair amount of overlap, so you can get > away just passing those strings to sqlite3_open() for ASCII names and > _some_ localized names, but in general, this won't work if the > filename is not pure ASCII. > > If you can't use the Win32 unicode APIs (in which case you'd call > sqlite3_open16), you'll need to make sure you convert from the > Windows encoding to UTF-8 before passing the filenames. > Unfortunately, I don't know enough details about the Windows APIs to > tell you exactly how to do this :( You'll have to look up on MSDN - > searching for "MultiByteToWideChar" might get you started... > (I'm lucky enough to code on Mac OS X, where filenames are generally > passed in UTF-8 ;-) Thanks for the clarification. This information got me to the right direction. I have found the "MultiByteToWideChar" function. It does what I need. But instead of making conversions I decided to use the "IsTextUnicode" function (I didn't know that one either) which provides far more information about the string. If it returns true then I can safely (I suppose) use the sqlite3_open16 to open the database. In my tests it worked perfectly, but I guess the best way to really test this is asking some of our users to do so. ;-) Thanks a lot for your time and help on this. You really helped me a lot. Best Regards, COS
Re: [sqlite] Using sqlite3_open or sqlite3_open16?
Am 24.04.2006 um 14:51 schrieb COS: Hi Jens, - Original Message - From: "Jens Miltner" <[EMAIL PROTECTED]> To: <sqlite-users@sqlite.org> Sent: Monday, April 24, 2006 5:30 AM Subject: Re: [sqlite] Using sqlite3_open or sqlite3_open16? Am 22.04.2006 um 15:48 schrieb COS: Thanks for the info. I did have found in the manual that UTF16 is converted to UTF8 on Windows environments. But one little question: How would one know if the filename is in UTF8 or UTF16? You see, my little application is installed in PC's all around the world. I don't know in advance if the filename is UTF8 or UTF16. I noticed that always using UTF16 to open the database can lead to an exception in filenames with UTF8. Hmmh - I'm not sure I entirely understand your problem: usually strings passed in UTF-16 are passed in 'wide' strings, i.e. each character is 16 bit as opposed to UTF-8 strings, which are usually passed in strings where each character is 8 bit. I don't know the Windows APIs very well, but from what I remember, you build your app either as Unicode app or as "ASCII" app and only unicode apps get their filenames as UTF-16, "ASCII" apps get whatever code page is installed/activated on the system, usually not UTF-8, though... My application is not built using Unicode. But I use a lot of MFC and CString is aware of International characters. The problem is a little different. I don't need to use UTF16 strings at all. The problem comes up when the application is started and it needs to open the database from the My documents folder. In Windows NT/2000/XP and probably others too, the userid is a string to compose the personal folder location (i.e. C:\Documents and Settings\userid\My Documents). As you can see this location can change a lot. It may even work for a particular user and don't for another. Where do you get your filenames from so you don't know the encoding? I get this information directly from Windows, using the Win32 SDK API. The string is correct, even for UTF16 strings. The Windows SDK API allows to open the file directly without problems or any conversion (maybe some conversion is done internally by the Win32 SDK). If you're coding against the 'standard' Win32 APIs, Windows will send you the strings in the character encoding specified by the either the System or the current user settings. This has nothing to do with unicode. However, for many roman characters, the standard Windows code pages and UTF-8 have a fair amount of overlap, so you can get away just passing those strings to sqlite3_open() for ASCII names and _some_ localized names, but in general, this won't work if the filename is not pure ASCII. If you can't use the Win32 unicode APIs (in which case you'd call sqlite3_open16), you'll need to make sure you convert from the Windows encoding to UTF-8 before passing the filenames. Unfortunately, I don't know enough details about the Windows APIs to tell you exactly how to do this :( You'll have to look up on MSDN - searching for "MultiByteToWideChar" might get you started... (I'm lucky enough to code on Mac OS X, where filenames are generally passed in UTF-8 ;-) HTH,
Re: [sqlite] Using sqlite3_open or sqlite3_open16?
> I tried to look for some information in the MSDN, without success. I'm > starting to look at SQlite code to find out how the conversion is made and > maybe I can simulate the same process before opening the file and choose the > proper way. If I remember correctly there are different subroutines in the windows system DLL's for the unicode versions of the functions. I think there was an extra underline character and a letter as a suffix or a prefix? I wish I could remember correctly... Good luck
Re: [sqlite] Using sqlite3_open or sqlite3_open16?
Hi Jens, - Original Message - From: "Jens Miltner" <[EMAIL PROTECTED]> To: <sqlite-users@sqlite.org> Sent: Monday, April 24, 2006 5:30 AM Subject: Re: [sqlite] Using sqlite3_open or sqlite3_open16? > > Am 22.04.2006 um 15:48 schrieb COS: > > Thanks for the info. I did have found in the manual that UTF16 is > > converted > > to UTF8 on Windows environments. > > But one little question: How would one know if the filename is in > > UTF8 or > > UTF16? > > You see, my little application is installed in PC's all around the > > world. I > > don't know in advance if the filename is UTF8 or UTF16. I noticed that > > always using UTF16 to open the database can lead to an exception in > > filenames with UTF8. > > Hmmh - I'm not sure I entirely understand your problem: usually > strings passed in UTF-16 are passed in 'wide' strings, i.e. each > character is 16 bit as opposed to UTF-8 strings, which are usually > passed in strings where each character is 8 bit. > I don't know the Windows APIs very well, but from what I remember, > you build your app either as Unicode app or as "ASCII" app and only > unicode apps get their filenames as UTF-16, "ASCII" apps get whatever > code page is installed/activated on the system, usually not UTF-8, > though... My application is not built using Unicode. But I use a lot of MFC and CString is aware of International characters. The problem is a little different. I don't need to use UTF16 strings at all. The problem comes up when the application is started and it needs to open the database from the My documents folder. In Windows NT/2000/XP and probably others too, the userid is a string to compose the personal folder location (i.e. C:\Documents and Settings\userid\My Documents). As you can see this location can change a lot. It may even work for a particular user and don't for another. > Where do you get your filenames from so you don't know the encoding? I get this information directly from Windows, using the Win32 SDK API. The string is correct, even for UTF16 strings. The Windows SDK API allows to open the file directly without problems or any conversion (maybe some conversion is done internally by the Win32 SDK). > As a rule of thumb: anything you get as a plain character pointer > probably isn't UTF-16 (unless there's some badly designed API in the > middle). OTOH, you can't assume anything that's plain character > pointer is UTF-8 - most likely on Windows it's something different. > > I tried to look for some information in the MSDN, without success. I'm starting to look at SQlite code to find out how the conversion is made and maybe I can simulate the same process before opening the file and choose the proper way. Thanks for your time on this. Best Regards, COS
Re: [sqlite] Using sqlite3_open or sqlite3_open16?
Am 22.04.2006 um 15:48 schrieb COS: Thanks for the info. I did have found in the manual that UTF16 is converted to UTF8 on Windows environments. But one little question: How would one know if the filename is in UTF8 or UTF16? You see, my little application is installed in PC's all around the world. I don't know in advance if the filename is UTF8 or UTF16. I noticed that always using UTF16 to open the database can lead to an exception in filenames with UTF8. Hmmh - I'm not sure I entirely understand your problem: usually strings passed in UTF-16 are passed in 'wide' strings, i.e. each character is 16 bit as opposed to UTF-8 strings, which are usually passed in strings where each character is 8 bit. I don't know the Windows APIs very well, but from what I remember, you build your app either as Unicode app or as "ASCII" app and only unicode apps get their filenames as UTF-16, "ASCII" apps get whatever code page is installed/activated on the system, usually not UTF-8, though... Where do you get your filenames from so you don't know the encoding? As a rule of thumb: anything you get as a plain character pointer probably isn't UTF-16 (unless there's some badly designed API in the middle). OTOH, you can't assume anything that's plain character pointer is UTF-8 - most likely on Windows it's something different.
Re: [sqlite] Using sqlite3_open or sqlite3_open16?
On Thu, 20 Apr 2006, DBTools Software wrote: >Hi, > >I have an application that needs to open a database in the users's personal >folder. I noticed that in some circunstances the sqlite3_open fail as the >filename is in UTF16 format. I don't know that in advance so I could open >the db with sqlite3_open16. > >The question is: > >Is it safe to always use the UTF16 functions independent of the >localization? What I mean is can I use the functions sqlite3_xxx16 instead >of sqlite3_xxx for all cases? Your data should always be in UTF-16 to use sqlite3_xxx16, else you'll incur the cost of transforming your date to/from UTF-8. If most of your data is UTF-8 or ASCII, use the regular UTF-8 functions. If your filename is UTF-16, just convert it to UTF-8 (it's a relatively simple, unambiguous transformation) and use sqlite3_open. > >Thanks, > >COS > > -- /"\ \ /ASCII RIBBON CAMPAIGN - AGAINST HTML MAIL X - AGAINST MS ATTACHMENTS / \
[sqlite] Using sqlite3_open or sqlite3_open16?
Hi, I have an application that needs to open a database in the users's personal folder. I noticed that in some circunstances the sqlite3_open fail as the filename is in UTF16 format. I don't know that in advance so I could open the db with sqlite3_open16. The question is: Is it safe to always use the UTF16 functions independent of the localization? What I mean is can I use the functions sqlite3_xxx16 instead of sqlite3_xxx for all cases? Thanks, COS
