Re: [sqlite] Unicode quote characters parsing in FTS5 queries

2020-02-03 Thread Simon Slavin
On 3 Feb 2020, at 9:30pm, David Guillen Fandos wrote: > However my queries where choking on some input that contained the U+2018 and > U+2019 characters. It seems like these chars are being treated like a regular > quote character At least one version of SQL respects those characters for quoti

Re: [sqlite] unicode case insensitive

2014-10-24 Thread dd
typo: *I am summarizing options to support unicode* case-insensitive*: On Sat, Oct 25, 2014 at 10:34 AM, dd wrote: > I am summarizing options to support unicode case-sensitive: > > 1. Richard Hipp: icu ext > 2. Aleksey Tulinov: https://bitbucket.org/alekseyt/nunicode#markdown- > header-sqlite3-e

Re: [sqlite] unicode case insensitive

2014-10-24 Thread dd
I am summarizing options to support unicode case-sensitive: 1. Richard Hipp: icu ext 2. Aleksey Tulinov: https://bitbucket.org/alekseyt/nunicode#markdown- header-sqlite3-extension 3. Grey's suggestion: custom collation Please add in options list if I missed. I don't have experience in

Re: [sqlite] unicode case insensitive

2014-10-24 Thread James K. Lowden
On Fri, 24 Oct 2014 21:44:50 +0400 dd wrote: > >>Convert everything to upper (or lower) case brute force. >Sorry. I am not clear. Can you please elaborate this. The standard function tolower(3) is locale-dependent. If your locale is set to match the data's single-byte encoding,

Re: [sqlite] unicode case insensitive

2014-10-24 Thread Gerry Snyder
By "brute force" I just meant specifying each conversion (such as Ö to ö) individually. In my Tcl code, it is done with a single [string map ...] statement containing all of the conversions. The down side being, as I mentioned earlier, that each time I run it on a new set of data I have to chec

Re: [sqlite] unicode case insensitive

2014-10-24 Thread Aleksey Tulinov
On 24/10/14 20:44, dd wrote: dd, Any sample/open source avail to custom collation. Will it work for like queries. Any performance degradation? You might try nunicode: https://bitbucket.org/alekseyt/nunicode#markdown-header-sqlite3-extension. I think COLLATE NU700_NOCASE should do what y

Re: [sqlite] unicode case insensitive

2014-10-24 Thread Richard Hipp
On Fri, Oct 24, 2014 at 1:44 PM, dd wrote: > Hi, > > Any sample/open source avail to custom collation. http://www.sqlite.org/compile.html#enable_icu > Will it work for like > queries. Yes > Any performance degradation? > Yes. Such is the price of unicode. -- D. Richard Hipp d...@sq

Re: [sqlite] unicode case insensitive

2014-10-24 Thread dd
Hi, Any sample/open source avail to custom collation. Will it work for like queries. Any performance degradation? >>Convert everything to upper (or lower) case brute force. Sorry. I am not clear. Can you please elaborate this. Thanks. On Fri, Oct 24, 2014 at 9:16 PM, Gerry Snyd

Re: [sqlite] unicode case insensitive

2014-10-24 Thread Gerry Snyder
In a vaguely similar situation I wrote a custom collation that converted accented letters to their non-accented cousins. Since the conversion is on a case-by-case basis I also had to do a pre-screening that would show any non-ascii characters that I wasn't converting, so that I could add them t

Re: [sqlite] Unicode support in SQLite

2014-10-14 Thread Aleksey Tulinov
On 14/10/14 17:02, Kevin Benson wrote: https://bitbucket.org/alekseyt/nunicode/downloads/libnusqlite3-1.4-4a0e4773-win32.zip <--- 404 response code Thank you, fixed now. ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/c

Re: [sqlite] Unicode support in SQLite

2014-10-14 Thread Kevin Benson
On Tue, Oct 14, 2014 at 4:37 AM, Aleksey Tulinov wrote: > Hello, > > I'm glad to announce that nunicode SQLite extension was updated to support > Unicode-conformant case folding and was improved on performance of every > component provided to SQLite. > > You can read about and download this exten

Re: [sqlite] Unicode support in SQLite

2014-07-04 Thread Aleksey Tulinov
Hello, I'm glad to announce that nunicode SQLite extension was updated to support Unicode 7.0.0 character set. It also implements LIKE operation which is faster compared to previous releases. This extension provides the following Unicode-aware components: - upper(X) - lower(X) - X LIKE Y ESC

Re: [sqlite] Unicode support in SQLite

2014-04-02 Thread Aleksey Tulinov
Hey, According to previous discussion in this mailing list, i've updated nunicode SQLite extension not to override default NOCASE collation due to possible issues with database indexing. Version 1.2.1 removes nunicode-specific NOCASE and NUNICODE collations and introduces NU630 and NU630_NOC

Re: [sqlite] Unicode support in SQLite

2013-11-10 Thread Gert Van Assche
Very nice! Thanks for sharing, Aleksey. 2013/11/9 Aleksey Tulinov > On 11/04/2013 11:50 AM, Aleksey Tulinov wrote: > > Hey, > > > As you can see, this is truly full Unicode collation and case mapping >> with untailored special casing. Extension provides the following functions, >> statements a

Re: [sqlite] Unicode support in SQLite

2013-11-09 Thread Aleksey Tulinov
On 11/04/2013 11:50 AM, Aleksey Tulinov wrote: Hey, As you can see, this is truly full Unicode collation and case mapping with untailored special casing. Extension provides the following functions, statements and collations: I've updated extension, examples and documentation, now it's easier

Re: [sqlite] unicode() and char() functions does not exists

2013-06-27 Thread Richard Hipp
On Thu, Jun 27, 2013 at 4:19 AM, Stefano Zaglio wrote: > Hi, > I'm tring this: > > select unicode('2') as a; > > and > > select char(50) > > but firefox.sqlite_manager (sqlite 3.7) and others say:"no such function: > unicode". > > Where I'm wrong? > Those functions where added to SQLite

Re: [sqlite] Unicode problem when setting PRAGMA journal_mode

2012-04-11 Thread Nick Shaw
-Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Nick Shaw Sent: 11 April 2012 16:29 To: General Discussion of SQLite Database Subject: Re: [sqlite] Unicode problem when setting PRAGMA journal_mode -Original Message

Re: [sqlite] Unicode problem when setting PRAGMA journal_mode

2012-04-11 Thread Nick Shaw
-Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Dan Kennedy Sent: 11 April 2012 16:07 To: sqlite-users@sqlite.org Subject: Re: [sqlite] Unicode problem when setting PRAGMA journal_mode > After sqlite3_close() retu

Re: [sqlite] Unicode problem when setting PRAGMA journal_mode

2012-04-11 Thread Nick Shaw
-Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Dan Kennedy Sent: 11 April 2012 16:07 To: sqlite-users@sqlite.org Subject: Re: [sqlite] Unicode problem when setting PRAGMA journal_mode > > I'll try your suggestion

Re: [sqlite] Unicode problem when setting PRAGMA journal_mode

2012-04-11 Thread Dan Kennedy
On 04/11/2012 09:50 PM, Nick Shaw wrote: -Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Simon Slavin Sent: 10 April 2012 19:04 To: General Discussion of SQLite Database Subject: Re: [sqlite] Unicode problem when setting

Re: [sqlite] Unicode problem when setting PRAGMA journal_mode

2012-04-11 Thread Nick Shaw
-Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Simon Slavin Sent: 10 April 2012 19:04 To: General Discussion of SQLite Database Subject: Re: [sqlite] Unicode problem when setting PRAGMA journal_mode >> Realised I made

Re: [sqlite] Unicode problem when setting PRAGMA journal_mode

2012-04-10 Thread Simon Slavin
On 10 Apr 2012, at 1:42pm, Nick Shaw wrote: > Realised I made a typo below: should have said "PRAGMA journal_mode = DELETE" > (though setting it to WAL or OFF causes the same problem). Are you by any chance having a technical problem with the PRAGMA command itself ? For instance, suppose the

Re: [sqlite] Unicode problem when setting PRAGMA journal_mode

2012-04-10 Thread Nick Shaw
Realised I made a typo below: should have said "PRAGMA journal_mode = DELETE" (though setting it to WAL or OFF causes the same problem). -Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Nick Shaw Sent: 05 April 2012 13:34 To:

Re: [sqlite] Unicode Confusion and Database Size

2011-09-04 Thread Mohit Sindhwani
Hi Simon, On 4/9/2011 12:01 AM, Simon Slavin wrote: Have you tried speed tests on your platform ? It's hard to tell which will be faster because it depends on what language and OS you're using that interacts with SQLite. So if you have your schema designed and any part of your application writ

Re: [sqlite] Unicode Confusion and Database Size

2011-09-03 Thread Simon Slavin
On 3 Sep 2011, at 4:27pm, Mohit Sindhwani wrote: > for our data, we can get savings in the region of 25% - 33% in the case of > strings being stored in a language that does require 3bytes/ character. So, > given that, we should explore UTF-16 in more detail. However, we also have a > lot of

Re: [sqlite] Unicode Confusion and Database Size

2011-09-03 Thread Mohit Sindhwani
Hi Igor, Thanks for your advice and guidance. On 1/9/2011 11:57 PM, Igor Tandetnik wrote: On 9/1/2011 10:24 AM, Mohit Sindhwani wrote: On the other hand, the other language that we are storing seems to require 3 bytes in UTF-8. Given that, it would appear that using UTF-8 would be a better id

Re: [sqlite] Unicode Confusion and Database Size

2011-09-01 Thread Mohit Sindhwani
Hi Igor, On 1/9/2011 11:57 PM, Igor Tandetnik wrote: On 9/1/2011 10:24 AM, Mohit Sindhwani wrote: I understand that the database could be either UTF-8 or UTF-16 - but that would apply to the full DB not to a single column, right? Right. *many useful answers snipped* Thank you very much!!

Re: [sqlite] Unicode Confusion and Database Size

2011-09-01 Thread Igor Tandetnik
On 9/1/2011 10:24 AM, Mohit Sindhwani wrote: I understand that the database could be either UTF-8 or UTF-16 - but that would apply to the full DB not to a single column, right? Right. If that is the case, would it not make the database larger if we had a lot of content that was originally ASC

Re: [sqlite] Unicode function for collapsing upper/lower case

2011-08-07 Thread Igor Tandetnik
Simon Slavin wrote: > But I wondered > whether there were patterns in the unicode which made it simple to ignore > string case. http://www.sqlite.org/cvstrac/fileview?f=sqlite/ext/icu/README.txt -- Igor Tandetnik ___ sqlite-users mailing list sqlite

Re: [sqlite] Unicode command line bug in Windows version of sqlite3 with patch

2010-03-16 Thread Roger Binns
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jean-Christophe Deschamps wrote: > A much better solution is to use a MSYS terminal (installed by MinGW), > so you have UTF-8 command-line and data entry/display without > conversion. No need to "patch" anything. No need for msys. You can make a r

Re: [sqlite] Unicode command line bug in Windows version of sqlite3 with patch

2010-03-16 Thread Teg
Hello Jean-Christophe, JCD> If you change input encoding and use your code page, then it's likely JCD> you'll going to do the same with data, which is plain wrong: SQLite JCD> needs UTF-8 (or UTF-16) data, not ANSI. Yeah, I solved this long ago by simply insuring I always used UTF-8 encoding on

Re: [sqlite] Unicode command line bug in Windows version of sqlite3 with patch

2010-03-16 Thread Jean-Christophe Deschamps
>We currently use sqlite 3.6.23. We have a big problem with characters with >accents or other special characters in path to database file, for >example in >Czech Windows XP the "Application Data" folder is translated to "Data >aplikací" so if the accented 'í' is in path the sqlite3.exe writes tha

Re: [sqlite] Unicode command line bug in Windows version of sqlite3 with patch

2010-03-16 Thread Pavel Ivanov
sqlite3_open[_v2] accepts all filenames in UTF-8 (although it doesn't check for valid UTF-8 string). So CP_UTF8 cannot be changed anywhere. OTOH maybe command line utility should have some logic of re-encoding of command line parameter from terminal encoding to UTF-8. But I'm not sure about that.

Re: [sqlite] Unicode support

2009-11-20 Thread Nicolas Williams
On Tue, Nov 17, 2009 at 09:31:46PM -0500, Tim Romano wrote: > but if ORDER BY is > relying on an index for ordering, then flip() can have negative > effects. > > > Substr() could have negative effects on ordering too. That is a red > herring. Flip() is merely a function that reverses the or

Re: [sqlite] Unicode support

2009-11-17 Thread Tim Romano
but if ORDER BY is relying on an index for ordering, then flip() can have negative effects. Substr() could have negative effects on ordering too. That is a red herring. Flip() is merely a function that reverses the order of codepoints "as found" without knowing anything about what those co

Re: [sqlite] Unicode support

2009-11-17 Thread Jean-Christophe Deschamps
Tim, >For those who are insisting on Unicode graphemic codepoint-combination >intelligence: why can't we have a function that simply reverses the >order of the codepoints, and is blissfully ignorant about what those >individual codepoints or codepoint-combinations might signify as >graphemes in

Re: [sqlite] Unicode support

2009-11-17 Thread Tim Romano
For those who are insisting on Unicode graphemic codepoint-combination intelligence: why can't we have a function that simply reverses the order of the codepoints, and is blissfully ignorant about what those individual codepoints or codepoint-combinations might signify as graphemes in a writin

Re: [sqlite] Unicode support

2009-11-17 Thread Nicolas Williams
On Tue, Nov 17, 2009 at 05:15:16PM -0500, Igor Tandetnik wrote: > Nicolas Williams wrote: > > This is no longer true, either of 'ch' nor 'll'. > > There is a number of contractions in Hungarian that are still very > much in use, but I can't recall them off the top of my head the way I > can 'ch'

Re: [sqlite] Unicode support

2009-11-17 Thread Simon Slavin
On 17 Nov 2009, at 10:05pm, Beau Wilkinson wrote: > I think a better approach (to the design of Unicode) would have been for > Spanish and German (for instance) to share absolutely nothing in the encoding > standards. Each language ought to have its own little span of letters, > immortalized i

Re: [sqlite] Unicode support

2009-11-17 Thread Beau Wilkinson
esday, November 17, 2009 1:01 PM To: sqlite-users@sqlite.org Subject: Re: [sqlite] Unicode support Simon Slavin wrote: > On 17 Nov 2009, at 6:37pm, Igor Tandetnik wrote: > >> Simon Slavin wrote: >>> First split the string into characters, then reassemble them in >>> rever

Re: [sqlite] Unicode support

2009-11-17 Thread Igor Tandetnik
Nicolas Williams wrote: > On Tue, Nov 17, 2009 at 02:01:55PM -0500, Igor Tandetnik wrote: >> This would mean that the result of the hypothetical flip() function >> would be locale-dependent. E.g. in Spanish Traditional sort, a >> combination 'ch' sorts as if it were a single letter between 'c' and

Re: [sqlite] Unicode support

2009-11-17 Thread Beau Wilkinson
fuse to implement things and still look smart in so refusing. I suggest that this a very detrimental pattern, though. ____ From: sqlite-users-boun...@sqlite.org [sqlite-users-boun...@sqlite.org] On Behalf Of Igor Tandetnik [itandet...@mvps.org] Sent: Tuesda

Re: [sqlite] Unicode support

2009-11-17 Thread Nicolas Williams
On Tue, Nov 17, 2009 at 02:01:55PM -0500, Igor Tandetnik wrote: > This would mean that the result of the hypothetical flip() function > would be locale-dependent. E.g. in Spanish Traditional sort, a > combination 'ch' sorts as if it were a single letter between 'c' and > 'd', forming a single sort

Re: [sqlite] Unicode support

2009-11-17 Thread Igor Tandetnik
Simon Slavin wrote: > On 17 Nov 2009, at 6:37pm, Igor Tandetnik wrote: > >> Simon Slavin wrote: >>> First split the string into characters, then reassemble them in >>> reverse order. >> >> The problem is, in Unicode it's not quite clear what constitutes a >> "character". Are we talking about co

Re: [sqlite] Unicode searches

2008-04-07 Thread Jay A. Kreibich
> > On Sat, Apr 5, 2008 at 11:42 PM, Cory Nelson <[EMAIL PROTECTED]> wrote: > > > Sort order is highly dependent on locale. You can add custom > > > collations to do this. > <[EMAIL PROTECTED]> wrote: > > That was not was I was talking about. I was not talking about Sort Order but > > about

Re: [sqlite] Unicode searches

2008-04-07 Thread Keith Stemmer
Cory, sorry, I had a bad day. Keith ___ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

Re: [sqlite] Unicode searches

2008-04-07 Thread Miha Vrhovnik
Someone sent a sqlite3_unicode.c file to this mailing list in the last week of December, 1st week of January which implemented upper/lower and some other functions. File was released as public domain if I remember correctly and used data from Unicode 5.1 standard. As ICU brings a lot of bulk i

Re: [sqlite] Unicode searches

2008-04-06 Thread Dan
On Apr 6, 2008, at 5:10 AM, Keith Stemmer wrote: > Yes, I can add a custom collation which works for ASCII chars LOL. > If you don't understand the problem, just don't reply. > > By the way, you can read on the SQLite website that the developer > describes > my problem as a BUG which is nice to

Re: [sqlite] Unicode searches

2008-04-05 Thread Gunnar Roth
Keith Stemmer schrieb: > Yes, I can add a custom collation which works for ASCII chars LOL. > Plain wrong > If you don't understand the problem, just don't reply. > > Plain unreasonable carefulle read ( and understand) this http://sqlite.org/c3ref/create_collation.html and this http://www

Re: [sqlite] Unicode searches

2008-04-05 Thread Keith Stemmer
Yes, I can add a custom collation which works for ASCII chars LOL. If you don't understand the problem, just don't reply. By the way, you can read on the SQLite website that the developer describes my problem as a BUG which is nice to read. At least he doesn't call it a feature. Keith. > > Sort

Re: [sqlite] Unicode searches

2008-04-05 Thread Cory Nelson
They are one and the same. Look up collations. On Sat, Apr 5, 2008 at 2:55 PM, Keith Stemmer <[EMAIL PROTECTED]> wrote: > That was not was I was talking about. I was not talking about Sort Order but > about Searches. > Keith > > > > On Sat, Apr 5, 2008 at 11:42 PM, Cory Nelson <[EMAIL PROTECTE

Re: [sqlite] Unicode searches

2008-04-05 Thread Keith Stemmer
That was not was I was talking about. I was not talking about Sort Order but about Searches. Keith On Sat, Apr 5, 2008 at 11:42 PM, Cory Nelson <[EMAIL PROTECTED]> wrote: > Sort order is highly dependent on locale. You can add custom > collations to do this. > > On Sat, Apr 5, 2008 at 10:41 AM,

Re: [sqlite] Unicode searches

2008-04-05 Thread Cory Nelson
Sort order is highly dependent on locale. You can add custom collations to do this. On Sat, Apr 5, 2008 at 10:41 AM, Keith Stemmer <[EMAIL PROTECTED]> wrote: > Hello! > > I found SQLite quite amazing, but I think there is one showstopper for me. > It seems that searches for Unicode strings are

RE: [sqlite] Unicode support for Sqlite?

2007-12-12 Thread Sreedhar.a
Thankyou all for the quick replies. Best Regards, A.Sreedhar. -Original Message- From: Trevor Talbot [mailto:[EMAIL PROTECTED] Sent: Wednesday, December 12, 2007 5:08 PM To: sqlite-users@sqlite.org Subject: Re: [sqlite] Unicode support for Sqlite? On 12/12/07, Sreedhar.a <[EM

Re: [sqlite] Unicode support for Sqlite?

2007-12-12 Thread Trevor Talbot
On 12/12/07, Sreedhar.a <[EMAIL PROTECTED]> wrote: > I am using the sqlite to store the metadata of audio files. > Is it possible to store the metadata in unicode character format in sqlite. Yes; SQLite assumes all TEXT type data in the database is Unicode. You can work with it in UTF-8 with the

Re: [sqlite] Unicode support for Sqlite?

2007-12-12 Thread Daniel Önnerby
utf-8 and utf-16 ARE unicode formats. But there are some things that sqlite does not handle without the ICU extension. The ICU extension extends SQLite with the following functionallity: 1.1 SQL Scalars upper() and lower() 1.2 Unicode Aware LIKE Operator 1.3 ICU Collation

RE: [sqlite] Unicode support for Sqlite?

2007-12-12 Thread Sreedhar.a
: sqlite-users@sqlite.org Subject: Re: [sqlite] Unicode support for Sqlite? On 12/12/07, Sreedhar.a <[EMAIL PROTECTED]> wrote: > Does Sqlite support unicode? > I have seen that it supports utf-8 and utf-16. > I want to know whether it supports unicode character formats. Unicode is a

Re: [sqlite] Unicode support for Sqlite?

2007-12-12 Thread Trevor Talbot
On 12/12/07, Sreedhar.a <[EMAIL PROTECTED]> wrote: > Does Sqlite support unicode? > I have seen that it supports utf-8 and utf-16. > I want to know whether it supports unicode character formats. Unicode is a very large and complex topic, so that question is way too vague to answer. Can you provid

Re: [sqlite] Unicode Capabilities in Simple Terms

2007-09-14 Thread Cory Nelson
On 9/14/07, Asif Lodhi <[EMAIL PROTECTED]> wrote: > Hi, > > I've been just glancing over the Unicode related posts. However, I > would like to know how it's possible for me to insert text in > different languages in an Sqlite database. Can you fixate the "type" > of the language for any particular

Re: [sqlite] Unicode Again... Sti ll Stuck... A Challenge... Store and retrieve the word résumé with out using a unicode string literal

2007-07-31 Thread wcmadness
Folks: Thanks so much for your replies. I have absorbed a lot of information about code pages and unicode in the last couple of days. My understanding is far from complete, but I'm ahead of where I was... In the end, my best answer was to set the text_factory property of the connection object

Re: [sqlite] Unicode Again... Still Stuck... A Challenge... Store and retrieve the word résumé without using a unicode string literal

2007-07-31 Thread Nuno Lucas
First, let me start by saying I don't have much experience with Python, but this isn't a python problem. On 7/31/07, wcmadness <[EMAIL PROTECTED]> wrote: [...] > I'm on a Windows machine. It turns out that the default code page on > Windows is cp437. So, in my Python code, if I type: Wrong. The

Re: [sqlite] Unicode Again... Sti ll Stuck... A Challenge... Store and retrieve the word résumé with out using a unicode string literal

2007-07-30 Thread wcmadness
Well, I have a solution to my own problem, and I wanted to post it for two reasons: First, it might help someone; second, I'm wondering if someone can explain it to me... Here's the scoop... I'm on a Windows machine. It turns out that the default code page on Windows is cp437. So, in my Python

Re: [sqlite] Unicode

2007-07-30 Thread Srebrenko Sehic
On 7/30/07, wcmadness <[EMAIL PROTECTED]> wrote: > I'm stuck on this. I'm writing a data layer that potentially needs to handle > diacritical (sp?) characters, such a French accented é characters or German > umlauted characters (sp?). It should be rare that I would run into > something like this

Re: [sqlite] Unicode collation

2007-06-28 Thread Trevor Talbot
On 6/28/07, Nuno Lucas <[EMAIL PROTECTED]> wrote: It seems more natural to use the standard C locale names to me (the usual "pt_PT" and "pt_BR" for Portuguese/Portugal culture and Portuguese/Brazil culture) , but I'm open to suggestions when that problem arise, and I'm sure there are already st

Re: [sqlite] Unicode collation

2007-06-28 Thread Nuno Lucas
On 6/28/07, Trevor Talbot <[EMAIL PROTECTED]> wrote: On 6/28/07, Nuno Lucas <[EMAIL PROTECTED]> wrote: > One thing I noticed is that "collations" != "case change". This will > not make it possible to use UPPER/LOWER with the same data on the > table, but maybe we can work on something in that re

Re: [sqlite] Unicode collation

2007-06-28 Thread Nuno Lucas
On 6/28/07, Jiri Hajek <[EMAIL PROTECTED]> wrote: > My idea is to implement the UCA collation in SQLite (with the usual > OMIT_* #ifdef's), using the DUCET table as base, and if people need > the tailoring part for localized sorting, have it be optional by > having a "sqlite_collation_data" table

Re: [sqlite] Unicode collation

2007-06-28 Thread Trevor Talbot
On 6/28/07, Jiri Hajek <[EMAIL PROTECTED]> wrote: Btw, even if this is implemented, there is still a need for a standardization such new collation names. E.g. that new language neutral collation could be called Unicode or DUCET? And how about language specific collations? After some thoughts, I'

Re: [sqlite] Unicode collation

2007-06-28 Thread Trevor Talbot
On 6/28/07, Nuno Lucas <[EMAIL PROTECTED]> wrote: One thing I noticed is that "collations" != "case change". This will not make it possible to use UPPER/LOWER with the same data on the table, but maybe we can work on something in that respect, also. If the intent is case-insensitive comparison

Re: [sqlite] Unicode collation

2007-06-28 Thread Jiri Hajek
My idea is to implement the UCA collation in SQLite (with the usual OMIT_* #ifdef's), using the DUCET table as base, and if people need the tailoring part for localized sorting, have it be optional by having a "sqlite_collation_data" table with the needed locale data included on the database. Th

Re: [sqlite] Unicode collation

2007-06-28 Thread Nuno Lucas
On 6/27/07, Jiri Hajek <[EMAIL PROTECTED]> wrote: > Trevor proposed method doesn't need that, because the collation data > goes with the database (which is the big advantage to me). And if you > want to use another collation, you only need to import the collation > data needed for that locale.

Re: [sqlite] Unicode collation

2007-06-27 Thread Jiri Hajek
> Actually, reading one of the links you posted > (http://blogs.msdn.com/michkap/archive/2005/05/04/414520.aspx - > everybody please read it before continuing in discussion), I got (what > I think is a great) idea: Let's include version information about > collation algorithm we are using. So, ext

Re: [sqlite] Unicode collation

2007-06-27 Thread Nuno Lucas
On 6/27/07, Jiri Hajek <[EMAIL PROTECTED]> wrote: Actually, reading one of the links you posted (http://blogs.msdn.com/michkap/archive/2005/05/04/414520.aspx - everybody please read it before continuing in discussion), I got (what I think is a great) idea: Let's include version information about

Re: [sqlite] Unicode collation

2007-06-27 Thread Jiri Hajek
After thinking a bit, it occurs to me that there's a compromise for the Unicode case that might be workable. The algorithm for collation is pretty stable, it's just the locale data that's the problem. If SQLite understands the algorithm, then locale data can go into special tables in the databas

Re: [sqlite] Unicode collation

2007-06-27 Thread Nuno Lucas
On 6/27/07, Trevor Talbot <[EMAIL PROTECTED]> wrote: On 6/25/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > > > So the choices seem to be: > > > > > > (1) Databases that corrupt if you move across platforms. > > > (2) A 10MB database engine > > > (3) Leave things as they are > OK. Here

Re: [sqlite] Unicode collation

2007-06-27 Thread Trevor Talbot
On 6/25/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > > So the choices seem to be: > > > > (1) Databases that corrupt if you move across platforms. > > (2) A 10MB database engine > > (3) Leave things as they are OK. Here is a crazy idea for consideration: You know that you can creat

Re: [sqlite] Unicode collation

2007-06-27 Thread Trevor Talbot
On 6/27/07, Jiri Hajek <[EMAIL PROTECTED]> wrote: I tried to think about this in as many details as possible and I'd say that the original problem that I'd like to solve here (i.e. collation of Unicode character sets) doesn't require any special complex handling in SQLite. We really should just

Re: [sqlite] Unicode collation

2007-06-27 Thread Jiri Hajek
A moments reflection convinces me that this is not a good idea as stated. But perhaps it can be the seed for a idea that will actually work. Is there some way of adding customizations to the database file itself, or perhaps to a separate file in a standard place the SQLite always knows to look,

Re: [sqlite] Unicode collation

2007-06-25 Thread Joe Wilson
--- [EMAIL PROTECTED] wrote: > OK. Here is a crazy idea for consideration: > > You know that you can create a custom collating function as a > DLL or shared library and load it at runtime, right? This has > been a capability since version 3.3.7. Suppose we define a > special table in the databa

RE: [sqlite] Unicode collation

2007-06-25 Thread James Dennett
> -Original Message- > From: Jiri Hajek [mailto:[EMAIL PROTECTED] > Sent: Monday, June 25, 2007 8:14 AM > To: sqlite-users@sqlite.org > Subject: Re: [sqlite] Unicode collation > > > So the choices seem to be: > > > > (1) Databases that corrupt if y

Re: [sqlite] Unicode collation

2007-06-25 Thread drh
"Jiri Hajek" <[EMAIL PROTECTED]> wrote: > > So the choices seem to be: > > > > (1) Databases that corrupt if you move across platforms. > > (2) A 10MB database engine > > (3) Leave things as they are > > > > I prefer to go with (3) > > Yes, based on the facts I agree that (3) is the best way to

Re: [sqlite] Unicode collation

2007-06-25 Thread Jiri Hajek
So the choices seem to be: (1) Databases that corrupt if you move across platforms. (2) A 10MB database engine (3) Leave things as they are I prefer to go with (3) Yes, based on the facts I agree that (3) is the best way to go - a developer can decide whether to use Windows methods only, or

Re: [sqlite] Unicode collation

2007-06-25 Thread drh
"Jiri Hajek" <[EMAIL PROTECTED]> wrote: > Ok, drh, could you share your thoughts about this. Could something > like this, i.e. some kind of recommendation be created, so that all > applications are consistent in Unicode support? > Dan's observation is correct. It seems to me that the collating s

Re: [sqlite] Unicode collation

2007-06-25 Thread Jiri Hajek
Ok, drh, could you share your thoughts about this. Could something like this, i.e. some kind of recommendation be created, so that all applications are consistent in Unicode support? Thanks, Jiri - To unsubscribe, send e

Re: [sqlite] Unicode collation

2007-06-23 Thread Dan Kennedy
On Sat, 2007-06-23 at 10:56 +0200, Jiri Hajek wrote: > > The reason is as you've surmised. Not all systems have full unicode > > support (I'm not sure, but if I had to guess, I would say very few > > systems do). Including an implementation with SQLite would bloat > > the library to at least severa

Re: [sqlite] Unicode collation

2007-06-23 Thread Jiri Hajek
The reason is as you've surmised. Not all systems have full unicode support (I'm not sure, but if I had to guess, I would say very few systems do). Including an implementation with SQLite would bloat the library to at least several times it's current size. I know really well only Windows where i

Re: [sqlite] Unicode collation

2007-06-22 Thread Dan Kennedy
On Fri, 2007-06-22 at 18:57 +0200, Jiri Hajek wrote: > Hello, > > I wonder whether there are any plans to include internally proper > Unicode comparisons? Don't get me wrong, I think that it's great that > SQLite supports custom collations, there's absolutely no problem to > handle it in internall

Re: [sqlite] unicode like UPPER and LOWER ?

2007-05-08 Thread Dan Kennedy
On Tue, 2007-05-08 at 10:45 +0700, Kirill wrote: > Good day, > > SQLite version 3.3.17 > Enter ".help" for instructions > sqlite> create table tbl1(t1 varchar(10)); > sqlite> insert into tbl1 values('софт'); - lowChar > sqlite> insert into tbl1 values('СОФТ'); - upChar > sqlite> select * from tbl1

Re: [sqlite] Unicode Help

2006-12-07 Thread Scott Hess
On 12/7/06, Nuno Lucas <[EMAIL PROTECTED]> wrote: On 12/7/06, Da Martian <[EMAIL PROTECTED]> wrote: > Ok, that does answer one of my questions I think. If I passed something not > in UTF-8 to sqlite would it return it exactly the same way I passed it in? > From your statement of chaos below I ass

Re: [sqlite] Unicode Help

2006-12-07 Thread Nuno Lucas
On 12/7/06, Da Martian <[EMAIL PROTECTED]> wrote: Ok, that does answer one of my questions I think. If I passed something not in UTF-8 to sqlite would it return it exactly the same way I passed it in? From your statement of chaos below I assume it wont if that data somehow violates UTF-8. So I ne

Re: [sqlite] Unicode Help

2006-12-07 Thread Da Martian
Hi You seem really confused about the whole encoding issue. Yes definatly confused, I had always hope unicode would simplify the world, but my experiences have shown no such luck :-) Codepages haunted my past and encodings haunt my future :-) Ok, that does answer one of my questions I think.

Re: [sqlite] Unicode Help

2006-12-07 Thread drh
"Da Martian" <[EMAIL PROTECTED]> wrote: > When using the > NON16 version of prepare: > If I add text which is in UTF16 what happens? > > 16 Version: > If I add UTF16 text what happnes? > if I add UTF-8 Text what happens? > if I add ASCIII text what happnes? > You seem really confused about the

Re: [sqlite] UNICODE Support

2006-08-08 Thread Jens Miltner
Am 04.08.2006 um 19:23 schrieb Cory Nelson: I was not talking about sorting in my post - I've had simple = index comparisons fail in UTF-8. I'm pretty sure you can get the same kind of 'failure' when using UTF-16, e.g. when comparing decomposed against composed forms of unicode strings. S

Re: [sqlite] UNICODE Support

2006-08-04 Thread Nathaniel Smith
On Fri, Aug 04, 2006 at 10:02:58PM -0700, Cory Nelson wrote: > On 8/4/06, Trevor Talbot <[EMAIL PROTECTED]> wrote: > >On 8/4/06, Cory Nelson <[EMAIL PROTECTED]> wrote: > > > >> But, since you brought it up - I have no expectations of SQLite > >> integrating a full Unicode locale library, however it

Re: [sqlite] UNICODE Support

2006-08-04 Thread Trevor Talbot
On 8/4/06, Cory Nelson <[EMAIL PROTECTED]> wrote: On 8/4/06, Trevor Talbot <[EMAIL PROTECTED]> wrote: > On 8/4/06, Cory Nelson <[EMAIL PROTECTED]> wrote: > > > But, since you brought it up - I have no expectations of SQLite > > integrating a full Unicode locale library, however it would be a grea

Re: [sqlite] UNICODE Support

2006-08-04 Thread Cory Nelson
On 8/4/06, Trevor Talbot <[EMAIL PROTECTED]> wrote: On 8/4/06, Cory Nelson <[EMAIL PROTECTED]> wrote: > But, since you brought it up - I have no expectations of SQLite > integrating a full Unicode locale library, however it would be a great > improvement if it would respect the current locale an

Re: [sqlite] UNICODE Support

2006-08-04 Thread Trevor Talbot
On 8/4/06, Cory Nelson <[EMAIL PROTECTED]> wrote: But, since you brought it up - I have no expectations of SQLite integrating a full Unicode locale library, however it would be a great improvement if it would respect the current locale and use wcs* functions when available, or at least order by

Re: [sqlite] UNICODE Support

2006-08-04 Thread Nuno Lucas
On 8/5/06, Cory Nelson <[EMAIL PROTECTED]> wrote: On 8/4/06, Nuno Lucas <[EMAIL PROTECTED]> wrote: > On 8/4/06, Cory Nelson <[EMAIL PROTECTED]> wrote: > > IE, using memcmp() to compare strings. I've been bitten by this > > before, with SQLite producing unexpected results when using UTF-8. > > Us

Re: [sqlite] UNICODE Support

2006-08-04 Thread Cory Nelson
On 8/4/06, Nuno Lucas <[EMAIL PROTECTED]> wrote: On 8/4/06, Cory Nelson <[EMAIL PROTECTED]> wrote: > IE, using memcmp() to compare strings. I've been bitten by this > before, with SQLite producing unexpected results when using UTF-8. > Using UTF-16 has worked more reliably in my experience. SQL

Re: [sqlite] UNICODE Support

2006-08-04 Thread Nuno Lucas
On 8/4/06, Cory Nelson <[EMAIL PROTECTED]> wrote: IE, using memcmp() to compare strings. I've been bitten by this before, with SQLite producing unexpected results when using UTF-8. Using UTF-16 has worked more reliably in my experience. SQLite only knows how to sort ASCII, so memcmp does that

Re: [sqlite] UNICODE Support

2006-08-04 Thread Cory Nelson
On 8/4/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: "Cory Nelson" <[EMAIL PROTECTED]> wrote: > On 8/3/06, RohitPatel <[EMAIL PROTECTED]> wrote: > > I recommend using utf-16 in the database - sqlite doesn't fully > support utf-8, and some things may give unexpected results if you use > it.

  1   2   >