Re: [sqlite] The byte order mark problem in fields with ICUcollation

Nicolas Williams wrote: > I believe the right thing to do is to normalize strings when creating > index entries, but to leave the table data unnormalized. You'd have > to make the equality operator also normalize though. I believe that's precisely what ICU collations

Re: [sqlite] The byte order mark problem in fields with ICUcollation

On Mon, Dec 07, 2009 at 05:35:49PM -0500, Igor Tandetnik wrote: > Alexey Pechnikov > wrote: > > The normalization is now performed by any string operation. But more > > fast and useful to do it once at data store. > > So, which normalization form should the data store

Re: [sqlite] The byte order mark problem in fields with ICUcollation

Alexey Pechnikov wrote: > On Tuesday 08 December 2009 01:35:49 Igor Tandetnik wrote: >> So, which normalization form should the data store choose for me? >> And what if I need a different one? >> >> I'd rather the database store my data exactly the way I put it in. I >>

Re: [sqlite] The byte order mark problem in fields with ICUcollation

Hello! On Tuesday 08 December 2009 01:35:49 Igor Tandetnik wrote: > So, which normalization form should the data store choose for me? And what if > I need a different one? > > I'd rather the database store my data exactly the way I put it in. I really > don't want it to decide for me what my

Re: [sqlite] The byte order mark problem in fields with ICUcollation

Alexey Pechnikov wrote: > The normalization is now performed by any string operation. But more > fast and useful to do it once at data store. So, which normalization form should the data store choose for me? And what if I need a different one? I'd rather the database

Re: [sqlite] The byte order mark problem in fields with ICUcollation

Hello! On Tuesday 08 December 2009 01:07:54 Igor Tandetnik wrote: > Alexey Pechnikov > wrote: > > Yes, the BOM is on the original string. But with ICU collation we can > > see that 17 symbols string is equal to 16 symbols string. I think > > this result is not right. >

Re: [sqlite] The byte order mark problem in fields with ICUcollation

Alexey Pechnikov wrote: > Yes, the BOM is on the original string. But with ICU collation we can > see that 17 symbols string is equal to 16 symbols string. I think > this result is not right. What's the basis for this belief? It's not at all uncommon for two Unicode

Re: [sqlite] The byte order mark problem in fields with ICUcollation

Hello! On Monday 07 December 2009 22:29:47 Igor Tandetnik wrote: > What do you mean by "SQLite stores" the BOM? Are you saying that you are > passing in a string without the BOM, and SQLite spontaneoulsy manufactures > one? Double-check your application - I suspect you'll find that you are >

Re: [sqlite] The byte order mark problem in fields with ICUcollation

Alexey Pechnikov wrote: > The SQLite in the field with ICU collation does store the BOM in the > start of string. What do you mean by "SQLite stores" the BOM? Are you saying that you are passing in a string without the BOM, and SQLite spontaneoulsy manufactures one?