On 2017-06-26 15:01, jose isaias cabrera wrote:
I have made a desicion to always include the BOM in all my text files whether they are UTF8, UTF16 or UTF32 little or big endian. I think all of us should also.
I'm sorry, if I introduced ambiguity, but I had described SQLite's and SQLite shell's behavior -- the sole. I'm not entreating to kill all UTF-8 BOMs in the universe.
Just because the "Unicode Gurus" didn't think so, does not mean they are right. I had a doctor give me the wrong diagnose. There were just too many symptoms that looked alike and they chosed one and went with it. The same thing happened, the Unicode Gurus, they never thought about the problems they would be causing today. Some applications do not place BOM on UTF8, UTF16 files, and then you have to go and find which one is it, and decode the file correctly.
The problem, which you described, had not been introduced nor created by ``Unicode Gurus''. AFAIR, finding of a correct encoding/codepage (of files with an unknown origin) was present in the olden days, far before Unicode. UTF-8 is far easier recognizable then others.
This can all be prevented by having a BOM.
This would have helped, if it had been only UTF-8 and some single-byte code page.
Yes, I know I am saying everything every body is, but what I am also saying is to let us all use the BOM, and also have every application we write welcome the BOM.
What if I want to place 0xFEFF at the beginning of UTF-8? The second EF BB BF as BOM? OK - but the standard says ``there is no BOM''. This is what the standard is for. I agree with you -- where a character set is unmarked, there UTF-8 BOM is useful as an encoding signature. However, where SQLite is accepting only UTF-8, there I expect that placing EF BB BF at the beginning will be interpreted as codepoint -- not BOM, until it will say explicitly: ``My interpretation of EF BB BF is BOM''. I am not going to tell whether zero or all of my UTF-8 files have BOM neither whether or not to use/kill/welcome all UTF-8 BOMs -- it does not matter. However, in case of SQLite, Clemens' arguments are very stringent -- I hope SQLite shell's behavior will not change. For the sake of the standard conformance, thus predictability and determinedness. -- best regards Cezary H. Noweta _______________________________________________ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
