Re: [sqlite] Can SQLite import Latin1 data?
It's odd that SQLite doesn't support this, since it's not uncommun to have big Latin1 input files that can take a while to load in eg. Excel. For others' benefit, GNUWin32's iconv is very fast: "C:\Program Files\GnuWin32\bin\iconv.exe" -f "windows-1252" -t "UTF-8" "input.1252txt" > "output.UTF8.txt" Thank you. -- Sent from: http://sqlite.1065341.n5.nabble.com/ ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Can SQLite import Latin1 data?
On Nov 15, 2019, at 2:15 PM, Jose Isaias Cabrera wrote: > > Shawn Wagner, on Friday, November 15, 2019 04:01 PM, wrote... >> >> If you're on Windows, which cp1252 suggests, just make sure that you don't >> end up with a BOM at the start of the file when you convert it. Windows >> tools that output utf-8 are sometimes prone to add one even though it's >> pointless to have. > > Why do you think it's pointless? https://www.unicode.org/faq/utf_bom.html#bom5 ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Can SQLite import Latin1 data?
Unlike UTF-16, which uses 2 byte code units without a fixed endianess (meaning to be robust you need to account for both little and big endian encodings when reading files using it), UTF-8 uses a 1 byte code unit and thus doesn't have any endian issues or a need for a byte order mark. On Fri, Nov 15, 2019, 1:15 PM Jose Isaias Cabrera wrote: > > Shawn Wagner, on Friday, November 15, 2019 04:01 PM, wrote... > > > > If you're on Windows, which cp1252 suggests, just make sure that you > don't > > end up with a BOM at the start of the file when you convert it. Windows > > tools that output utf-8 are sometimes prone to add one even though it's > > pointless to have. > > Why do you think it's pointless? > > josé > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Can SQLite import Latin1 data?
Shawn Wagner, on Friday, November 15, 2019 04:01 PM, wrote... > > If you're on Windows, which cp1252 suggests, just make sure that you don't > end up with a BOM at the start of the file when you convert it. Windows > tools that output utf-8 are sometimes prone to add one even though it's > pointless to have. Why do you think it's pointless? josé ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Can SQLite import Latin1 data?
If you're on Windows, which cp1252 suggests, just make sure that you don't end up with a BOM at the start of the file when you convert it. Windows tools that output utf-8 are sometimes prone to add one even though it's pointless to have. On Linux etc., you can try .import '| iconv -f cp1252 -t utf8 myfile.csv' mytable to convert on the fly when importing into sqlite. On Fri, Nov 15, 2019, 12:00 PM Winfried wrote: > Hello, > > I have a big CSV file that's encoded in Latin1 (cp1252), while SQLite > stores > strings as Unicode. > > Neither Google nor ".help" helped to find if SQLite offers a switch to > convert Latin1 to UTF-8 on the fly before running the ".import" command. > > Should I first convert the file into UTF-8 before importing it into SQLite? > > Thank you. > > > > -- > Sent from: http://sqlite.1065341.n5.nabble.com/ > ___ > sqlite-users mailing list > sqlite-users@mailinglists.sqlite.org > http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users > ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Can SQLite import Latin1 data?
Winfried, on Friday, November 15, 2019 03:13 PM, wrote... > > Hello, > > I have a big CSV file that's encoded in Latin1 (cp1252), while SQLite stores > strings as Unicode. > > Neither Google nor ".help" helped to find if SQLite offers a switch to > convert Latin1 to UTF-8 on the fly before running the ".import" command. > > Should I first convert the file into UTF-8 before importing it into SQLite? I would just take that file and using your favorite Text Editor (EmEditor is a good one), and save it as UFT-8. SQLite will handle it beautifully. Thanks. josé ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
Re: [sqlite] Can SQLite import Latin1 data?
On 15 Nov 2019, at 20:13, Winfried wrote: > I have a big CSV file that's encoded in Latin1 (cp1252), while SQLite stores > strings as Unicode. > > Neither Google nor ".help" helped to find if SQLite offers a switch to > convert Latin1 to UTF-8 on the fly before running the ".import" command. > > Should I first convert the file into UTF-8 before importing it into SQLite? Far as I know, SQLite stores your text as UTF-8 and you have to do the conversion yourself. I think there's a UTF-16 option too. I'd start here: https://www.sqlite.org/faq.html -- Cheers -- Tim ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
[sqlite] Can SQLite import Latin1 data?
Hello, I have a big CSV file that's encoded in Latin1 (cp1252), while SQLite stores strings as Unicode. Neither Google nor ".help" helped to find if SQLite offers a switch to convert Latin1 to UTF-8 on the fly before running the ".import" command. Should I first convert the file into UTF-8 before importing it into SQLite? Thank you. -- Sent from: http://sqlite.1065341.n5.nabble.com/ ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org http://mailinglists.sqlite.org/cgi-bin/mailman/listinfo/sqlite-users
