Re: [sqlite] Importing ~1000 CSV files faster

Joe Fisher wrote on Tuesday, August 19, 2014 5:11 PM > I use the temp table because every CSV files has a header with the > column names. Can you just import the files, header row and all, into your destination table and just delete the 2000 header rows at the end?

Re: [sqlite] Importing ~1000 CSV files faster

On Aug 19, 2014, at 11:11 PM, joe.fis...@tanguaylab.com wrote: > Is there something better I can do to improve this process? PRAGMA journal_mode = off; http://www.sqlite.org/pragma.html#pragma_journal_mode > Perhaps one transaction? Perhaps turn something off?

Re: [sqlite] Importing ~1000 CSV files faster

On 19 Aug 2014, at 10:13pm, John Drescher wrote: >> I'm running the following script on more than 1000 2MB CSV files and I'd >> like to speed it up if possible. I noticed that a 'WAL' is running. Is there >> something better I can do to improve this process? > > Use an

Re: [sqlite] Importing ~1000 CSV files faster

Directly after opening your db: PRAGMA synchronous = OFF; PRAGMA journal_mode = WAL; It should fly then, at the cost of risking db corruption in case of a crash. Ben(jamin Stadin) Am 19.08.14 23:11 schrieb "joe.fis...@tanguaylab.com" unter : >I'm running the

Re: [sqlite] Importing ~1000 CSV files faster

> I'm running the following script on more than 1000 2MB CSV files and I'd > like to speed it up if possible. I noticed that a 'WAL' is running. Is there > something better I can do to improve this process? Use an SSD. John ___ sqlite-users mailing

[sqlite] Importing ~1000 CSV files faster

I'm running the following script on more than 1000 2MB CSV files and I'd like to speed it up if possible. I noticed that a 'WAL' is running. Is there something better I can do to improve this process? Perhaps one transaction? Perhaps turn something off? It took about 1.5 hours to run. I use