Re: [sqlite] importing a large TSV file

2019-04-04 Thread Gert Van Assche
Thank you all for these tips. Very helpful! Op di 2 apr. 2019 om 08:35 schreef Rowan Worth : > On Mon, 1 Apr 2019 at 19:20, Domingo Alvarez Duarte > wrote: > > > Hello Gert ! > > > > I normally do this (be aware that if there is a power outage the > > database is screwed): > > > > === > > > >

Re: [sqlite] importing a large TSV file

2019-04-02 Thread Rowan Worth
On Mon, 1 Apr 2019 at 19:20, Domingo Alvarez Duarte wrote: > Hello Gert ! > > I normally do this (be aware that if there is a power outage the > database is screwed): > > === > > PRAGMA synchronous = OFF; > begin; > > --processing here > > commit; > PRAGMA synchronous = ON; > You can probably

Re: [sqlite] importing a large TSV file

2019-04-01 Thread David Raymond
Of Simon Slavin Sent: Monday, April 01, 2019 9:47 AM To: SQLite mailing list Subject: Re: [sqlite] importing a large TSV file On 1 Apr 2019, at 2:41pm, Simon Slavin wrote: > BEGIN; >... CREATE all your INDEXes ... >VACUUM; -- optional > END; Apologies. The documentation say

Re: [sqlite] importing a large TSV file

2019-04-01 Thread Simon Slavin
On 1 Apr 2019, at 2:41pm, Simon Slavin wrote: > BEGIN; >... CREATE all your INDEXes ... >VACUUM; -- optional > END; Apologies. The documentation says "A VACUUM will fail if there is an open transaction" I'm not sure whether this means that VACUUM must be outside the transaction,

Re: [sqlite] importing a large TSV file

2019-04-01 Thread Simon Slavin
On 1 Apr 2019, at 12:14pm, Gert Van Assche wrote: > I need to create an SQLite db from a large TSV file. (30 GB) > Are there any setting I can give to the db so I can speed up the import? If you're doing it using the SQLite CLI tool, then just rely on the tool to do it in the most convenient

Re: [sqlite] importing a large TSV file

2019-04-01 Thread Donald Griggs
I believe it's also helpful to avoid creating indexes (and enforcing foreign keys) until after the import. On Mon, Apr 1, 2019 at 7:15 AM Gert Van Assche wrote: > Hi all, > > I need to create an SQLite db from a large TSV file. (30 GB) > Are there any setting I can give to the db so I can speed

Re: [sqlite] importing a large TSV file

2019-04-01 Thread Domingo Alvarez Duarte
Hello Gert ! I normally do this (be aware that if there is a power outage the database is screwed): === PRAGMA synchronous = OFF; begin; --processing here commit; PRAGMA synchronous = ON; === Cheers ! On 1/4/19 13:14, Gert Van Assche wrote: Hi all, I need to create an SQLite db from

[sqlite] importing a large TSV file

2019-04-01 Thread Gert Van Assche
Hi all, I need to create an SQLite db from a large TSV file. (30 GB) Are there any setting I can give to the db so I can speed up the import? thank you Gert ___ sqlite-users mailing list sqlite-users@mailinglists.sqlite.org