Re: [sqlite] Bulk load strategy

2017-05-17 Thread Joseph L. Casale
> -Original Message- > From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On > Behalf Of Simon Slavin > Sent: Wednesday, May 17, 2017 2:02 PM > To: SQLite mailing list <sqlite-users@mailinglists.sqlite.org> > Subject: Re: [sqlite] Bulk load str

Re: [sqlite] Bulk load strategy

2017-05-17 Thread Simon Slavin
On 17 May 2017, at 5:05pm, Simon Slavin wrote: > Fastest way to do bulk inserts would be to delete all the indexes which don’t > play any part in identifying duplicates, then do the inserting, then remake > the indexes. I forgot: once you’ve remade the indexes run

Re: [sqlite] Bulk load strategy

2017-05-17 Thread David Raymond
, 2017 3:40 PM To: sqlite-users@mailinglists.sqlite.org Subject: Re: [sqlite] Bulk load strategy Am 17.05.2017 um 19:08 schrieb David Raymond: > The unique index on DistinguishedName though is what gets used for that sub > query of the insert, so most definitely keep that one index for the

Re: [sqlite] Bulk load strategy

2017-05-17 Thread Olaf Schmidt
Am 17.05.2017 um 19:08 schrieb David Raymond: The unique index on DistinguishedName though is what gets used for that sub query of the insert, so most definitely keep that one index for the whole load. (The others can be left out until the end though) I once had a similar scenario, and

Re: [sqlite] Bulk load strategy

2017-05-17 Thread Simon Slavin
On 17 May 2017, at 7:07pm, Joseph L. Casale wrote: > So I have one query which if I expect if I encounter will be painful: > > UPDATE AdAttribute > SET Value = @NewValue > WHERE Type = @Type > AND Value = @Value; > > I may pass member or memberOf to

Re: [sqlite] Bulk load strategy

2017-05-17 Thread Joseph L. Casale
> -Original Message- > From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On > Behalf Of Simon Slavin > Sent: Wednesday, May 17, 2017 10:05 AM > To: SQLite mailing list <sqlite-users@mailinglists.sqlite.org> > Subject: Re: [sqlite] B

Re: [sqlite] Bulk load strategy

2017-05-17 Thread David Raymond
Of Simon Slavin Sent: Wednesday, May 17, 2017 12:05 PM To: SQLite mailing list Subject: Re: [sqlite] Bulk load strategy So if you never insert duplicates on AdObject(DistinguishedName), DROP that index. And definitely DROP all the others. Then do your users. The reCREATE the indexes

Re: [sqlite] Bulk load strategy

2017-05-17 Thread Simon Slavin
On 17 May 2017, at 4:06pm, Joseph L. Casale wrote: > CREATE TABLE AdObject ( >IdINTEGER PRIMARY KEY NOT NULL, >DistinguishedName TEXTNOT NULL COLLATE NOCASE, >SamAccountNameTEXTCOLLATE NOCASE > ); > CREATE UNIQUE INDEX

Re: [sqlite] Bulk load strategy

2017-05-17 Thread Joseph L. Casale
> -Original Message- > From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On > Behalf Of Gerry Snyder > Sent: Wednesday, May 17, 2017 9:14 AM > To: SQLite mailing list <sqlite-users@mailinglists.sqlite.org> > Subject: Re: [sq

Re: [sqlite] Bulk load strategy

2017-05-17 Thread Gerry Snyder
On Wed, May 17, 2017 at 3:52 AM, Joseph L. Casale wrote: > I am trying to bulk load about a million records each with ~20 related > records > into two tables. I am using WAL journal mode, synchronous is off and > temp_store > is memory. The source data is static and

Re: [sqlite] Bulk load strategy

2017-05-17 Thread Joseph L. Casale
> From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On > Behalf Of Richard Hipp > Sent: Wednesday, May 17, 2017 8:54 AM > To: SQLite mailing list <sqlite-users@mailinglists.sqlite.org> > Subject: Re: [sqlite] Bulk load strategy > > Can you send

Re: [sqlite] Bulk load strategy

2017-05-17 Thread Richard Hipp
: Wednesday, May 17, 2017 6:04 AM >> To: sqlite-users@mailinglists.sqlite.org >> Subject: Re: [sqlite] Bulk load strategy >> >> Without an index, searching for a previous entry is likely to involve >> a scan through the entire table. It might be a better idea to hav

Re: [sqlite] Bulk load strategy

2017-05-17 Thread Joseph L. Casale
> From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On > Behalf Of Clemens Ladisch > Sent: Wednesday, May 17, 2017 6:04 AM > To: sqlite-users@mailinglists.sqlite.org > Subject: Re: [sqlite] Bulk load strategy > > Without an index, searching for a pre

Re: [sqlite] Bulk load strategy

2017-05-17 Thread Clemens Ladisch
Joseph L. Casale wrote: > I am deferring index creation to after the load. > The load proceeds along quickly to about 150k records where I encounter > statements > which perform modifications to previous entries. Without an index, searching for a previous entry is likely to involve a scan

Re: [sqlite] Bulk load strategy

2017-05-17 Thread Richard Hipp
On 5/17/17, Joseph L. Casale wrote: > I am trying to bulk load about a million records each with ~20 related > records > into two tables. I am using WAL journal mode, synchronous is off and > temp_store > is memory. The source data is static and the database will only