Re: [sqlite] Is there a way to perform a muti-level sort and extract of large data sets?

2017-09-27 Thread Donald Griggs
Hi Ron, It may be that increasing the sqlite cache size will substantially reduce the time for either the CREATE INDEX or the SELECT ... ORDER BY instructions (depending upon which method you choose). https://sqlite.org/pragma.html#pragma_cache_size On Wed, Sep 27, 2017 at 1:58 AM, jungle

Re: [sqlite] Is there a way to perform a muti-level sort and extract of large data sets?

2017-09-26 Thread jungle Boogie
Hi there, Just because I'm interested, I'm wondering if you can identify your hardware, and how long it takes your system to do your desired operation on such a large number of records. Do let us know which option you performed the query with as well. P.S. For best results, I'd recommend using t

Re: [sqlite] Is there a way to perform a muti-level sort and extract of large data sets?

2017-09-26 Thread Ron Barnes
😊 Than you! -Original Message- From: sqlite-users [mailto:sqlite-users-boun...@mailinglists.sqlite.org] On Behalf Of Stephen Chrzanowski Sent: Tuesday, September 26, 2017 4:31 PM To: SQLite mailing list Subject: Re: [sqlite] Is there a way to perform a muti-level sort and extract of

Re: [sqlite] Is there a way to perform a muti-level sort and extract of large data sets?

2017-09-26 Thread Stephen Chrzanowski
uh.. UNIQUE... DISTINCT... , On Tue, Sep 26, 2017 at 4:30 PM, Stephen Chrzanowski wrote: > > > On Tue, Sep 26, 2017 at 1:36 PM, Simon Slavin > wrote: > >> >> >> My one concern in reading your post is how your dates are formatted. >> When putting your date fields into your SQL table you will hav

Re: [sqlite] Is there a way to perform a muti-level sort and extract of large data sets?

2017-09-26 Thread Stephen Chrzanowski
On Tue, Sep 26, 2017 at 1:36 PM, Simon Slavin wrote: > > > My one concern in reading your post is how your dates are formatted. When > putting your date fields into your SQL table you will have to ensure that > dates are saved as a day number, or as text which naturally sorts into date > order,

Re: [sqlite] Is there a way to perform a muti-level sort and extract of large data sets?

2017-09-26 Thread Ron Barnes
[sqlite] Is there a way to perform a muti-level sort and >extract of large data sets? > > > >On 26 Sep 2017, at 6:24pm, Ron Barnes wrote: > >> I need to sort them as follows... >> >> Sort Field 1 Ascending >> Sort Field 2 Ascending WITHIN field 1 Sort

Re: [sqlite] Is there a way to perform a muti-level sort and extract of large data sets?

2017-09-26 Thread Ron Barnes
Sent: Tuesday, September 26, 2017 2:14 PM To: SQLite mailing list Subject: Re: [sqlite] Is there a way to perform a muti-level sort and extract of large data sets? On Sep 26, 2017, at 11:24 AM, Ron Barnes wrote: > > I have approximately 600 million records that need to be sorted Where

Re: [sqlite] Is there a way to perform a muti-level sort and extract of large data sets?

2017-09-26 Thread Ron Barnes
perform a muti-level sort and extract of large data sets? On 26 Sep 2017, at 6:24pm, Ron Barnes wrote: > I need to sort them as follows... > > Sort Field 1 Ascending > Sort Field 2 Ascending WITHIN field 1 > Sort Field 3 Ascending WITHIN field 2 WITHIN field 1 Sort Field 4 >

Re: [sqlite] Is there a way to perform a muti-level sort and extract of large data sets?

2017-09-26 Thread Keith Medcalf
about anticipated traffic volume. >-Original Message- >From: sqlite-users [mailto:sqlite-users- >boun...@mailinglists.sqlite.org] On Behalf Of Simon Slavin >Sent: Tuesday, 26 September, 2017 11:37 >To: SQLite mailing list >Subject: Re: [sqlite] Is there a way to perfor

Re: [sqlite] Is there a way to perform a muti-level sort and extract of large data sets?

2017-09-26 Thread Warren Young
On Sep 26, 2017, at 11:24 AM, Ron Barnes wrote: > > I have approximately 600 million records that need to be sorted Where is the data now? > There are 18 table entries. You mean 18 columns per row, right? > I also need to deduplicate the records based upon the sorted output file. You speak o

Re: [sqlite] Is there a way to perform a muti-level sort and extract of large data sets?

2017-09-26 Thread Simon Slavin
On 26 Sep 2017, at 6:24pm, Ron Barnes wrote: > I need to sort them as follows... > > Sort Field 1 Ascending > Sort Field 2 Ascending WITHIN field 1 > Sort Field 3 Ascending WITHIN field 2 WITHIN field 1 > Sort Field 4 Descending WITHIN field 3 WITHIN field 2 WITHIN field 1 <== This > is a Dat

[sqlite] Is there a way to perform a muti-level sort and extract of large data sets?

2017-09-26 Thread Ron Barnes
Hello All, I have approximately 600 million records that need to be sorted and then extracted to a flat file. I am unable to code a solution using visual Basic .NET. It was suggested to me that a DB engine could perform my task for me. Is there a way to accomplish this using the multi-level s