Re: Fast file data retrieval?

2012-03-21 Thread Stefan Behnel
Jorgen Grahn, 13.03.2012 21:44: On Mon, 2012-03-12, MRAB wrote: Probably the best solution is to put it into a database. Have a look at the sqlite3 module. Some people like to use databases for everything, others never use them. I'm in the latter crowd, so to me this sounds as overkill

Re: Fast file data retrieval?

2012-03-13 Thread Jorgen Grahn
On Mon, 2012-03-12, MRAB wrote: On 12/03/2012 19:39, Virgil Stokes wrote: I have a rather large ASCII file that is structured as follows header line 9 nonblank lines with alphanumeric data header line 9 nonblank lines with alphanumeric data ... ... ... header line 9 nonblank lines with

Fast file data retrieval?

2012-03-12 Thread Virgil Stokes
I have a rather large ASCII file that is structured as follows header line 9 nonblank lines with alphanumeric data header line 9 nonblank lines with alphanumeric data ... ... ... header line 9 nonblank lines with alphanumeric data EOF where, a data set contains 10 lines (header + 9 nonblank)

Re: Fast file data retrieval?

2012-03-12 Thread MRAB
On 12/03/2012 19:39, Virgil Stokes wrote: I have a rather large ASCII file that is structured as follows header line 9 nonblank lines with alphanumeric data header line 9 nonblank lines with alphanumeric data ... ... ... header line 9 nonblank lines with alphanumeric data EOF where, a data set

Re: Fast file data retrieval?

2012-03-12 Thread Arnaud Delobelle
On 12 March 2012 19:39, Virgil Stokes v...@it.uu.se wrote: I have a rather large ASCII file that is structured as follows header line 9 nonblank lines with alphanumeric data header line 9 nonblank lines with alphanumeric data ... ... ... header line 9 nonblank lines with alphanumeric

Re: Fast file data retrieval?

2012-03-12 Thread Dan Stromberg
If the ID's are sorted, you could probably rig a binary search using seek. This'll be easier if the records have a constant length, but it's still possible for variable-length, just messier. Otherwise you could stash them all in a dictionary (in memory) or anydbm (on disk) to get indexed access.

RE: Fast file data retrieval?

2012-03-12 Thread Prasad, Ramit
header line 9 nonblank lines with alphanumeric data header line 9 nonblank lines with alphanumeric data ... ... ... header line 9 nonblank lines with alphanumeric data EOF where, a data set contains 10 lines (header + 9 nonblank) and there can be several thousand data

Re: Fast file data retrieval?

2012-03-12 Thread Jon Clements
On Monday, 12 March 2012 20:31:35 UTC, MRAB wrote: On 12/03/2012 19:39, Virgil Stokes wrote: I have a rather large ASCII file that is structured as follows header line 9 nonblank lines with alphanumeric data header line 9 nonblank lines with alphanumeric data ... ... ...