Re: [Numpy-discussion] mysql -> record array

2006-11-16 Thread Tim Hochberg
Francesc Altet wrote: > A Dimarts 14 Novembre 2006 23:08, Erin Sheldon escrigué: > >> On 11/14/06, John Hunter <[EMAIL PROTECTED]> wrote: >> >>> Has anyone written any code to facilitate dumping mysql query results >>> (mainly arrays of floats) into numpy arrays directly at the extension >>

Re: [Numpy-discussion] mysql -> record array

2006-11-16 Thread Francesc Altet
A Dimarts 14 Novembre 2006 23:08, Erin Sheldon escrigué: > On 11/14/06, John Hunter <[EMAIL PROTECTED]> wrote: > > Has anyone written any code to facilitate dumping mysql query results > > (mainly arrays of floats) into numpy arrays directly at the extension > > code layer. The query results->list

Re: [Numpy-discussion] mysql -> record array

2006-11-14 Thread Tim Hochberg
Tim Hochberg wrote: > [CHOP] > > The timings of these are pretty consistent with each other with the > previous runs except that the difference between retrieve1 and retrieve2 > has disappeared. In fact, all of the runs that produce lists have gotten > faster by about the same amount.. Odd! A li

Re: [Numpy-discussion] mysql -> record array

2006-11-14 Thread Erin Sheldon
On 11/14/06, Tim Hochberg <[EMAIL PROTECTED]> wrote: SNIP > > Interesting results Tim. From Pierre's results > > we saw that fromiter is the fastest way to get data > > into arrays. With your results we see there is a > > difference between iterating over the cursor and > > doing a fetchall() as

Re: [Numpy-discussion] mysql -> record array

2006-11-14 Thread Tim Hochberg
Erin Sheldon wrote: > On 11/14/06, Tim Hochberg <[EMAIL PROTECTED]> wrote: > >> Tim Hochberg wrote: >> >>> [SNIP] >>> >>> I'm no database user, but a glance at the at the docs seems to indicate >>> that you can get your data via an iterator (by iterating over the cursor >>> or some such db

Re: [Numpy-discussion] mysql -> record array

2006-11-14 Thread Keith Goodman
On 11/14/06, Erin Sheldon <[EMAIL PROTECTED]> wrote: > As an aside, your database is running on a local disk, right, so > the overehead of retrieving data is minimized here? > For my tests I think I am data retrieval limited because I > get exactly the same time for the equivalent of retrieve1 > a

Re: [Numpy-discussion] mysql -> record array

2006-11-14 Thread Erin Sheldon
On 11/14/06, Tim Hochberg <[EMAIL PROTECTED]> wrote: > Tim Hochberg wrote: > > John Hunter wrote: > > > >>> "Erin" == Erin Sheldon <[EMAIL PROTECTED]> writes: > >>> > >>> > >> Erin> The question I have been asking myself is "what is the > >> Erin> advantage of such an approach?"

Re: [Numpy-discussion] mysql -> record array

2006-11-14 Thread Tim Hochberg
Tim Hochberg wrote: > John Hunter wrote: > >>> "Erin" == Erin Sheldon <[EMAIL PROTECTED]> writes: >>> >>> >> Erin> The question I have been asking myself is "what is the >> Erin> advantage of such an approach?". It would be faster, but by >> >> In

Re: [Numpy-discussion] mysql -> record array

2006-11-14 Thread John Hunter
> "John" == John Hunter <[EMAIL PROTECTED]> writes: > "Erin" == Erin Sheldon <[EMAIL PROTECTED]> writes: Erin> The question I have been asking myself is "what is the Erin> advantage of such an approach?". It would be faster, but by John> In the use case that prompted this mes

Re: [Numpy-discussion] mysql -> record array

2006-11-14 Thread Pierre GM
John, I just added the following to your example: ... tnow = time.time() y = numpy.fromiter((tuple(i) for i in x), dtype=[('a',numpy.float_), ('b', numpy.float_)]) tdone = time.time() print 'Numpy.fromiter %1.2f elapsed seconds'%(tdone - tnow) ..

Re: [Numpy-discussion] mysql -> record array

2006-11-14 Thread Travis Oliphant
John Hunter wrote: >>"John" == John Hunter <[EMAIL PROTECTED]> writes: >> >> > > > >>"Erin" == Erin Sheldon <[EMAIL PROTECTED]> writes: >> >> >Erin> The question I have been asking myself is "what is the >Erin> advantage of such an approach

Re: [Numpy-discussion] mysql -> record array

2006-11-14 Thread Tim Hochberg
John Hunter wrote: >> "Erin" == Erin Sheldon <[EMAIL PROTECTED]> writes: >> > > Erin> The question I have been asking myself is "what is the > Erin> advantage of such an approach?". It would be faster, but by > > In the use case that prompted this message, the pull fro

Re: [Numpy-discussion] mysql -> record array

2006-11-14 Thread John Hunter
> "Erin" == Erin Sheldon <[EMAIL PROTECTED]> writes: Erin> The question I have been asking myself is "what is the Erin> advantage of such an approach?". It would be faster, but by In the use case that prompted this message, the pull from mysql took almost 3 seconds, and the conversio

Re: [Numpy-discussion] mysql -> record array

2006-11-14 Thread Erin Sheldon
On 11/14/06, John Hunter <[EMAIL PROTECTED]> wrote: > > Has anyone written any code to facilitate dumping mysql query results > (mainly arrays of floats) into numpy arrays directly at the extension > code layer. The query results->list->array conversion can be slow. > > Ideally, one could do this

[Numpy-discussion] mysql -> record array

2006-11-14 Thread John Hunter
Has anyone written any code to facilitate dumping mysql query results (mainly arrays of floats) into numpy arrays directly at the extension code layer. The query results->list->array conversion can be slow. Ideally, one could do this semi-automagically with record arrays and table introspection.