Both loadtxt and genfromtxt read the entire data into memory which is not
desirable. Is there a way to achieve streaming writes?
Thanks,
Pushkar
On Wed, Jul 17, 2013 at 7:04 PM, Pushkar Raj Pande <topgun...@gmail.com>wrote:
> Thanks Antonio and Anthony. I will give this a try.
>
> -Pushkar
>
>
> On Wed, Jul 17, 2013 at 2:59 PM, <
> pytables-users-requ...@lists.sourceforge.net> wrote:
>
>> Date: Wed, 17 Jul 2013 16:59:16 -0500
>> From: Anthony Scopatz <scop...@gmail.com>
>> Subject: Re: [Pytables-users] Pytables bulk loading data
>> To: Discussion list for PyTables
>> <pytables-users@lists.sourceforge.net>
>> Message-ID:
>> <
>> capk-6t4ht9+ncdd_1oojrbn4u_6+ouekobklmokeufjojjk...@mail.gmail.com>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>> Hi Pushkar,
>>
>> I agree with Antonio. You should load your data with NumPy functions and
>> then write back out to PyTables. This is the fastest way to do things.
>>
>> Be Well
>> Anthony
>>
>>
>> On Wed, Jul 17, 2013 at 2:12 PM, Antonio Valentino <
>> antonio.valent...@tiscali.it> wrote:
>>
>> > Hi Pushkar,
>> >
>> > Il 17/07/2013 19:28, Pushkar Raj Pande ha scritto:
>> > > Hi all,
>> > >
>> > > I am trying to figure out the best way to bulk load data into
>> pytables.
>> > > This question may have been already answered but I couldn't find what
>> I
>> > was
>> > > looking for.
>> > >
>> > > The source data is in form of csv which may require parsing, type
>> > checking
>> > > and setting default values if it doesn't conform to the type of the
>> > column.
>> > > There are over 100 columns in a record. Doing this in a loop in python
>> > for
>> > > each row of the record is very slow compared to just fetching the rows
>> > from
>> > > one pytable file and writing it to another. Difference is almost a
>> factor
>> > > of ~50.
>> > >
>> > > I believe if I load the data using a C procedure that does the parsing
>> > and
>> > > builds the records to write in pytables I can get close to the speed
>> of
>> > > just copying and writing the rows from 1 pytable to another. But may
>> be
>> > > there is something simple and better that already exists. Can someone
>> > > please advise? But if it is a C procedure that I should write can
>> someone
>> > > point me to some examples or snippets that I can refer to put this
>> > together.
>> > >
>> > > Thanks,
>> > > Pushkar
>> > >
>> >
>> > numpy has some tools for loading data from csv files like loadtxt [1],
>> > genfromtxt [2] and other variants.
>> >
>> > Non of them is OK for you?
>> >
>> > [1]
>> >
>> >
>> http://docs.scipy.org/doc/numpy/reference/generated/numpy.loadtxt.html#numpy.loadtxt
>> > [2]
>> >
>> >
>> http://docs.scipy.org/doc/numpy/reference/generated/numpy.genfromtxt.html#numpy.genfromtxt
>> >
>> >
>> > cheers
>> >
>> > --
>> > Antonio Valentino
>> >
>> >
>> >
>> ------------------------------------------------------------------------------
>> > See everything from the browser to the database with AppDynamics
>> > Get end-to-end visibility with application monitoring from AppDynamics
>> > Isolate bottlenecks and diagnose root cause in seconds.
>> > Start your free trial of AppDynamics Pro today!
>> >
>> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
>> > _______________________________________________
>> > Pytables-users mailing list
>> > Pytables-users@lists.sourceforge.net
>> > https://lists.sourceforge.net/lists/listinfo/pytables-users
>> >
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>>
>> ------------------------------
>>
>>
>> ------------------------------------------------------------------------------
>> See everything from the browser to the database with AppDynamics
>> Get end-to-end visibility with application monitoring from AppDynamics
>> Isolate bottlenecks and diagnose root cause in seconds.
>> Start your free trial of AppDynamics Pro today!
>>
>> http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
>>
>> ------------------------------
>>
>> _______________________________________________
>> Pytables-users mailing list
>> Pytables-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>
>>
>> End of Pytables-users Digest, Vol 86, Issue 8
>> *********************************************
>>
>
>
------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
