Re: [Pytables-users] Pytables bulk loading data

2013-07-18 Thread Pushkar Raj Pande
27;. > > .. _pandas: http://pandas.pydata.org/ On Thu, Jul 18, 2013 at 1:00 AM, Antonio Valentino < antonio.valent...@tiscali.it> wrote: > Hi Pushkar, > > Il 18/07/2013 08:45, Pushkar Raj Pande ha scritto: > > Both loadtxt and genfromtxt read the entire data into

Re: [Pytables-users] Pytables-users Digest, Vol 86, Issue 8

2013-07-17 Thread Pushkar Raj Pande
Both loadtxt and genfromtxt read the entire data into memory which is not desirable. Is there a way to achieve streaming writes? Thanks, Pushkar On Wed, Jul 17, 2013 at 7:04 PM, Pushkar Raj Pande wrote: > Thanks Antonio and Anthony. I will give this a try. > > -Pushkar > > &

Re: [Pytables-users] Pytables-users Digest, Vol 86, Issue 8

2013-07-17 Thread Pushkar Raj Pande
ta with NumPy functions and > then write back out to PyTables. This is the fastest way to do things. > > Be Well > Anthony > > > On Wed, Jul 17, 2013 at 2:12 PM, Antonio Valentino < > antonio.valent...@tiscali.it> wrote: > > > Hi Pushkar, > > > > Il

[Pytables-users] Pytables bulk loading data

2013-07-17 Thread Pushkar Raj Pande
Hi all, I am trying to figure out the best way to bulk load data into pytables. This question may have been already answered but I couldn't find what I was looking for. The source data is in form of csv which may require parsing, type checking and setting default values if it doesn't conform to t