Thanks everyone for the great and well thought out responses!
To make matters worse, this is actually a 50gb compressed csv file. So
it looks like this, 2009.06.01.plasmasub.csv.gz
We get this data from another lab from the Westcoast every night
therefore I don't have the option to have this file
A Friday 26 June 2009 12:38:11 Mag Gam escrigué:
Thanks everyone for the great and well thought out responses!
To make matters worse, this is actually a 50gb compressed csv file. So
it looks like this, 2009.06.01.plasmasub.csv.gz
We get this data from another lab from the Westcoast every
I really like the slice by slice idea!
But, I don't know how to implement the code. Do you have any sample code?
I suspect its the writing portion thats taking the lonest. I did a
simple decompress test and its fast.
On Fri, Jun 26, 2009 at 7:05 AM, Francesc Altedfal...@pytables.org wrote:
A Friday 26 June 2009 13:09:13 Mag Gam escrigué:
I really like the slice by slice idea!
Hmm, after looking at the np.loadtxt() docstrings it seems it works by loading
the complete file at once, so you shouldn't use this directly (unless you
split your big file before, but this will take time
Yes, you are correct!
I think this is the best path.
However, I need to learn how to append a hdf5 dataset . I looked at
this, http://code.google.com/p/h5py/wiki/FAQ#Appending_data_to_a_dataset
but was not able to do so. Do you happen to have any sample code for
this, if you used hdf5.
On
A Friday 26 June 2009 13:46:13 Mag Gam escrigué:
Yes, you are correct!
I think this is the best path.
However, I need to learn how to append a hdf5 dataset . I looked at
this, http://code.google.com/p/h5py/wiki/FAQ#Appending_data_to_a_dataset
but was not able to do so. Do you happen to have
2009/6/25 Mag Gam magaw...@gmail.com:
Hello.
I am very new to NumPy and Python. We are doing some research in our
Physics lab and we need to store massive amounts of data (100GB
daily). I therefore, am going to use hdf5 and h5py. The problem is I
am using np.loadtxt() to create my array and
On Thu, June 25, 2009 7:59 pm, Mag Gam wrote:
I am very new to NumPy and Python. We are doing some research in our
Physics lab and we need to store massive amounts of data (100GB
daily). I therefore, am going to use hdf5 and h5py. The problem is I
am using np.loadtxt() to create my array and
Hi Tim,
Look at the thread posted a couple of weeks ago named: loadtxt and missing
values
I'm guessing you'll find answers to your questions, if not, don't hesitate
to ask.
David
2008/4/3, Tim Michelsen [EMAIL PROTECTED]:
Hello!
How can I load a data file (e.g. CSV, DAT) in ASCII which
Hello!
How can I load a data file (e.g. CSV, DAT) in ASCII which has some gaps?
The file has been saved with from a spreadsheet program which leaves
cells with not data empty:
1,23.
2,13.
3,
4,34.
Would this code be correct:
### test_loadtxt.py ###
import numpy
import maskedarray
# load
10 matches
Mail list logo
