Re: [Pytables-users] Pytables bulk loading data

Thanks. I will try it out and post any findings. Pushkar On Thu, Jul 18, 2013 at 12:36 AM, Andreas Hilboll wrote: > > > > You could use pandas_ and the read_table function. There, you have nrows > and skiprows parameters with which you can easily do your own 'streaming'. > > .. _pandas: http://

Re: [Pytables-users] Pytables-users Digest, Vol 86, Issue 8

>> Date: Wed, 17 Jul 2013 16:59:16 -0500 >>> From: Anthony Scopatz >>> Subject: Re: [Pytables-users] Pytables bulk loading data >>> To: Discussion list for PyTables >>> >>> Message-ID: >>> < >>> capk-6t4ht9+ncdd_1o

Re: [Pytables-users] Pytables-users Digest, Vol 86, Issue 8

, 17 Jul 2013 16:59:16 -0500 > From: Anthony Scopatz mailto:scop...@gmail.com>> > Subject: Re: [Pytables-users] Pytables bulk loading data > To: Discussion list for PyTables > <mailto:pytables-users@lists.sourceforge.net>> &

Re: [Pytables-users] Pytables-users Digest, Vol 86, Issue 8

gt; On Wed, Jul 17, 2013 at 2:59 PM, < > pytables-users-requ...@lists.sourceforge.net> wrote: > >> Date: Wed, 17 Jul 2013 16:59:16 -0500 >> From: Anthony Scopatz >> Subject: Re: [Pytables-users] Pytables bulk loading data >> To: Discussion list for PyTables

Re: [Pytables-users] Pytables-users Digest, Vol 86, Issue 8

Thanks Antonio and Anthony. I will give this a try. -Pushkar On Wed, Jul 17, 2013 at 2:59 PM, < pytables-users-requ...@lists.sourceforge.net> wrote: > Date: Wed, 17 Jul 2013 16:59:16 -0500 > From: Anthony Scopatz > Subject: Re: [Pytables-users] Pytables bulk loading data > T

Re: [Pytables-users] Pytables bulk loading data

Hi Pushkar, I agree with Antonio. You should load your data with NumPy functions and then write back out to PyTables. This is the fastest way to do things. Be Well Anthony On Wed, Jul 17, 2013 at 2:12 PM, Antonio Valentino < antonio.valent...@tiscali.it> wrote: > Hi Pushkar, > > Il 17/07/201

Re: [Pytables-users] Pytables bulk loading data

Hi Pushkar, Il 17/07/2013 19:28, Pushkar Raj Pande ha scritto: > Hi all, > > I am trying to figure out the best way to bulk load data into pytables. > This question may have been already answered but I couldn't find what I was > looking for. > > The source data is in form of csv which may requir

[Pytables-users] Pytables bulk loading data

Hi all, I am trying to figure out the best way to bulk load data into pytables. This question may have been already answered but I couldn't find what I was looking for. The source data is in form of csv which may require parsing, type checking and setting default values if it doesn't conform to t

Re: [Pytables-users] PyTables and Multiprocessing

On Fri, Jul 12, 2013 at 1:51 AM, Mathieu Dubois wrote: > Hi Anthony, > > Thank you very much for your answer (it works). I will try to remodel my > code around this trick but I'm not sure it's possible because I use a > framework that need arrays. > I think that this method still works. You ca

Re: [Pytables-users] PyTables and Multiprocessing

Hi Anthony, Thank you very much for your answer (it works). I will try to remodel my code around this trick but I'm not sure it's possible because I use a framework that need arrays. Can somebody explain what is going on? I was thinking that PyTables keep weakref to the file for lazy loading

Re: [Pytables-users] PyTables and Multiprocessing

Hi Mathieu, I think you should try opening a new file handle per process. The following works for me on v3.0: import tables import random import multiprocessing # Reload the data # Use multiprocessing to perform a simple computation (column average) def f(filename): h5file = tables.openFi

Re: [Pytables-users] PyTables and Multiprocessing

Le 11/07/2013 21:56, Anthony Scopatz a écrit : On Thu, Jul 11, 2013 at 2:49 PM, Mathieu Dubois mailto:duboismathieu_g...@yahoo.fr>> wrote: Hello, I wanted to use PyTables in conjunction with multiprocessing for some embarrassingly parallel tasks. However, it seems that it

Re: [Pytables-users] PyTables and Multiprocessing

On Thu, Jul 11, 2013 at 2:49 PM, Mathieu Dubois wrote: > Hello, > > I wanted to use PyTables in conjunction with multiprocessing for some > embarrassingly parallel tasks. > > However, it seems that it is not possible. In the following (very > stupid) example, X is a Carray of size (100, 10) store

[Pytables-users] PyTables and Multiprocessing

Hello, I wanted to use PyTables in conjunction with multiprocessing for some embarrassingly parallel tasks. However, it seems that it is not possible. In the following (very stupid) example, X is a Carray of size (100, 10) stored in the file test.hdf5: import tables import multiprocessing #

Re: [Pytables-users] PyTables in-kernel query using Time64Col returns wrong results

l in practice? >> >> Thanks again, >> >> Charles de Villiers >> >> "They have computers, and they may have other weapons of mass >> destruction." >> (Janet Reno) >> >> -- >> *From:* Anthony Sco

Re: [Pytables-users] PyTables in-kernel query using Time64Col returns wrong results

t; > Thanks again, > > Charles de Villiers > > "They have computers, and they may have other weapons of mass destruction." > (Janet Reno) > > -- > *From:* Anthony Scopatz > *To:* Charles de Villiers ; Discussion list for > PyTables

Re: [Pytables-users] PyTables in-kernel query using Time64Col returns wrong results

: Charles de Villiers ; Discussion list for PyTables Sent: Monday, April 15, 2013 5:13 PM Subject: Re: [Pytables-users] PyTables in-kernel query using Time64Col returns wrong results Hi Charles,  We just discussed this last week and I am too lazy to retype it all so here is a link to the a

Re: [Pytables-users] PyTables in-kernel query using Time64Col returns wrong results

Hi Charles, We just discussed this last week and I am too lazy to retype it all so here is a link to the archive post [1]. Be Well Anthony 1. http://sourceforge.net/mailarchive/message.php?msg_id=30708089 On Mon, Apr 15, 2013 at 9:20 AM, Charles de Villiers wrote: > > 0down > votefavorite

[Pytables-users] PyTables in-kernel query using Time64Col returns wrong results

0 down vote favorite I'm using PyTables 2.4.0 and Python 2.7 I've got a database that contains the following typical table: /anc/asc_wind_speed (Table(87591,),shuffle,blosc(3))'Wind speed'description :={"value_seconds":Time64Col(shape=(),dflt=0.0,pos=0),"update_seconds":Time64Col(shape=(),dflt=0

Re: [Pytables-users] Pytables-users Digest, Vol 81, Issue 4

gt; You can reach the person managing the list at >>> pytables-users-ow...@lists.sourceforge.net >>> >>> When replying, please edit your Subject line so it is more specific >>> than "Re: Contents of Pytables-users digest..." >>> >>> >>> Today's To

Re: [Pytables-users] Pytables-users Digest, Vol 80, Issue 4

can reach the person managing the list at >>> pytables-users-ow...@lists.sourceforge.net >>> >>> When replying, please edit your Subject line so it is more specific >>> than "Re: Contents of Pytables-users digest..." >>> >>> >

Re: [Pytables-users] Pytables-users Digest, Vol 80, Issue 3

On Thu, Jan 3, 2013 at 2:17 PM, David Reed wrote: > Thanks a lot for the help so far guys! > > Looking at itertools, I found what I believe to be the perfect function > for what I need, itertools.combinations. This appears to be a valid > replacement to the method proposed. > Yes, combinations i

Re: [Pytables-users] Pytables-users Digest, Vol 80, Issue 4

it is more specific >> than "Re: Contents of Pytables-users digest..." >> >> >> Today's Topics: >> >>1. Re: Pytables-users Digest, Vol 80, Issue 2 (David Reed) >>2. Re: Pytables-users Digest, Vol 80, Issue 3 (David Reed) >> >> &

Re: [Pytables-users] Pytables-users Digest, Vol 80, Issue 4

st, Vol 80, Issue 2 (David Reed) >2. Re: Pytables-users Digest, Vol 80, Issue 3 (David Reed) > > > ------ > > Message: 1 > Date: Thu, 3 Jan 2013 13:44:29 -0500 > From: David Reed > Subject: Re: [Pytables-users] Pytables-users Digest, Vol 80, Issue 2 > To: pytables-users@lists.sourc

Re: [Pytables-users] Pytables-users Digest, Vol 80, Issue 3

Thanks a lot for the help so far guys! Looking at itertools, I found what I believe to be the perfect function for what I need, itertools.combinations. This appears to be a valid replacement to the method proposed. There is a small problem that I didn't mention is that my compare function actuall

Re: [Pytables-users] Pytables-users Digest, Vol 80, Issue 2

Thanks Anthony, but unless Im missing something I don't think that method will work since this will only be comparing the ith element with ith+1 element. I still need 2 for loops right? Using itertools might speed things up though, I've never used them so I will give it a shot and let you know ho

Re: [Pytables-users] pytables: could not find the HDF5 runtime

Thanks Anthony. I will check it out. Cheers, Jennifer -- LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial Remotely access PCs and mobile devices and provide instant support Improve your efficiency, a

Re: [Pytables-users] pytables: could not find the HDF5 runtime

Hi Jennifer, Yeah, that is right, they are not in EPD Free. However, they are in Anaconda CE (http://continuum.io/downloads.html). Note the CE rather than the full version. Be Well Anthony On Mon, Dec 10, 2012 at 4:07 PM, Jennifer Flegg wrote: > Hi Anthony, > Thanks for your reply. I installe

Re: [Pytables-users] pytables: could not find the HDF5 runtime

Hi Anthony, Thanks for your reply. I installed HDF5 also from source. The reason I'm building hdf5 and pytables myself is that they don't seem to be available through EPD any more (at least in the free version: http://www.enthought.com/products/epdlibraries.php) They used to both come bundled in

Re: [Pytables-users] pytables: could not find the HDF5 runtime

Hi Jennifer, Oh, right, I am sorry. Your end error message looks very similar to another, more common issue. How did you install HDF5? On Mac I typically use MacPorts or have to install it from source. IIRC the macports build fails to make the shared libraries and you typically have to configu

Re: [Pytables-users] pytables: could not find the HDF5 runtime

HI Anthony, I'm not in the pytables source dir when I'm running IPython, so I don't think this is the problem. Thanks, Jennifer -- LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial Remotely access PCs an

Re: [Pytables-users] pytables: could not find the HDF5 runtime

Try leaving the pytables source dir and then running then running IPython. On Mon, Dec 10, 2012 at 9:20 AM, Jennifer Flegg wrote: > Hi, > I'm trying to install pytables and its proving difficult (using MAC OS > 10.6.4). > I have installed in "/usr/local/hdf5" and set the environment variable > $

[Pytables-users] pytables: could not find the HDF5 runtime

Hi, I'm trying to install pytables and its proving difficult (using MAC OS 10.6.4). I have installed in "/usr/local/hdf5" and set the environment variable $HDF5_DIR to /usr/local/hdf5. When I run setup, I get a warning about not being able to find the HDF5 runtime. ndmmac149:tables-2.4.0 jflegg

Re: [Pytables-users] Pytables 2.4.0 fails to import

On Sat, Nov 24, 2012 at 1:42 PM, Ondřej Čertík wrote: > Hi, > > I am using Ubuntu 12.04, numexpr-1.4.2, hdf5-1.8.6 and tables-2.4.0. I > am using the latest numpy 1.7.x from the release branch, in > particular, I am using > the commit 3a52aa0. > > After installing everything I get: > > $ python >

[Pytables-users] Pytables 2.4.0 fails to import

Hi, I am using Ubuntu 12.04, numexpr-1.4.2, hdf5-1.8.6 and tables-2.4.0. I am using the latest numpy 1.7.x from the release branch, in particular, I am using the commit 3a52aa0. After installing everything I get: $ python Python 2.6.4 (r264:75706, Dec 17 2011, 17:17:12) [GCC 4.6.1] on linux3 Typ

Re: [Pytables-users] Pytables-users Digest, Vol 78, Issue 6

2012. 11. 17. 오후 12:46에 님이 작성: > Send Pytables-users mailing list submissions to > pytables-users@lists.sourceforge.net > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.sourceforge.net/lists/listinfo/pytables-users > or, via email, send a message with s

Re: [Pytables-users] Pytables-users Digest, Vol 78, Issue 6

2012. 11. 17. 오후 12:46에 님이 작성: > > Send Pytables-users mailing list submissions to > pytables-users@lists.sourceforge.net > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.sourceforge.net/lists/listinfo/pytables-users > or, via email, send a message with

Re: [Pytables-users] PyTables broke in Ubuntu 12.10

I just tried installing python-tables on a clean install of 12.10 on a different machine and all went fine. So I've got something corrupted on my machine...just a localized bug. Jason On Fri, Oct 26, 2012 at 2:40 PM, Jason Moore wrote: > I've posted a bug report here: > https://bugs.launchpad.n

Re: [Pytables-users] PyTables broke in Ubuntu 12.10

I've posted a bug report here: https://bugs.launchpad.net/ubuntu/+source/pytables/+bug/1071918 Maybe others could see if it is reproducible in Ubuntu 12.10. Thanks, Jason On Fri, Oct 26, 2012 at 2:09 PM, Jason Moore wrote: > The simlink is a workaround for Ubuntu 12.10. It is certainly not th

Re: [Pytables-users] PyTables broke in Ubuntu 12.10

The simlink is a workaround for Ubuntu 12.10. It is certainly not the long term solution, but I don't see why it is a bad idea. python-tables in the Ubuntu 12.10 repos cannot find the HDF5 library because it is looking for libhdf5.so.6 on the path but there is only libhdf5.so.7 (which is a simlink

Re: [Pytables-users] PyTables broke in Ubuntu 12.10

Hi Jason, Il 26/10/2012 21:59, Jason Moore ha scritto: > Solution was simple once I found it. Here is the workaround: > > https://bugs.launchpad.net/ubuntu/+source/octave/+bug/1005243 > > Just make a symlink to the new file. > > Jason Honestly I don't think it is a good idea. ciao -- Antonio

Re: [Pytables-users] PyTables broke in Ubuntu 12.10

Hi Jason, Il 26/10/2012 21:37, Jason Moore ha scritto: > I'll post the bug report, but I'd like to get this working on my system. > I've always had trouble compiling pytables from source due to the > dependencies. Right now I just need to get this working because I can no > longer use my software

Re: [Pytables-users] PyTables broke in Ubuntu 12.10

Solution was simple once I found it. Here is the workaround: https://bugs.launchpad.net/ubuntu/+source/octave/+bug/1005243 Just make a symlink to the new file. Jason On Fri, Oct 26, 2012 at 12:37 PM, Jason Moore wrote: > I'll post the bug report, but I'd like to get this working on my system.

Re: [Pytables-users] PyTables broke in Ubuntu 12.10

I'll post the bug report, but I'd like to get this working on my system. I've always had trouble compiling pytables from source due to the dependencies. Right now I just need to get this working because I can no longer use my software now that PyTabes is broken. Question 1: What are the exact com

Re: [Pytables-users] PyTables broke in Ubuntu 12.10

Hi Jason, Il 26/10/2012 18:44, Jason Moore ha scritto: > Where exactly do I submit the bug report? There doesn't seem to be a bug > option here: https://launchpad.net/ubuntu/quantal/+package/python-tables > > Jason of course you need a launchpad account, than you can follow instruction of the Re

Re: [Pytables-users] PyTables broke in Ubuntu 12.10

Where exactly do I submit the bug report? There doesn't seem to be a bug option here: https://launchpad.net/ubuntu/quantal/+package/python-tables Jason On Thu, Oct 25, 2012 at 11:27 PM, Antonio Valentino < antonio.valent...@tiscali.it> wrote: > Hi Jason, > > Il giorno 26/ott/2012, alle ore 07:28

Re: [Pytables-users] PyTables broke in Ubuntu 12.10

Hi Jason, Il giorno 26/ott/2012, alle ore 07:28, Jason Moore ha scritto: > So it looks like python-tables in Ubuntu 12.10 requires libhdf5-7 and > libhdf5-7 has /usr/lib/libhdf5.so.7 not libhdf5.so.6. Correct, the hdf5 package has been updated in ubuntu 12.10. If you use the standard python-t

Re: [Pytables-users] PyTables broke in Ubuntu 12.10

So it looks like python-tables in Ubuntu 12.10 requires libhdf5-7 and libhdf5-7 has /usr/lib/libhdf5.so.7 not libhdf5.so.6. Jason On Thu, Oct 25, 2012 at 4:10 PM, Aquil Abdullah wrote: > Can you check to see if libhdf5.so is in your p

Re: [Pytables-users] PyTables broke in Ubuntu 12.10

Can you check to see if libhdf5.so is in your path? If not, you can add it to the path where it resides to your PATH variable. Hopefully, that helps. Aquil H. Abdullah On Oct 25, 2012, at 18:42, Jason Moore wrote: I just updated to Ubuntu 12.10 and my pytables install is broken. I reinstalled

[Pytables-users] PyTables broke in Ubuntu 12.10

I just updated to Ubuntu 12.10 and my pytables install is broken. I reinstalled and it seems like I have hdf5 1.8.4 installed but I get this error: moorepants@moorepants-LT:BicycleDataProcessor(master)$ vitables InstrumentedBicycleData.h5 Traceback (most recent call last): File "/usr/bin/vitable

Re: [Pytables-users] pyTables modifies data

Hi Jernej, This is a floating point representation issue. With 32 bits, the representations of the numbers you want are as given. You can see the same issue if you look closely with numpy. Observe the last couple of digits: In [1]: import numpy as np In [2]: np.array([12.12, 12.1234567], 'f4'

[Pytables-users] pyTables modifies data

Hi. pyTables saves wrong data to file. I read the docs, but I don't know what I'm doing wrong. It's the first time I noticed this kind of behavior. Any suggestions? *problem: * writing items in testL = [12.12, 12.1234567] to table data in table: col1 = 12.118856, col2 = 12.1234569550 expected:

Re: [Pytables-users] PyTables data files for a tutorial

Hey, thanks to everybody that contributed datasets! I'll look into them and hope to be able to select something to show. Francesc On 10/21/12 10:55 PM, Jason Moore wrote: > This is a PyTables generated file with data collected from vehicle > (bicycle) dynamics measurements. Meta data are in ta

Re: [Pytables-users] PyTables data files for a tutorial

This is a PyTables generated file with data collected from vehicle (bicycle) dynamics measurements. Meta data are in tables and time series are stored in array objects. http://mae.ucdavis.edu/~biosport/InstrumentedBicycleData/InstrumentedBicycleData.h5.bz2 It is about 308 mb compressed and 610 mb

Re: [Pytables-users] PyTables data files for a tutorial

On Sun, Oct 21, 2012 at 10:41 AM, Francesc Alted wrote: > Hi, > > I'm going to give a tutorial on PyTables next Thursday during the PyData > conference in New York (http://nyc2012.pydata.org/) and I'd like to use > some real life data files. So, if you have some public repository with > data gen

Re: [Pytables-users] PyTables data files for a tutorial

Hello Francesc, I look forward to your pydata hearing how your tutorial goes! Here [1] is a file that stores some basic nuclear data that is freely redistributable. It stores atomic weights, bound neutron scattering lengths, and pre-compiled neutron cross sections (xs) for 5 different energy reg

[Pytables-users] PyTables data files for a tutorial

Hi, I'm going to give a tutorial on PyTables next Thursday during the PyData conference in New York (http://nyc2012.pydata.org/) and I'd like to use some real life data files. So, if you have some public repository with data generated with PyTables, please tell me. I'm looking for files that

Re: [Pytables-users] PyTables hangs while opening file in worker process

Hmm sorry to hear that Owen Let me know how it goes. On Thu, Oct 11, 2012 at 11:07 AM, Owen Mackwood < owen.mackw...@bccn-berlin.de> wrote: > Hi Anthony, > > I tried your suggestion and it has not solved the problem. It could be > that it makes the problem go away in the test code because it

Re: [Pytables-users] PyTables hangs while opening file in worker process

Hi Anthony, I tried your suggestion and it has not solved the problem. It could be that it makes the problem go away in the test code because it changes the timing of the processes. I'll see if I can modify the test code to reproduce the hang even with reloading the tables module. Regards, Owen

Re: [Pytables-users] PyTables hangs while opening file in worker process

So Owen, I am still not sure what the underlying problem is, but I altered your parallel function to forciably reload pytables each time it is called. This seemed to work perfectly on my larger system but not at all on my smaller one. If there is a way that you can isolate pytables and not impor

Re: [Pytables-users] PyTables hangs while opening file in worker process

On 10 October 2012 20:08, Anthony Scopatz wrote: > So just to confirm this behavior, having run your sample on a couple of my > machines, what you see is that the code looks like it gets all the way to > the end, and then it stalls right before it is about to exit, leaving some > small number of

Re: [Pytables-users] PyTables hangs while opening file in worker process

Hi Owen, So just to confirm this behavior, having run your sample on a couple of my machines, what you see is that the code looks like it gets all the way to the end, and then it stalls right before it is about to exit, leaving some small number of processes (here names python tables_test.py) in t

Re: [Pytables-users] PyTables hangs while opening file in worker process

Hi Anthony, I've created a reduced example which reproduces the error. I suppose the more processes you can run in parallel the more likely it is you'll see the hang. On a machine with 8 cores, I see 5-6 processes hang out of 2000. All of the hung tasks had a call stack that looked like this: #0

Re: [Pytables-users] PyTables hangs while opening file in worker process

On Mon, Oct 8, 2012 at 11:19 AM, Owen Mackwood wrote: > Hi Anthony, > > On 8 October 2012 15:54, Anthony Scopatz wrote: > >> Hmmm, Are you actually copying the data (f.root.data[:]) or are you >> simply passing a reference as arguments (f.root.data)? >> > > I call f.root.data.read() on any arra

Re: [Pytables-users] PyTables hangs while opening file in worker process

Hi Anthony, On 8 October 2012 15:54, Anthony Scopatz wrote: > Hmmm, Are you actually copying the data (f.root.data[:]) or are you > simply passing a reference as arguments (f.root.data)? > I call f.root.data.read() on any arrays to load them into the process target args dictionary. I had assum

Re: [Pytables-users] PyTables hangs while opening file in worker process

On Mon, Oct 8, 2012 at 5:13 AM, Owen Mackwood wrote: > Hi Anthony, > > There is a single multiprocessing.Pool which usually has 6-8 processes, > each of which is used to run a single task, after which a new process is > created for the next task (maxtasksperchild=1 for the Pool constructor). > The

Re: [Pytables-users] PyTables hangs while opening file in worker process

Hi Anthony, There is a single multiprocessing.Pool which usually has 6-8 processes, each of which is used to run a single task, after which a new process is created for the next task (maxtasksperchild=1 for the Pool constructor). There is a master process that regularly opens an HDF5 file to read

Re: [Pytables-users] PyTables hangs while opening file in worker process

Hi Owen, How many pools do you have? Is this a random runtime failure? What kind of system is this one? Is there some particular fucntion in Python that you are running? (It seems to be openFile(), but I can't be sure...) The error is definitely happening down in the H5open() routine. Now wh

Re: [Pytables-users] PyTables hangs while opening file in worker process

Hi Anthony, I'm not trying to write in parallel. Each worker process has its own file to write to. After all tasks are completed, I collect the results in the master process. So the problem I'm seeing (a hang in the worker process) shouldn't have anything to do with parallel writes. Do you have an

Re: [Pytables-users] PyTables hangs while opening file in worker process

Hello Owen, While you can use process pools to read from a file in parallel just fine, writing is another story completely. While HDF5 itself supports parallel writing though MPI, this comes at the high cost of compression no longer being available and a much more complicated code base. So for t

[Pytables-users] PyTables hangs while opening file in worker process

Hello, I'm using a multiprocessing.Pool to parallelize a set of tasks which record their results into separate hdf5 files. Occasionally (less than 2% of the time) the worker process will hang. According to gdb, the problem occurs while opening the hdf5 file, when it attempts to obtain the associat

Re: [Pytables-users] PyTables Simultaneous Read Write from Current File

Jacob, I just put together a small example demonstrating this. You can find it in the develop branch of the PyTables repository. https://github.com/PyTables/PyTables/blob/develop/examples/multiprocess_access_queues.py It's somewhat limited, because all the client processes have to be known at t

Re: [Pytables-users] Pytables file reading

Hi Antonio, Last question about this, from pytables point of view and based on your experience, is it better to manage a table with 3 million rows and multidimensional cells or a table with 300 million rows and plain cells? Thank you, Juanma El Aug 5, 2012, a las 17:32, Antonio Valentino es

Re: [Pytables-users] Pytables file reading

Thank you Antonio, I will try Cheers Juanma El Aug 5, 2012, a las 17:32, Antonio Valentino escribió: > Hi Juan Manuel, > > Il 05/08/2012 22:52, Juan Manuel Vázquez Tovar ha scritto: >> Hi Antonio, >> >> This is the piece of code I use to read the part of the table I need: >> >> data = [cas

Re: [Pytables-users] Pytables file reading

Hi Juan Manuel, Il 05/08/2012 22:52, Juan Manuel Vázquez Tovar ha scritto: > Hi Antonio, > > This is the piece of code I use to read the part of the table I need: > > data = [case[´loads´][i] for case in table] > > where i is the index of the row that I need to read from the matrix (133x6) > st

Re: [Pytables-users] Pytables file reading

Hi Antonio, This is the piece of code I use to read the part of the table I need: data = [case[´loads´][i] for case in table] where i is the index of the row that I need to read from the matrix (133x6) stored in each cell of the column "loads". Juanma 2012/8/5 Antonio Valentino > Hi Juan Man

Re: [Pytables-users] Pytables file reading

Hi Juan Manuel, Il 05/08/2012 22:28, Juan Manuel Vázquez Tovar ha scritto: > Hi Antonio, > > You are right, I don´t need to load the entire table into memory. > The fourth column has multidimensional cells and when I read a single row > from every cell in the column, I almost fill the workstation

Re: [Pytables-users] Pytables file reading

Hi Antonio, You are right, I don´t need to load the entire table into memory. The fourth column has multidimensional cells and when I read a single row from every cell in the column, I almost fill the workstation memory. I didn´t expect that process to use so much memory, but the fact is that it u

Re: [Pytables-users] Pytables file reading

Hi Juan Manuel, Il 04/08/2012 01:55, Juan Manuel Vázquez Tovar ha scritto: > Hello all, > > I´m managing a file close to 26 Gb size. It´s main structure is a table > with a bit more than 8 million rows. The table is made by four columns, the > first two columns store names, the 3rd one has a 53

[Pytables-users] Pytables file reading

Hello all, I´m managing a file close to 26 Gb size. It´s main structure is a table with a bit more than 8 million rows. The table is made by four columns, the first two columns store names, the 3rd one has a 53 items array in each cell and the last column has a 133x6 matrix in each cell. I use to

Re: [Pytables-users] Pytables file structure

Hello Juan, Just make an account at github [1] and then go to the PyTables issues page. Be Well Anthony 1. https://github.com/ 2. https://github.com/PyTables/PyTables/issues On Tue, Jul 17, 2012 at 6:27 PM, Juan Manuel Vázquez Tovar < jmv.to...@gmail.com> wrote: > Thank you very much Anthony.

Re: [Pytables-users] Pytables file structure

Thank you very much Anthony. Do I have to sign up to store a ticket? 2012/7/15 Anthony Scopatz > Ahh I see, tricky. > > So I think what is killing you is that you are pulling each row of the > table individually over the network. Ideally you should be able to do > something like the following:

Re: [Pytables-users] [pytables-dev] SciPy 2012 Tutorial

On Tue, Jul 17, 2012 at 2:43 AM, Alvaro Tejero Cantero wrote: > It is a very nice presentation. > > Makes me wonder if using the terminology > > 'in memory' for 'in-core' and 'in disk' for 'out of core' would not be > more straightforward! > Thanks Alvaro! I agree the existing terminology here i

Re: [Pytables-users] [pytables-dev] SciPy 2012 Tutorial

Hey Anthony, I was not there, but judging by the slices, this should have been a very nice tutorial. Some remarks: - In slide 19, you state that, if data comes from datasets with the 'numpy' flavor, they can be accessed in a numpy-like fashion. In fact, you should be able to access data thi

Re: [Pytables-users] [pytables-dev] SciPy 2012 Tutorial

It is a very nice presentation. Makes me wonder if using the terminology 'in memory' for 'in-core' and 'in disk' for 'out of core' would not be more straightforward! -á. On 17 July 2012 06:46, Anthony Scopatz wrote: > Hello PyTables, > > I'd like to present the tutorial I gave at SciPy 2012 t

Re: [Pytables-users] PyTables Simultaneous Read Write from Current File

On Mon, Jul 16, 2012 at 3:30 PM, Jacob Bennett wrote: > Wait, is there perhaps a way to simulataneously read and write without any > kind of blocking? Perhaps the "a" mode or the "r+" mode might help for > simultaneous read/write? I am currently implementing the > multithreading.Queue, but I think

Re: [Pytables-users] PyTables Simultaneous Read Write from Current File

Wait, is there perhaps a way to simulataneously read and write without any kind of blocking? Perhaps the "a" mode or the "r+" mode might help for simultaneous read/write? I am currently implementing the multithreading.Queue, but I think that a large number of query requests might put an necessary l

Re: [Pytables-users] Pytables file structure

Ahh I see, tricky. So I think what is killing you is that you are pulling each row of the table individually over the network. Ideally you should be able to do something like the following: f.root.table.cols.my_col[:,n,:] using numpy-esque multidimensional slicing. However, this fails when I

Re: [Pytables-users] Pytables file structure

The column I´m requesting the data from has multidimensional cells, so each time I request data from the table, I need to get a specific row for all the multidimensional cells in the column. I hope this clarifies a bit. I have at the office a Linux workstation, but it is part of a computing cluster

Re: [Pytables-users] Pytables file structure

Rereading the original post, I am a little confused are your trying to read the whole table, just a couple of rows that meet some condition, or just one whole column, or one part of the column. To request the whole table without looping over each row in Python, index every element: f.root.table[:

Re: [Pytables-users] Pytables file structure

Hello Anthony, I have to loop over the whole set of rows. Does the where method has any advantages in that case? Thank you, Juanma 2012/7/15 Anthony Scopatz > Hello Juan, > > Try using the where() method [1], It has a lot of nice features under the > covers. > > Be Well > Anthony > > 1. > htt

Re: [Pytables-users] Pytables file structure

Hello Juan, Try using the where() method [1], It has a lot of nice features under the covers. Be Well Anthony 1. http://pytables.github.com/usersguide/libref.html?highlight=where#tables.Table.where On Sun, Jul 15, 2012 at 4:01 PM, Juan Manuel Vázquez Tovar < jmv.to...@gmail.com> wrote: > Hell

[Pytables-users] Pytables file structure

Hello, I have been using pytables for a few moths. The main structure of my files has a four column table, two of which have multidimensional cells, (56,1) and (133,6) respectively. The previous structure had more columns instead of storing the 56x1 array into the same cell. The largest file has a

Re: [Pytables-users] PyTables Simultaneous Read Write from Current File

+1 to example of this! On Sat, Jul 14, 2012 at 1:36 PM, Jacob Bennett wrote: > Awesome, I think this sounds like a very workable solution and the idea is > very neat. I will try to implement this right away. I definitely agree to > putting a small example. > > Let you know how this works, thanks

Re: [Pytables-users] PyTables Simultaneous Read Write from Current File

Awesome, I think this sounds like a very workable solution and the idea is very neat. I will try to implement this right away. I definitely agree to putting a small example. Let you know how this works, thanks guys! Thanks, Jacob On Sat, Jul 14, 2012 at 2:36 AM, Antonio Valentino < antonio.valen

Re: [Pytables-users] PyTables Simultaneous Read Write from Current File

Hi all, Il 14/07/2012 00:44, Josh Ayers ha scritto: > My first instinct would be to handle all access (read and write) to > that file from a single process. You could create two > multiprocessing.Queue objects, one for data to write and one for read > requests. Then the process would check the qu

Re: [Pytables-users] PyTables Simultaneous Read Write from Current File

My first instinct would be to handle all access (read and write) to that file from a single process. You could create two multiprocessing.Queue objects, one for data to write and one for read requests. Then the process would check the queues in a loop and handle each request serially. The data r

Re: [Pytables-users] PyTables Simultaneous Read Write from Current File

On Fri, Jul 13, 2012 at 2:09 PM, Jacob Bennett wrote: [snip] > My first implementation was to have a set of current files stay in write > mode and have an overall lock over these files for the current day, but > (stupidly) I forgot that lock instances cannot be shared over separate > processes,

[Pytables-users] PyTables Simultaneous Read Write from Current File

Hello PyTables Discussion, Could you perhaps give me the best advice on read/write from a single file in PyTables? I currently have a parsing system that accepts a stream of data and consistently writes the data to pytables; however, on the other hand, I have an independent server that accepts req

Re: [Pytables-users] [Pytables-announce] ANN: PyTables 2.4.0 beta1

On 7/11/12 7:06 PM, Antonio Valentino wrote: > >> BTW, which is the status of the 3.x support? I vaguely remember >> you asking me for some help on this, but I don't remember well. Not >> that I have a lot of time to spend on it, but perhaps I can use >> some hours in the next days. > well, in PyT

  1   2   3   4   >