A Divendres 15 Juny 2007 20:28, Conor Robinson escrigué: > Hello, > > I'm trying to append a single table from multiple arrays at the same > time (all arrays have same column types). I was wondering if this has > been covered before.
I don't quite understand what you mean exactly. If what you want is to append several rows in one shot, you absolutely can do this by using the Table.append() method which accepts a rich variety of inputs (take a look at the docs). This way of feeding data to a table is one of the fastest around indeed. > Furthermore, would it be faster to create > separate tables for each source (column types equivalent across the > individual tables) and then merge the tables together at the end, if > this is possible? I just don't know, but you can always check this with a small test. > I compiled hdf5 with mpi as well as pytables with pympi, although this > may not be relevant here. > > I searched "parallel" in the archives and came up empty. > Sorry, but PyTables doesn't support paralellism at all at this time. > Problem: > > All arrays will have the same column types, they're created from the > same data source. At the end I would like to write them into the same > table, in parallel, thus reducing the write bottle neck. My advice is: try to consolidate your data on a single buffer (it can be a NumPy recarray, for example) and then feed the table with this buffer using Table.append(). My guts are saying that, even in the hypotetical case that PyTables would support parallelism (which is not the case, as I've already said), this approach would be as fast as writing in parallel (unless you are writing into a parallel filesystem, do you?). Cheers, -- >0,0< Francesc Altet http://www.carabos.com/ V V Cárabos Coop. V. Enjoy Data "-" ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
