Thanks Francesc. Does Table.copy() supports start and stop? I need to copy 4mm rows out of a table of 15mm rows. So copying the whole table and then deleting rows is not optimal either I suspect.
I will file a ticket. Best, Koert -----Original Message----- From: Francesc Alted [mailto:fal...@pytables.org] Sent: September 27 2010 10:27 To: pytables-users@lists.sourceforge.net Subject: Re: [Pytables-users] Copying block of rows between tables A Monday 27 September 2010 14:24:21 Koert Kuipers escrigué: > Hello all, > I need to copy a large block of rows (start=1,000,000 and > stop=5,000,000) from table a to table b in the same file. Looking at > the pytables documentation I came up with a few options. > > Option 1: > for row in a.iterrows(start, stop): > b.append([row[:]]) > > However a loop in python over millions of rows doesn't seem optimal. > > Option 2: > b.append(a.read(start, stop)) > > All dthe ata is first copied from a to memory before being written to > b, which is not scalable. > > Option 3: > Use whereAppend() > > This seems the best (assuming the copying of rows from table to table > is done much more efficiently in C than I could do in Python). > However whereAppend doesn't want to run without a condition, and I > do not have a condition, just start and stop parameters.... Well, if you want to do some timings on whereAppend() you can always use some trivial condition like 'col1 == col1' or similar. But I agree that `condition` should support a ``None`` value meaning ``without condition``. Could you please add a ticket on this? > Did I miss the obvious alternative? Any suggestions? Well, I'd say that you forgot the most obvious one, i.e. `Table.copy()` :-) -- Francesc Alted ------------------------------------------------------------------------------ Start uncovering the many advantages of virtual appliances and start using them to simplify application deployment and accelerate your shift to cloud computing. http://p.sf.net/sfu/novell-sfdev2dev _______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users ------------------------------------------------------------------------------ Start uncovering the many advantages of virtual appliances and start using them to simplify application deployment and accelerate your shift to cloud computing. http://p.sf.net/sfu/novell-sfdev2dev _______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
