Re: [Pytables-users] Use of recarrays as representation for Tables in memory
Hi Alvaro, I think if you save the table as a record array, it should return you a record array. Or does it return a structured array? Have you tried this? Be Well Anthony On Thu, Jun 28, 2012 at 11:22 AM, Alvaro Tejero Cantero alv...@minin.eswrote: Hi, I've noticed that tables are loaded in memory as structured arrays. It seems that returning recarrays by default would be much in the spirit of the natural naming preferences of PyTables. Is there a reason not to do so? Cheers, Álvaro. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Use of recarrays as representation for Tables in memory
I just tested: passing an object of type numpy.core.records.recarray to the constructor of createTable and then reading back it into memory via slicing (h5f.root.myobj[:] ) returns to me a numpy.ndarray. Best, -á. On Thu, Jun 28, 2012 at 5:30 PM, Anthony Scopatz scop...@gmail.com wrote: Hi Alvaro, I think if you save the table as a record array, it should return you a record array. Or does it return a structured array? Have you tried this? Be Well Anthony On Thu, Jun 28, 2012 at 11:22 AM, Alvaro Tejero Cantero alv...@minin.es wrote: Hi, I've noticed that tables are loaded in memory as structured arrays. It seems that returning recarrays by default would be much in the spirit of the natural naming preferences of PyTables. Is there a reason not to do so? Cheers, Álvaro. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Use of recarrays as representation for Tables in memory
Hmmm Ok. Maybe there needs to be a recarray flavor. I kind of like just returning a normal ndarray, though I see your argument for returning a recarray. Maybe some of the other devs can jump in here with an opinion. Be Well Anthony On Thu, Jun 28, 2012 at 12:37 PM, Alvaro Tejero Cantero alv...@minin.eswrote: I just tested: passing an object of type numpy.core.records.recarray to the constructor of createTable and then reading back it into memory via slicing (h5f.root.myobj[:] ) returns to me a numpy.ndarray. Best, -á. On Thu, Jun 28, 2012 at 5:30 PM, Anthony Scopatz scop...@gmail.com wrote: Hi Alvaro, I think if you save the table as a record array, it should return you a record array. Or does it return a structured array? Have you tried this? Be Well Anthony On Thu, Jun 28, 2012 at 11:22 AM, Alvaro Tejero Cantero alv...@minin.es wrote: Hi, I've noticed that tables are loaded in memory as structured arrays. It seems that returning recarrays by default would be much in the spirit of the natural naming preferences of PyTables. Is there a reason not to do so? Cheers, Álvaro. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Use of recarrays as representation for Tables in memory
Yes, I think it would make more sense to return a recarray too. However, I remember many time ago (3, 4 years?) that NumPy developers were recommending using structured arrays instead of recarrays. I don't remember exactly the arguments, but I think that was the reason why the structured arrays were declared the default for reading tables. But this could be changed, of course... Francesc On 6/28/12 8:25 PM, Anthony Scopatz wrote: Hmmm Ok. Maybe there needs to be a recarray flavor. I kind of like just returning a normal ndarray, though I see your argument for returning a recarray. Maybe some of the other devs can jump in here with an opinion. Be Well Anthony On Thu, Jun 28, 2012 at 12:37 PM, Alvaro Tejero Cantero alv...@minin.es mailto:alv...@minin.es wrote: I just tested: passing an object of type numpy.core.records.recarray to the constructor of createTable and then reading back it into memory via slicing (h5f.root.myobj[:] ) returns to me a numpy.ndarray. Best, -á. On Thu, Jun 28, 2012 at 5:30 PM, Anthony Scopatz scop...@gmail.com mailto:scop...@gmail.com wrote: Hi Alvaro, I think if you save the table as a record array, it should return you a record array. Or does it return a structured array? Have you tried this? Be Well Anthony On Thu, Jun 28, 2012 at 11:22 AM, Alvaro Tejero Cantero alv...@minin.es mailto:alv...@minin.es wrote: Hi, I've noticed that tables are loaded in memory as structured arrays. It seems that returning recarrays by default would be much in the spirit of the natural naming preferences of PyTables. Is there a reason not to do so? Cheers, Álvaro. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net mailto:Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net mailto:Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net mailto:Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Francesc Alted -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Use of recarrays as representation for Tables in memory
On Thu, Jun 28, 2012 at 3:23 PM, Francesc Alted fal...@pytables.org wrote: Yes, I think it would make more sense to return a recarray too. However, I remember many time ago (3, 4 years?) that NumPy developers were recommending using structured arrays instead of recarrays. I don't remember exactly the arguments, but I think that was the reason why the structured arrays were declared the default for reading tables. But this could be changed, of course... I remember this too Francesc. I don't think that this has changed, but I forgot the reasons. Maybe I'll write to the numpy list later tonight, unless someone else wants to... Francesc On 6/28/12 8:25 PM, Anthony Scopatz wrote: Hmmm Ok. Maybe there needs to be a recarray flavor. I kind of like just returning a normal ndarray, though I see your argument for returning a recarray. Maybe some of the other devs can jump in here with an opinion. Be Well Anthony On Thu, Jun 28, 2012 at 12:37 PM, Alvaro Tejero Cantero alv...@minin.eswrote: I just tested: passing an object of type numpy.core.records.recarray to the constructor of createTable and then reading back it into memory via slicing (h5f.root.myobj[:] ) returns to me a numpy.ndarray. Best, -á. On Thu, Jun 28, 2012 at 5:30 PM, Anthony Scopatz scop...@gmail.com wrote: Hi Alvaro, I think if you save the table as a record array, it should return you a record array. Or does it return a structured array? Have you tried this? Be Well Anthony On Thu, Jun 28, 2012 at 11:22 AM, Alvaro Tejero Cantero alv...@minin.es wrote: Hi, I've noticed that tables are loaded in memory as structured arrays. It seems that returning recarrays by default would be much in the spirit of the natural naming preferences of PyTables. Is there a reason not to do so? Cheers, Álvaro. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing listPytables-users@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/pytables-users -- Francesc Alted -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's
Re: [Pytables-users] Use of recarrays as representation for Tables in memory
Thank you Josh, that is representative enough. In my system the speedup of structured arrays is ~30x. A copy of the whole array is still ~6x faster. -á. On Thu, Jun 28, 2012 at 10:13 PM, Josh Ayers josh.ay...@gmail.com wrote: import time import numpy as np dtype = np.format_parser(['i4', 'i4'], [], []) N = 10 rec = np.recarray((N, ), dtype=dtype) struc = np.zeros((N, ), dtype=dtype) t1 = time.clock() for row in rec: pass print time.clock() - t1 t1 = time.clock() for row in struc: pass print time.clock() - t1 -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
