Re: [Pytables-users] How Fast is File.__contains, File.getNode, File.createTable?
Hi Jacob, This is not a solely PyTables issue. As described the methods you mention all involve attribute (or metadata) access, which is notaoriously slow in HDF5. Or rather, much slower that read/write from the datasets (Tables, Arrays) themselves.Generally, having a single table with 3E8 rows will be faster than searching through 3E3 tables with 1E5 rows.If there is any way you can represent you data in a sane way to have larger tables, I would recommend that you try this. The other option too is to simply have an initialization step where you create the all of the tables and then another loop where you append to all of them, rather than searching through 3000 tables 3000 times. For example: for i in range(3000): f.root.createTable(i + str(i)) for i in range(3000): tab = f.getNode(/i + str(i)) tab.append(...) In the above pseudocode, __contains__ is never called - let alone calling it 3 times, like in your previous email. In effect the time that you are spending searching in your previous email is 3000 tables x 3000 loop iterations times 3 if-else branches.So you are automatically in a 9 - 27 million iteration, just by the way you have been using contains. I really think that pre-creating the tables so that you *know* that they are there and just have to get the nodes will be far faster for you. Be Well Anthony On Wed, Jun 27, 2012 at 2:33 PM, Jacob Bennett jacob.bennet...@gmail.comwrote: Hello PyTables Users, I am asking this quick question because my application is currently horribly bottlenecking on these methods, all of which are called once before each Table.append(rows). The table writing on the other hand is much, much faster than the searching for the table. Any general discussion on this would be great. The current hierarchy consists of root leading to around 3000 nodes each of which have around 10 rows. Thanks, Jacob -- Jacob Bennett Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science Class of 2014| benne...@mit.edu -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] How Fast is File.__contains, File.getNode, File.createTable?
On Thu, Jun 28, 2012 at 10:41 AM, Jacob Bennett jacob.bennet...@gmail.comwrote: Hey Anthony, Awesome, I think I'm going to take your advice for aiming towards larger tables. Just an inquiry though, let's say you keep track of a dictionary/hashtable that maps node identifiers (keys) to instances of the node object (values) which can be assigned during node creation. ie* mydict['id'] = thisFile.createTable(params). I think this could actually help get away from the expensive search calls. Yup. This would probably help a lot. I hadn't even considered it. I guess you learn something new everyday ;) I'm still going to go with larger tables though, since I have to read the data eventually. Sounds good! Fee free to ask further questions here! Be Well Anthony Thanks Again For Your Time, Jacob On Thu, Jun 28, 2012 at 10:16 AM, Anthony Scopatz scop...@gmail.comwrote: Hi Jacob, This is not a solely PyTables issue. As described the methods you mention all involve attribute (or metadata) access, which is notaoriously slow in HDF5. Or rather, much slower that read/write from the datasets (Tables, Arrays) themselves.Generally, having a single table with 3E8 rows will be faster than searching through 3E3 tables with 1E5 rows.If there is any way you can represent you data in a sane way to have larger tables, I would recommend that you try this. The other option too is to simply have an initialization step where you create the all of the tables and then another loop where you append to all of them, rather than searching through 3000 tables 3000 times. For example: for i in range(3000): f.root.createTable(i + str(i)) for i in range(3000): tab = f.getNode(/i + str(i)) tab.append(...) In the above pseudocode, __contains__ is never called - let alone calling it 3 times, like in your previous email. In effect the time that you are spending searching in your previous email is 3000 tables x 3000 loop iterations times 3 if-else branches.So you are automatically in a 9 - 27 million iteration, just by the way you have been using contains. I really think that pre-creating the tables so that you *know* that they are there and just have to get the nodes will be far faster for you. Be Well Anthony On Wed, Jun 27, 2012 at 2:33 PM, Jacob Bennett jacob.bennet...@gmail.com wrote: Hello PyTables Users, I am asking this quick question because my application is currently horribly bottlenecking on these methods, all of which are called once before each Table.append(rows). The table writing on the other hand is much, much faster than the searching for the table. Any general discussion on this would be great. The current hierarchy consists of root leading to around 3000 nodes each of which have around 10 rows. Thanks, Jacob -- Jacob Bennett Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science Class of 2014| benne...@mit.edu -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Jacob Bennett Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science Class of 2014| benne...@mit.edu -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual
Re: [Pytables-users] Use of recarrays as representation for Tables in memory
Hi Alvaro, I think if you save the table as a record array, it should return you a record array. Or does it return a structured array? Have you tried this? Be Well Anthony On Thu, Jun 28, 2012 at 11:22 AM, Alvaro Tejero Cantero alv...@minin.eswrote: Hi, I've noticed that tables are loaded in memory as structured arrays. It seems that returning recarrays by default would be much in the spirit of the natural naming preferences of PyTables. Is there a reason not to do so? Cheers, Álvaro. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Use of recarrays as representation for Tables in memory
I just tested: passing an object of type numpy.core.records.recarray to the constructor of createTable and then reading back it into memory via slicing (h5f.root.myobj[:] ) returns to me a numpy.ndarray. Best, -á. On Thu, Jun 28, 2012 at 5:30 PM, Anthony Scopatz scop...@gmail.com wrote: Hi Alvaro, I think if you save the table as a record array, it should return you a record array. Or does it return a structured array? Have you tried this? Be Well Anthony On Thu, Jun 28, 2012 at 11:22 AM, Alvaro Tejero Cantero alv...@minin.es wrote: Hi, I've noticed that tables are loaded in memory as structured arrays. It seems that returning recarrays by default would be much in the spirit of the natural naming preferences of PyTables. Is there a reason not to do so? Cheers, Álvaro. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Use of recarrays as representation for Tables in memory
Hmmm Ok. Maybe there needs to be a recarray flavor. I kind of like just returning a normal ndarray, though I see your argument for returning a recarray. Maybe some of the other devs can jump in here with an opinion. Be Well Anthony On Thu, Jun 28, 2012 at 12:37 PM, Alvaro Tejero Cantero alv...@minin.eswrote: I just tested: passing an object of type numpy.core.records.recarray to the constructor of createTable and then reading back it into memory via slicing (h5f.root.myobj[:] ) returns to me a numpy.ndarray. Best, -á. On Thu, Jun 28, 2012 at 5:30 PM, Anthony Scopatz scop...@gmail.com wrote: Hi Alvaro, I think if you save the table as a record array, it should return you a record array. Or does it return a structured array? Have you tried this? Be Well Anthony On Thu, Jun 28, 2012 at 11:22 AM, Alvaro Tejero Cantero alv...@minin.es wrote: Hi, I've noticed that tables are loaded in memory as structured arrays. It seems that returning recarrays by default would be much in the spirit of the natural naming preferences of PyTables. Is there a reason not to do so? Cheers, Álvaro. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Use of recarrays as representation for Tables in memory
Yes, I think it would make more sense to return a recarray too. However, I remember many time ago (3, 4 years?) that NumPy developers were recommending using structured arrays instead of recarrays. I don't remember exactly the arguments, but I think that was the reason why the structured arrays were declared the default for reading tables. But this could be changed, of course... Francesc On 6/28/12 8:25 PM, Anthony Scopatz wrote: Hmmm Ok. Maybe there needs to be a recarray flavor. I kind of like just returning a normal ndarray, though I see your argument for returning a recarray. Maybe some of the other devs can jump in here with an opinion. Be Well Anthony On Thu, Jun 28, 2012 at 12:37 PM, Alvaro Tejero Cantero alv...@minin.es mailto:alv...@minin.es wrote: I just tested: passing an object of type numpy.core.records.recarray to the constructor of createTable and then reading back it into memory via slicing (h5f.root.myobj[:] ) returns to me a numpy.ndarray. Best, -á. On Thu, Jun 28, 2012 at 5:30 PM, Anthony Scopatz scop...@gmail.com mailto:scop...@gmail.com wrote: Hi Alvaro, I think if you save the table as a record array, it should return you a record array. Or does it return a structured array? Have you tried this? Be Well Anthony On Thu, Jun 28, 2012 at 11:22 AM, Alvaro Tejero Cantero alv...@minin.es mailto:alv...@minin.es wrote: Hi, I've noticed that tables are loaded in memory as structured arrays. It seems that returning recarrays by default would be much in the spirit of the natural naming preferences of PyTables. Is there a reason not to do so? Cheers, Álvaro. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net mailto:Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net mailto:Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net mailto:Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Francesc Alted -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Use of recarrays as representation for Tables in memory
On Thu, Jun 28, 2012 at 3:23 PM, Francesc Alted fal...@pytables.org wrote: Yes, I think it would make more sense to return a recarray too. However, I remember many time ago (3, 4 years?) that NumPy developers were recommending using structured arrays instead of recarrays. I don't remember exactly the arguments, but I think that was the reason why the structured arrays were declared the default for reading tables. But this could be changed, of course... I remember this too Francesc. I don't think that this has changed, but I forgot the reasons. Maybe I'll write to the numpy list later tonight, unless someone else wants to... Francesc On 6/28/12 8:25 PM, Anthony Scopatz wrote: Hmmm Ok. Maybe there needs to be a recarray flavor. I kind of like just returning a normal ndarray, though I see your argument for returning a recarray. Maybe some of the other devs can jump in here with an opinion. Be Well Anthony On Thu, Jun 28, 2012 at 12:37 PM, Alvaro Tejero Cantero alv...@minin.eswrote: I just tested: passing an object of type numpy.core.records.recarray to the constructor of createTable and then reading back it into memory via slicing (h5f.root.myobj[:] ) returns to me a numpy.ndarray. Best, -á. On Thu, Jun 28, 2012 at 5:30 PM, Anthony Scopatz scop...@gmail.com wrote: Hi Alvaro, I think if you save the table as a record array, it should return you a record array. Or does it return a structured array? Have you tried this? Be Well Anthony On Thu, Jun 28, 2012 at 11:22 AM, Alvaro Tejero Cantero alv...@minin.es wrote: Hi, I've noticed that tables are loaded in memory as structured arrays. It seems that returning recarrays by default would be much in the spirit of the natural naming preferences of PyTables. Is there a reason not to do so? Cheers, Álvaro. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing listPytables-users@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/pytables-users -- Francesc Alted -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's
Re: [Pytables-users] Problems with Converting Python Int to C Long
Hello Again Jacob, Hmm are they of Python type long? Also, what exactly is the number that is failing? Be Well Anthony On Thu, Jun 28, 2012 at 4:18 PM, Jacob Bennett jacob.bennet...@gmail.comwrote: Hello PyTables Users, I have a concern with a very strange error that references that my python ints cannot be converted to C longs when trying to run Table.append(rows). My python integers are definitely not big, at most they would probably be around 3 billion in size, which shouldn't be any problem for conversion to C long. This is the error that I am receiving... Exception in thread bookthread: Traceback (most recent call last): File C:\Python27\lib\threading.py, line 551, in __bootstrap_inner self.run() File C:\Users\jacob.bennett\development\MarketDataReader\PyTablesInterface\Acceptor.py, line 21, in run BookDataWrapper.acceptDict() File C:\Users\jacob.bennett\development\MarketDataReader\PyTablesInterface\BookDataWrapper.py, line 49, in acceptDict tableD.append(dataArray) File C:\Python27\lib\site-packages\tables\table.py, line 2076, in append rows parameter cannot be converted into a recarray object compliant with table '%s'. The error was: %s % (str(self), exc) ValueError: rows parameter cannot be converted into a recarray object compliant with table '/t301491615959191971 (Table(0,), shuffle, blosc(3)) 'Instrument''. The error was: Python int too large to convert to C long Thanks, Jacob -- Jacob Bennett Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science Class of 2014| benne...@mit.edu -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Use of recarrays as representation for Tables in memory
Thank you Josh, that is representative enough. In my system the speedup of structured arrays is ~30x. A copy of the whole array is still ~6x faster. -á. On Thu, Jun 28, 2012 at 10:13 PM, Josh Ayers josh.ay...@gmail.com wrote: import time import numpy as np dtype = np.format_parser(['i4', 'i4'], [], []) N = 10 rec = np.recarray((N, ), dtype=dtype) struc = np.zeros((N, ), dtype=dtype) t1 = time.clock() for row in rec: pass print time.clock() - t1 t1 = time.clock() for row in struc: pass print time.clock() - t1 -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Problems with Converting Python Int to C Long
It's strange really. It seems like anything int 64 in python (greater than 4billion) fails to convert and throws this message; however, for numbers that can be represented by int 32 can convert fine. Btw, this is for a field that is defined as UInt64 in pytables and only fails if I do Table.append(row). If I do insertion based upon table.row, then it works fine. I will look at this issue more later tonight, and will report my findings. Thanks, Jacob On Thu, Jun 28, 2012 at 5:37 PM, Anthony Scopatz scop...@gmail.com wrote: Hello Again Jacob, Hmm are they of Python type long? Also, what exactly is the number that is failing? Be Well Anthony On Thu, Jun 28, 2012 at 4:18 PM, Jacob Bennett jacob.bennet...@gmail.comwrote: Hello PyTables Users, I have a concern with a very strange error that references that my python ints cannot be converted to C longs when trying to run Table.append(rows). My python integers are definitely not big, at most they would probably be around 3 billion in size, which shouldn't be any problem for conversion to C long. This is the error that I am receiving... Exception in thread bookthread: Traceback (most recent call last): File C:\Python27\lib\threading.py, line 551, in __bootstrap_inner self.run() File C:\Users\jacob.bennett\development\MarketDataReader\PyTablesInterface\Acceptor.py, line 21, in run BookDataWrapper.acceptDict() File C:\Users\jacob.bennett\development\MarketDataReader\PyTablesInterface\BookDataWrapper.py, line 49, in acceptDict tableD.append(dataArray) File C:\Python27\lib\site-packages\tables\table.py, line 2076, in append rows parameter cannot be converted into a recarray object compliant with table '%s'. The error was: %s % (str(self), exc) ValueError: rows parameter cannot be converted into a recarray object compliant with table '/t301491615959191971 (Table(0,), shuffle, blosc(3)) 'Instrument''. The error was: Python int too large to convert to C long Thanks, Jacob -- Jacob Bennett Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science Class of 2014| benne...@mit.edu -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Jacob Bennett Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science Class of 2014| benne...@mit.edu -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Problems with Converting Python Int to C Long
Thanks Jacob, This definitely sounds like a bug. if you come up with a self contained example, please report it at https://github.com/PyTables/PyTables/issues Thanks! Anthony On Thu, Jun 28, 2012 at 7:40 PM, Jacob Bennett jacob.bennet...@gmail.comwrote: It's strange really. It seems like anything int 64 in python (greater than 4billion) fails to convert and throws this message; however, for numbers that can be represented by int 32 can convert fine. Btw, this is for a field that is defined as UInt64 in pytables and only fails if I do Table.append(row). If I do insertion based upon table.row, then it works fine. I will look at this issue more later tonight, and will report my findings. Thanks, Jacob On Thu, Jun 28, 2012 at 5:37 PM, Anthony Scopatz scop...@gmail.comwrote: Hello Again Jacob, Hmm are they of Python type long? Also, what exactly is the number that is failing? Be Well Anthony On Thu, Jun 28, 2012 at 4:18 PM, Jacob Bennett jacob.bennet...@gmail.com wrote: Hello PyTables Users, I have a concern with a very strange error that references that my python ints cannot be converted to C longs when trying to run Table.append(rows). My python integers are definitely not big, at most they would probably be around 3 billion in size, which shouldn't be any problem for conversion to C long. This is the error that I am receiving... Exception in thread bookthread: Traceback (most recent call last): File C:\Python27\lib\threading.py, line 551, in __bootstrap_inner self.run() File C:\Users\jacob.bennett\development\MarketDataReader\PyTablesInterface\Acceptor.py, line 21, in run BookDataWrapper.acceptDict() File C:\Users\jacob.bennett\development\MarketDataReader\PyTablesInterface\BookDataWrapper.py, line 49, in acceptDict tableD.append(dataArray) File C:\Python27\lib\site-packages\tables\table.py, line 2076, in append rows parameter cannot be converted into a recarray object compliant with table '%s'. The error was: %s % (str(self), exc) ValueError: rows parameter cannot be converted into a recarray object compliant with table '/t301491615959191971 (Table(0,), shuffle, blosc(3)) 'Instrument''. The error was: Python int too large to convert to C long Thanks, Jacob -- Jacob Bennett Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science Class of 2014| benne...@mit.edu -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Jacob Bennett Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science Class of 2014| benne...@mit.edu -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
