[Pytables-users] Optimizing pytables for reading entire columns at a time
Thanks for the information guys. I have joined the dev group on Google groups. I'm sure I can learn a lot just by watching the discussions. Also, I think for my current situation I'm going to stick with Pytables carrays. We already have Pytables as a dependency, and we are using it for some other stuff in the project as well. I will definitely keep the stand-alone carray project in mind for the future though. I guess by using Pytables.carrays I'm just losing the ability to query, etc.? Are there any other downsides in a Pytables.carray vs. Pytables.table comparison? -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Optimizing pytables for reading entire columns at a time
With CArrays you can only have one specific type for the array (int, float, etc) whereas with a table each column can have a different type (string, float, etc). If you want to replicate this with carray, you would have to have multiple carray's for each type. I think for storing numerical data where querying isn't that important, carrays are just fine. But even if you have to query, you can replicate the indexing behavior for example by adding a second carray with the values you want to index. On Mon, Sep 24, 2012 at 3:27 PM, Luke Lee durdenm...@gmail.com wrote: Thanks for the information guys. I have joined the dev group on Google groups. I'm sure I can learn a lot just by watching the discussions. Also, I think for my current situation I'm going to stick with Pytables carrays. We already have Pytables as a dependency, and we are using it for some other stuff in the project as well. I will definitely keep the stand-alone carray project in mind for the future though. I guess by using Pytables.carrays I'm just losing the ability to query, etc.? Are there any other downsides in a Pytables.carray vs. Pytables.table comparison? -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
[Pytables-users] where() with start/stop args returning incorrect result set
Hello, I'm hoping someone can help me. When I specify start and stop values for calls to where() and readWhere(), it is returning blatantly incorrect results: table.readWhere(id == 'ceec536a-394e-4dd7-a182-eea557f3bb93', start=3257, stop=table.nrows)[0]['id'] '7f589d3e-a0e1-4882-b69b-0223a7de3801' table.where(id == 'ceec536a-394e-4dd7-a182-eea557f3bb93', start=3257, stop=table.nrows).next()['id'] '7f589d3e-a0e1-4882-b69b-0223a7de3801' This happens with a sequential block of about 150 rows of data, and each time it seems to be 8 rows off (i.e. the row it returns is 8 rows ahead of the row it should be returning). If I remove the start and stop args, it behaves correctly. This seems to be a bug, unless I am misunderstanding something. I'm using Python 2.7.3, PyTables 2.4.0, and hdf5 1.8.9 on OS X 10.8.2. Any ideas? Thanks, Derek -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] where() with start/stop args returning incorrect result set
Hi Derek, Can you please run the following command and report back what you see? python -c import tables; tables.test() Be Well Anthony On Mon, Sep 24, 2012 at 10:56 PM, Derek Shockey derek.shoc...@gmail.comwrote: Hello, I'm hoping someone can help me. When I specify start and stop values for calls to where() and readWhere(), it is returning blatantly incorrect results: table.readWhere(id == 'ceec536a-394e-4dd7-a182-eea557f3bb93', start=3257, stop=table.nrows)[0]['id'] '7f589d3e-a0e1-4882-b69b-0223a7de3801' table.where(id == 'ceec536a-394e-4dd7-a182-eea557f3bb93', start=3257, stop=table.nrows).next()['id'] '7f589d3e-a0e1-4882-b69b-0223a7de3801' This happens with a sequential block of about 150 rows of data, and each time it seems to be 8 rows off (i.e. the row it returns is 8 rows ahead of the row it should be returning). If I remove the start and stop args, it behaves correctly. This seems to be a bug, unless I am misunderstanding something. I'm using Python 2.7.3, PyTables 2.4.0, and hdf5 1.8.9 on OS X 10.8.2. Any ideas? Thanks, Derek -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] where() with start/stop args returning incorrect result set
PS When I do this on linux all 5077 tests pass for me. On Mon, Sep 24, 2012 at 11:09 PM, Anthony Scopatz scop...@gmail.com wrote: Hi Derek, Can you please run the following command and report back what you see? python -c import tables; tables.test() Be Well Anthony On Mon, Sep 24, 2012 at 10:56 PM, Derek Shockey derek.shoc...@gmail.comwrote: Hello, I'm hoping someone can help me. When I specify start and stop values for calls to where() and readWhere(), it is returning blatantly incorrect results: table.readWhere(id == 'ceec536a-394e-4dd7-a182-eea557f3bb93', start=3257, stop=table.nrows)[0]['id'] '7f589d3e-a0e1-4882-b69b-0223a7de3801' table.where(id == 'ceec536a-394e-4dd7-a182-eea557f3bb93', start=3257, stop=table.nrows).next()['id'] '7f589d3e-a0e1-4882-b69b-0223a7de3801' This happens with a sequential block of about 150 rows of data, and each time it seems to be 8 rows off (i.e. the row it returns is 8 rows ahead of the row it should be returning). If I remove the start and stop args, it behaves correctly. This seems to be a bug, unless I am misunderstanding something. I'm using Python 2.7.3, PyTables 2.4.0, and hdf5 1.8.9 on OS X 10.8.2. Any ideas? Thanks, Derek -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] where() with start/stop args returning incorrect result set
I ran the tests. All 4988 passed. The information it output is: PyTables version: 2.4.0 HDF5 version: 1.8.9 NumPy version: 1.6.2 Numexpr version: 2.0.1 (not using Intel's VML/MKL) Zlib version: 1.2.5 (in Python interpreter) LZO version: 2.06 (Aug 12 2011) BZIP2 version: 1.0.6 (6-Sept-2010) Blosc version: 1.1.3 (2010-11-16) Cython version:0.16 Python version:2.7.3 (default, Jul 6 2012, 00:17:51) [GCC 4.2.1 Compatible Apple Clang 3.1 (tags/Apple/clang-318.0.58)] Platform: darwin-x86_64 Byte-ordering: little Detected cores:4 -Derek On Mon, Sep 24, 2012 at 9:09 PM, Anthony Scopatz scop...@gmail.com wrote: Hi Derek, Can you please run the following command and report back what you see? python -c import tables; tables.test() Be Well Anthony On Mon, Sep 24, 2012 at 10:56 PM, Derek Shockey derek.shoc...@gmail.com wrote: Hello, I'm hoping someone can help me. When I specify start and stop values for calls to where() and readWhere(), it is returning blatantly incorrect results: table.readWhere(id == 'ceec536a-394e-4dd7-a182-eea557f3bb93', start=3257, stop=table.nrows)[0]['id'] '7f589d3e-a0e1-4882-b69b-0223a7de3801' table.where(id == 'ceec536a-394e-4dd7-a182-eea557f3bb93', start=3257, stop=table.nrows).next()['id'] '7f589d3e-a0e1-4882-b69b-0223a7de3801' This happens with a sequential block of about 150 rows of data, and each time it seems to be 8 rows off (i.e. the row it returns is 8 rows ahead of the row it should be returning). If I remove the start and stop args, it behaves correctly. This seems to be a bug, unless I am misunderstanding something. I'm using Python 2.7.3, PyTables 2.4.0, and hdf5 1.8.9 on OS X 10.8.2. Any ideas? Thanks, Derek -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
