Hello Nikola,
Thanks for reporting this issue (and sorry about the delayed reply). I
have two requests for you:
1. could you come up with a self contained example that reproduces this
behaviour?
2. and could you maybe make a github issue related to this problem?
#1 is much more important. Thanks a ton!
Be Well
Anthony
On May 18, 2012 5:01 AM, "nikola stevanovic" <nidzadra...@gmail.com> wrote:
> *Hi,*
>
> Couple days ago, I make some experiments with pytables. I was curious
> about reading and writing speed for my future project.
> So, I decided make some tests. In my hdf5 files I have only one table
> named *Table_1*. I started tests with one million rows and after that
> keep continue testing with 100 000 000 and 500 000 000. This is how looks
> table structure:
>
> /Table_1 (Table(500000000,)) ''
> description := {
> "Device_ID": StringCol(itemsize=14, shape=(), dflt='', pos=0),
> "DateTime": Time32Col(shape=(), dflt=0, pos=1),
> "Value": Float32Col(shape=(), dflt=0.0, pos=2),
> "Status": StringCol(itemsize=10, shape=(), dflt='', pos=3)}
> byteorder := 'little'
> chunkshape := (2048,0)
> autoIndex := True
> colindexes := {
> "DateTime": Index(9, full, shuffle, zlib(1)).is_CSI=True}
>
>
> I didn't change chunkshape (default from creating table
> chunkshape=(2048,0)). Only thing I did is creating index on column
> DateTime. Everything worked fine. But, after 500 000 000 rows, I decide
> compare this table and table whith chunkshape=(65536). So I copy this table
> in other hdf5 file using ptrepack tool:
>
> ptrepack --chunkshape='(65536,0)' /home/azura/a.h5:/Table_1
> /home/azura/b.h5:/
>
> My new table work fine until I create index (CSIndex()) on DateTime
> column. Index creation was successful, but calling methods as *where(),
> getWhereList()* throws following exception:
>
> query = '(DateTime > 1293836400.0) & (DateTime < 1297292400.0)'
> a = numpy.array([ (x['Device_ID'],x['DateTime'],x['Value']) for x in
> tbl.where(query) ])
> Traceback (most recent call last):
> File "<pyshell#100>", line 1, in <module>
> a = numpy.array([ (x['Device_ID'],x['DateTime'],x['Value']) for x in
> tbl.where(query) ])
> File "tableExtension.pyx", line 858, in
> tables.tableExtension.Row.__next__ (tables/tableExtension.c:7788)
> File "tableExtension.pyx", line 879, in
> tables.tableExtension.Row.__next__indexed (tables/tableExtension.c:7922)
> AssertionError
>
>
> Then I decide make same table without ptrepack tool. So I created new
> table and fill with 500 000 000 rows (same chunkshape, same record
> structure). Everythings works fine, so my conclusion is that there is a bug
> in ptrepack tool. Note that exception appear in copied table after creating
> CS index. I'm just curious about this. What can be wrong?
>
> I'm using Ubuntu 12.04TLS with ext4
> Processor: Intel® Core™ i3 CPU M 380 @ 2.53GHz × 4
> RAM: 4GB
> HARD DISK: 500GB
>
>
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> PyTables version: 2.3.1
> HDF5 version: 1.8.4-patch1
> NumPy version: 1.6.0
> Numexpr version: 2.0.1 (not using Intel's VML/MKL)
> Zlib version: 1.2.3.4 (in Python interpreter)
> Blosc version: 1.1.2 (2010-11-04)
> Cython version: 0.16
> Python version: 2.7.3 (default, Apr 20 2012, 22:44:07)
> [GCC 4.6.3]
> Platform: linux2-i686
> Byte-ordering: little
> Detected cores: 4
>
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>
> File(filename=/home/azura/b.h5, title='', mode='a', rootUEP='/',
> filters=Filters(complevel=0, shuffle=False, fletcher32=False))
> / (RootGroup) ''
> /Table_1 (Table(500000000,)) ''
> description := {
> "Device_ID": StringCol(itemsize=14, shape=(), dflt='', pos=0),
> "DateTime": Time32Col(shape=(), dflt=0, pos=1),
> "Value": Float32Col(shape=(), dflt=0.0, pos=2),
> "Status": StringCol(itemsize=10, shape=(), dflt='', pos=3)}
> byteorder := 'little'
> chunkshape := (65536,)
> autoIndex := True
> colindexes := {
> "DateTime": Index(9, full, shuffle, zlib(1)).is_CSI=True}
>
> *Cheers!*
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pytables-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
