Re: [Pytables-users] Chunk selection for optimized data access

2013-06-04 Thread Andreas Hilboll
On 04.06.2013 05:35, Tim Burgess wrote: My thoughts are: - try it without any compression. Assuming 32 bit floats, your monthly 5760 x 2880 is only about 65MB. Uncompressed data may perform well and at the least it will give you a baseline to work from - and will help if you are

[Pytables-users] pytable 3 - with encoding

2013-06-04 Thread Jeff Reback
anthony, I can be reached on my cell (917)971-6387-- How ServiceNow helps IT people transform IT departments: 1. A cloud service to automate IT design, transition and operations 2. Dashboards that offer high-level views

[Pytables-users] pytable 30 - encoding

2013-06-04 Thread Jeff Reback
anthony, where am I going wrong here? #!/usr/local/bin/python3 import tables import numpy as np import datetime, time encoding = 'UTF-8' test_file = 'test_select.h5' handle = tables.openFile(test_file, w) node = handle.createGroup(handle.root, 'test') table = handle.createTable(node, 'table',

Re: [Pytables-users] pytable 30 - encoding

2013-06-04 Thread Anthony Scopatz
Hi Jeff, Have you also updated numexpr to the most recent version? The error is coming from numexpr not compiling the expression correctly. Also, you might try making selector a str, rather than bytes: selector = (column == 'str-2') rather than selector = (column == 'str-2').encode(encoding)

Re: [Pytables-users] pytable 30 - encoding

2013-06-04 Thread Jeff Reback
Anthony,   I am using numexpr 2.1 (latest)   this is puzzling; doesn't matter what I pass (bytes or str) , same result?   (column == 'str-2') /mnt/code/arb/test/pytables-3.py(38)module() - result = handle.root.test.table.readWhere(selector) (Pdb) handle.root.test.table.readWhere(selector) ***

Re: [Pytables-users] Chunk selection for optimized data access

2013-06-04 Thread Seref Arikan
I think I've seen this in the release notes of 3.0. This is actually something that I'm looking into as well. So any experience/feedback about creating files in memory would be much appreciated. Best regards Seref On Tue, Jun 4, 2013 at 2:09 PM, Andreas Hilboll li...@hilboll.de wrote: On

Re: [Pytables-users] pytable 30 - encoding

2013-06-04 Thread Jeff Reback
Anthony,   I created an issue with more info   I am not sure if this is a bug, or just a way both ne/pytables treat strings that need to touch an encoded value;   I found workaround by specifying the condvars to readWhere. Any more thoughts on this?   thanks Jeff    

Re: [Pytables-users] Chunk selection for optimized data access

2013-06-04 Thread Tim Burgess
I was playing around with in-memory HDF5 prior to the 3.0 release. Here's an example based on what I was doing.I looked over the docs and it does mention that there is an option to throw away the 'file' rather than write it to disk.Not sure how to do that and can't actually think of a use case