On 12/6/12 1:42 PM, Alvaro Tejero Cantero wrote:
> Thank you for the comprehensive round-up. I have some ideas and
> reports below.
>
> What about ctables? The documentation says that it is specificly
> column-access optimized, which is what I need in this scenario
> (sometimes sequential, sometimes random).
Yes, ctables is optimized for column access.
>
> Unfortunately I could not get the rootdir parameter for ctables
> __init__ to work in carray 0.4 and pip-installing 0.5 or 0.5.1 leads
> to compilation errors.
Yep, persistence for carray/ctables objects was added in 0.5.
>
> This is the ctables-to-disk error:
>
> ct2 = ca.ctable((np.arange(30000000),), names=('range2',),
> rootdir='/tmp/ctable2.ctable')
> ---------------------------------------------------------------------------
> TypeError Traceback (most recent call last)
> /home/tejero/Dropbox/O/nb/nonridge/<ipython-input-29-255842877a0b>
> in<module>()
> ----> 1 ct2= ca.ctable((np.arange(30000000),), names=('range2',),
> rootdir='/tmp/ctable2.ctable')
>
> /home/tejero/Local/Envs/test/lib/python2.7/site-packages/carray/ctable.pyc
> in__init__(self, cols, names, **kwargs)
> 158 if column.dtype== np.void:
> 159 raise ValueError, "`cols` elements cannot be
> of type void"
> --> 160 column= ca.carray(column, **kwargs)
> 161 elif ratype:
> 162 column= ca.carray(cols[name], **kwargs)
>
> /home/tejero/Local/Envs/test/lib/python2.7/site-packages/carray/carrayExtension.so
> incarray.carrayExtension.carray.__cinit__ (carray/carrayExtension.c:3917)()
>
> TypeError: __cinit__() got an unexpected keyword argument 'rootdir'
>
>
> And this is cut from the pip output when trying to upgrade carray.
>
> gcc: carray/carrayExtension.c
>
> gcc: error: carray/carrayExtension.c: No such file or directory
Hmm, that's strange, because the carrayExtension should have been
cythonized automatically. Here it is part of my install process with pip:
Running setup.py install for carray
* Found Cython 0.17.1 package installed.
* Found numpy 1.7.0b2 package installed.
* Found numexpr 2.0.1 package installed.
cythoning carray/carrayExtension.pyx to carray/carrayExtension.c
building 'carray.carrayExtension' extension
C compiler: gcc -fno-strict-aliasing
-I/Users/faltet/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3
-Wall -Wstrict-prototypes
Hmm, perhaps you need a newer version of Cython?
>
>
> Two more notes:
>
> * a way was added to check in-disk (compressed) vs in-memory
> (uncompressed) node sizes. I was unable to find the way to use it
> either from the 2.4.0 release notes or from the git issue
> https://github.com/PyTables/PyTables/issues/141#issuecomment-5018763
You already found the answer.
>
> * is/will it be possible to load PyTables carrays as in-memory carrays
> without decompression?
Actually, that has been my idea from the very beginning. The concept of
'flavor' for the returned objects when reading is already there, so it
should be relatively easy to add a new 'carray' flavor. Maybe you can
contribute this?
--
Francesc Alted
------------------------------------------------------------------------------
LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
Remotely access PCs and mobile devices and provide instant support
Improve your efficiency, and focus on delivering more value-add services
Discover what IT Professionals Know. Rescue delivers
http://p.sf.net/sfu/logmein_12329d2d
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
