Re: [Pytables-users] Tables with several billion rows.

Tue, 24 Jul 2007 09:16:32 -0700 

John Gill (el 2007-07-24 a les 12:02:53 +0100) va dir::

> There seem to be some issues with table with several billion rows.
> 
> This is with both pytables 2.0 and pytables 1.4.
> 
> In [2]: h = tables.openFile('data.hd5')
> In [3]: len(h.root.events)
> ---------------------------------------------------------------------------
> <type 'exceptions.OverflowError'>         Traceback (most recent call last)
> 
> /home/jng/mnt/ferrari/disk3/flasher/models/work07/ushu/aef/<ipython 
> console> in <module>()
> 
> <type 'exceptions.OverflowError'>: long int too large to convert to int

This is a limitation of the Python interpreter itself in versions <= 2.4
and 32-bit 2.5.  You can read more about the fact at
http://docs.python.org/whatsnew/pep-353.html

In short, Python can't use indexes greater than 2**31-1 in versions up
to 2.4, and in version 2.5 running on 32-bit machines.  Python 2.5 on
64-bit machines fully supports 64-bit indexes.  As an illustration,
the following code::

    class Foo(object):
        def __len__(self): 
            return 2**40 
    
    f = Foo()
    print len(f)

prints ``1099511627776`` under 64-bit Python 2.5, fails with
``OverflowError: long int too large to convert to int`` on a 32-bit
machine, and fails with ``OverflowError: __len__() should return 0 <=
outcome < 2**31`` with 64-bit Python 2.4.

So, if you need to use the length of a dataset, your best option is to
use the ``nrows`` attribute (``h.root.events.nrows`` in the example),
which is a NumPy signed 64-bit integer regardless of the platform and
Python version.  It is available for all datasets in PyTables.

Hope that helps,

::

        Ivan Vilata i Balaguer   >qo<   http://www.carabos.com/
               Cárabos Coop. V.  V  V   Enjoy Data
                                  ""

Attachment: signature.asc
Description: Digital signature

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users

Reply via email to