Re: [Pytables-users] Pytables-users Digest, Vol 86, Issue 8
Hi Pushkar, Il 18/07/2013 08:45, Pushkar Raj Pande ha scritto: Both loadtxt and genfromtxt read the entire data into memory which is not desirable. Is there a way to achieve streaming writes? OK, probably fromfile [1] can help you to cook something that works without loading the entire file into memory (and without too much iterations over the file). Anyway I strongly recommend you to not perform read/write cycles on single lines, rather define a reasonable data block size (number of rows) and process the file in chunks. If you find a reasonably simple solution it would be nice to include it in out documentation as an example or a recipe [2] [1] http://docs.scipy.org/doc/numpy/reference/generated/numpy.fromfile.html#numpy.fromfile [2] http://pytables.github.io/latest/cookbook/index.html best regards antonio Thanks, Pushkar On Wed, Jul 17, 2013 at 7:04 PM, Pushkar Raj Pande topgun...@gmail.comwrote: Thanks Antonio and Anthony. I will give this a try. -Pushkar On Wed, Jul 17, 2013 at 2:59 PM, pytables-users-requ...@lists.sourceforge.net wrote: Date: Wed, 17 Jul 2013 16:59:16 -0500 From: Anthony Scopatz scop...@gmail.com Subject: Re: [Pytables-users] Pytables bulk loading data To: Discussion list for PyTables pytables-users@lists.sourceforge.net Message-ID: capk-6t4ht9+ncdd_1oojrbn4u_6+ouekobklmokeufjojjk...@mail.gmail.com Content-Type: text/plain; charset=iso-8859-1 Hi Pushkar, I agree with Antonio. You should load your data with NumPy functions and then write back out to PyTables. This is the fastest way to do things. Be Well Anthony On Wed, Jul 17, 2013 at 2:12 PM, Antonio Valentino antonio.valent...@tiscali.it wrote: Hi Pushkar, Il 17/07/2013 19:28, Pushkar Raj Pande ha scritto: Hi all, I am trying to figure out the best way to bulk load data into pytables. This question may have been already answered but I couldn't find what I was looking for. The source data is in form of csv which may require parsing, type checking and setting default values if it doesn't conform to the type of the column. There are over 100 columns in a record. Doing this in a loop in python for each row of the record is very slow compared to just fetching the rows from one pytable file and writing it to another. Difference is almost a factor of ~50. I believe if I load the data using a C procedure that does the parsing and builds the records to write in pytables I can get close to the speed of just copying and writing the rows from 1 pytable to another. But may be there is something simple and better that already exists. Can someone please advise? But if it is a C procedure that I should write can someone point me to some examples or snippets that I can refer to put this together. Thanks, Pushkar numpy has some tools for loading data from csv files like loadtxt [1], genfromtxt [2] and other variants. Non of them is OK for you? [1] http://docs.scipy.org/doc/numpy/reference/generated/numpy.loadtxt.html#numpy.loadtxt [2] http://docs.scipy.org/doc/numpy/reference/generated/numpy.genfromtxt.html#numpy.genfromtxt cheers -- Antonio Valentino -- Antonio Valentino -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Pytables bulk loading data
Hi Pushkar, Il 17/07/2013 19:28, Pushkar Raj Pande ha scritto: Hi all, I am trying to figure out the best way to bulk load data into pytables. This question may have been already answered but I couldn't find what I was looking for. The source data is in form of csv which may require parsing, type checking and setting default values if it doesn't conform to the type of the column. There are over 100 columns in a record. Doing this in a loop in python for each row of the record is very slow compared to just fetching the rows from one pytable file and writing it to another. Difference is almost a factor of ~50. I believe if I load the data using a C procedure that does the parsing and builds the records to write in pytables I can get close to the speed of just copying and writing the rows from 1 pytable to another. But may be there is something simple and better that already exists. Can someone please advise? But if it is a C procedure that I should write can someone point me to some examples or snippets that I can refer to put this together. Thanks, Pushkar numpy has some tools for loading data from csv files like loadtxt [1], genfromtxt [2] and other variants. Non of them is OK for you? [1] http://docs.scipy.org/doc/numpy/reference/generated/numpy.loadtxt.html#numpy.loadtxt [2] http://docs.scipy.org/doc/numpy/reference/generated/numpy.genfromtxt.html#numpy.genfromtxt cheers -- Antonio Valentino -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] `__iter__` state and `itertools.islice` when
Hi Tony,
Il giorno 09/lug/2013, alle ore 06:38, Tony Yu tsy...@gmail.com ha scritto:
Hi,
I ran into a subtle, unexpected issue while using `itertools.islice`. I
wanted to pass slices of an array for processing without actually reading the
entire array, and I wanted that processing function to know nothing about how
I'm taking that slice. To that end, I had a loop that sliced the array using
`itertools.islice` and called the function on each slice. Instead of
returning the slice I specified, `islice` treated the previous end slice as
the starting point to the next slice.
That description is a bit confusing, but the example below (along with the
attached test data) should illustrate the point. Maybe I'm missing something,
but the only work around that I found was to set a private flag (e.g.
`h5.root.array._init = False`) on each call to `islice` to reset the counter
used in `__iter__`.
I'm not sure if this is expected behavior or not, but it does differ from how
`islice` works on numpy arrays (as demonstrated in the example below). I used
the google and nothing similar came up, so I thought I'd post here.
Best,
-Tony
#
import tables
import itertools
import numpy as np
h5 = tables.openFile('test.h5')
array = np.arange(100)
for i in range(5):
# Numpy array slice always returns 0..10
print list(itertools.islice(array, 0, 10))
# PyTables array slice shifts with each iteration
print list(itertools.islice(h5.root.array, 0, 10))
h5.close()
test.h5--
Yes, this is a bug IMO.
Thank you for reporting and thank you for the small demonstration script.
Can you please file a bug report on github [1]?
Please also add info about the PyTables version you used for the test..
[1] https://github.com/PyTables/PyTables/issues
--
Antonio Valentino
--
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk
___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Speed of in-kernel Full-Table Search
Hi Sebastian, Il 25/06/2013 09:36, Wagner Sebastian ha scritto: Hi Anthony and Antonio, Thanks for your fast responses. It's great to hear all features are now free to use, though I needed one and a half week to get this. The first reference I read to learn the usage of PyTables was Hints for SQL Users [1], where is stated several times, for example in the section ' Creating an index': Indexing is supported in the commercial version of PyTables (PyTablesPro). I would suggest that these texts should be updated. Being convinced it's only available in Pro-Version after I read it so often, I also overread the warning in the PyTables Pro page[2] (As I were only interested in the features not available in the free version I just scrolled down immediately, diagonal reading...). So the next suggestion is to give a color to the warning text there :) [1] http://www.pytables.org/moin/HintsForSQLUsers#Creatinganindex http://www.pytables.org/moin/HintsForSQLUsers#Selectingdata [2] http://www.pytables.org/moin/PyTablesPro regards, Sebastian thank you for reporting the issue, I will fix it ASAP. The same problem also affect the corresponding cookbook page [1]. Anyway, please, feel free to update the wiki if you find outdated material. [1] http://pytables.github.io/cookbook/hints_for_sql_users.html On Mon, Jun 24, 2013 at 4:25 AM, Wagner Sebastian sebastian.wagner...@ait.ac.at wrote: Dear PyTables-Users, ** ** For testing purposes I use a PyTables DB with 4 columns (1x Uint8 and 3xFloat) with 750k rows, the total file size about 90MB. As the free version does no support indexing I thought that a search (full-table) on this database would last a least one or two seconds, because the file has to be loaded first (throttleneck I/O), and then the search over ~20k rows can begin. But PyTables took only 0.05 seconds for a full table search (in-kernel, so near C-speed, but nevertheless full table), while my bisecting algorithm with a precomputed sorted list wrapped around PyTables (but saved in there), took about 0.5 seconds. ** ** So the thing I don?t understand: How can PyTables be so fast without any Indexing? Hi Sebastian, First, there is no longer a non-free version of PyTables and v3.0 *does* have indexing capabilities. However, you have to enable them so you probably weren't using them. PyTables is fast because HDF5 is a binary format, it using pthreads under the covers to parallelize some tasks, and it uses numexpr (which is also parallel) to evaluate many expressions. All of these things help make PyTables great! Be Well Anthony Il 24/06/2013 11:25, Wagner Sebastian ha scritto: Dear PyTables-Users, For testing purposes I use a PyTables DB with 4 columns (1x Uint8 and 3xFloat) with 750k rows, the total file size about 90MB. As the free version does no support indexing I thought that a search (full-table) on this database would last a least one or two seconds, because the file has to be loaded first (throttleneck I/O), and then the search over ~20k rows can begin. But PyTables took only 0.05 seconds for a full table search (in-kernel, so near C-speed, but nevertheless full table), while my bisecting algorithm with a precomputed sorted list wrapped around PyTables (but saved in there), took about 0.5 seconds. So the thing I don't understand: How can PyTables be so fast without any Indexing? I'm using 3.0.0rc2 coming with WinPython Regards, Sebastian The indexing features of PyTables Pro are now available in the open source version of PyTables since version 2.3 (please see [1]). [1] http://pytables.github.io/release-notes/RELEASE_NOTES_v2.3.x.html#changes-from-2-2-1-to-2-3 ciao -- Antonio Valentino -- Antonio Valentino -- This SF.net email is sponsored by Windows: Build for Windows Store. http://p.sf.net/sfu/windows-dev2dev ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Chunk selection for optimized data access
Hi list, Il 05/06/2013 00:38, Anthony Scopatz ha scritto: On Tue, Jun 4, 2013 at 12:30 PM, Seref Arikan serefari...@gmail.com wrote: I think I've seen this in the release notes of 3.0. This is actually something that I'm looking into as well. So any experience/feedback about creating files in memory would be much appreciated. I think that you want to set parameters.DRIVER to H5DF_CORE [1]. I haven't ever used this personally, but it would be great to have an example script, if someone wants to write one ;) Be Well Anthony 1. http://pytables.github.io/usersguide/parameter_files.html#hdf5-driver-management thare is also a small example of usage in the cookbook [1] [1] http://pytables.github.io/cookbook/inmemory_hdf5_files.html ciao -- Antonio Valentino -- How ServiceNow helps IT people transform IT departments: 1. A cloud service to automate IT design, transition and operations 2. Dashboards that offer high-level views of enterprise services 3. A single system of record for all IT processes http://p.sf.net/sfu/servicenow-d2d-j ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
[Pytables-users] ANN: PyTables 3.0 rc3
Announcing PyTables 3.0.0rc3 We are happy to announce PyTables 3.0.0rc3. Changes from 3.0rc2 to 3.0rc3 - * Fixed a crash on 32bit platforms. * Fixed a couple of issue related to stepped read/iteration on tables (see :issue:`260` and :issue:`262`). **Enjoy data!** -- The PyTables Developers -- Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get 100% visibility into your production application - at no cost. Code-level diagnostics for performance bottlenecks with 2% overhead Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap1 ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
[Pytables-users] ANN: PyTables 3.0 rc1
= Announcing PyTables 3.0.0rc1 = We are happy to announce PyTables 3.0.0rc1. PyTables 3.0.0rc1 comes after about 5 years from the last major release (2.0) and 7 months since the last stable release (2.4.0). This is new major release and an important milestone for the PyTables project since it provides the long waited support for Python 3.x, which has been around for 4 years. Almost all of the core numeric/scientific packages for Python already support Python 3 so we are very happy that now also PyTables can provide this important feature. What's new == A short summary of main new features: - Since this release, PyTables now provides full support to Python 3 - The entire code base is now more compliant with coding style guidelines described in PEP8. - Basic support for HDF5 drivers. It now is possible to open/create an HDF5 file using one of the SEC2, DIRECT, LOG, WINDOWS, STDIO or CORE drivers. - Basic support for in-memory image files. An HDF5 file can be set from or copied into a memory buffer. - Implemented methods to get/set the user block size in a HDF5 file. - All read methods now have an optional *out* argument that allows to pass a pre-allocated array to store data. - Added support for the floating point data types with extended precision (Float96, Float128, Complex192 and Complex256). - Consistent ``create_xxx()`` signatures. Now it is possible to create all data sets Array, CArray, EArray, VLArray, and Table from existing Python objects. - Complete rewrite of the `nodes.filenode` module. Now it is fully compliant with the interfaces defined in the standard `io` module. Only non-buffered binary I/O is supported currently. Please refer to the RELEASE_NOTES document for a more detailed list of changes in this release. As always, a large amount of bugs have been addressed and squashed as well. In case you want to know more in detail what has changed in this version, please refer to: http://pytables.github.io/release_notes.html You can download a source package with generated PDF and HTML docs, as well as binaries for Windows, from: http://sourceforge.net/projects/pytables/files/pytables/3.0.0rc1 For an online version of the manual, visit: http://pytables.github.io/usersguide/index.html What it is? === PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. PyTables includes OPSI, a new indexing technology, allowing to perform data lookups in tables exceeding 10 gigarows (10**10 rows) in less than a tenth of a second. Resources = About PyTables: http://www.pytables.org About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ Acknowledgments === Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Most specially, a lot of kudos go to the HDF5 and NumPy makers. Without them, PyTables simply would not exist. Share your experience = Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. **Enjoy data!** -- The PyTables Developers -- Learn Graph Databases - Download FREE O'Reilly Book Graph Databases is the definitive new guide to graph databases and their applications. This 200-page book is written by three acclaimed leaders in the field. The early access version is available now. Download your free book today! http://p.sf.net/sfu/neotech_d2d_may ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] ANN: numexpr 2.1 RC1 available!
Hi Francesc, Il 26/04/2013 14:11, Francesc Alted ha scritto: Hi Antonio, Al 26/04/13 08:46, En/na Antonio Valentino ha escrit: Hi Francesc, Il 25/04/2013 23:06, Francesc Alted ha scritto: Thanks. Will do! Thanks. For the record patches 0002 and 0003 close issue [75] and [77]. Also numexpr 2.1 closes [91] and [95] [75] https://code.google.com/p/numexpr/issues/detail?id=75 [77] https://code.google.com/p/numexpr/issues/detail?id=77 [91] https://code.google.com/p/numexpr/issues/detail?id=91 [95] https://code.google.com/p/numexpr/issues/detail?id=95 Just released a new version (2.1 RC3) addressing all of this. Please check it out and tell me how it goes. Thanks, Francesc It seems to be all OK for me and with the patch provided by Anthiny now PyTables is 100% compatible with numexpr 2.1rc3. Ubuntu users can find packages for numexpr 2.1rc3 (both for python 2 and pyhton 3) at [1]. [1] https://launchpad.net/~a.valentino/+archive/eotools thanks again -- Antonio Valentino -- Try New Relic Now We'll Send You this Cool Shirt New Relic is the only SaaS-based application performance monitoring service that delivers powerful full stack analytics. Optimize and monitor your browser, app, servers with just a few lines of code. Try New Relic and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
[Pytables-users] ANN: PyTables 3.0 beta1
= Announcing PyTables 3.0.0b1 = We are happy to announce PyTables 3.0.0b1. PyTables 3.0.0b1 comes after about 5 years from the last major release (2.0) and 7 months since the last stable release (2.4.0). This is new major release and an important milestone for the PyTables project since it provides the long waited support for Python 3.x that is being around for already 4 years now. Almost all the main numeric/scientific packages for python already support Python 3 so we are very happy that now also PyTables can provide this important feature. What's new == A short summary of main new features: - Since this release PyTables provides full support to Python 3 - The entire code base is now more compliant with coding style guidelines describe in the PEP8. - Basic support for HDF5 drivers. Now it is possible to open/create an HDF5 file using one of the SEC2, DIRECT, LOG, WINDOWS, STDIO or CORE drivers. - Basic support for in-memory image files. An HDF5 file can be set from or copied into a memory buffer. - Implemented methods to get/set the user block size in a HDF5 file. - All read methods now have an optional *out* argument that allows to pass a pre-allocated array to store data. - Added support for the floating point data types with extended precision (Float96, Float128, Complex192 and Complex256). Please refer to the RELEASE_NOTES document for a more detailed list of changes in this release. As always, a large amount of bugs have been addressed and squashed as well. In case you want to know more in detail what has changed in this version, please refer to: http://pytables.github.io/release_notes.html You can download a source package with generated PDF and HTML docs, as well as binaries for Windows, from: http://sourceforge.net/projects/pytables/files/pytables/3.0.0b1 For an online version of the manual, visit: http://pytables.github.io/usersguide/index.html What it is? === PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. PyTables includes OPSI, a new indexing technology, allowing to perform data lookups in tables exceeding 10 gigarows (10**10 rows) in less than a tenth of a second. Resources = About PyTables: http://www.pytables.org About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ Acknowledgments === Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Most specially, a lot of kudos go to the HDF5 and NumPy makers. Without them, PyTables simply would not exist. Share your experience = Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. **Enjoy data!** -- The PyTables Team -- Try New Relic Now We'll Send You this Cool Shirt New Relic is the only SaaS-based application performance monitoring service that delivers powerful full stack analytics. Optimize and monitor your browser, app, servers with just a few lines of code. Try New Relic and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] ANN: numexpr 2.1 RC1 available!
Hi Francesc, Il 25/04/2013 23:06, Francesc Alted ha scritto: Thanks. Will do! Thanks. For the record patches 0002 and 0003 close issue [75] and [77]. Also numexpr 2.1 closes [91] and [95] [75] https://code.google.com/p/numexpr/issues/detail?id=75 [77] https://code.google.com/p/numexpr/issues/detail?id=77 [91] https://code.google.com/p/numexpr/issues/detail?id=91 [95] https://code.google.com/p/numexpr/issues/detail?id=95 El 25/04/2013 21:02, Antonio Valentino antonio.valent...@tiscali.it va escriure: Hi Francesc, Il 14/04/2013 22:19, Francesc Alted ha scritto: Announcing Numexpr 2.1RC1 [CUT] probably it is a little bit late now but it would be nice if you could consider to include the patches (patch 0002 and 0003) used in debian [1]. Patch 0001 can probably be avoided by adding setup.cfg to the MANIFEST.in file. thanks in advance [1] http://anonscm.debian.org/gitweb/?p=debian-science/packages/numexpr.git;a=tree;f=debian/patches;h=73937039da00e4ecbd9318a856be4aa39325e55f;hb=HEAD -- Antonio Valentino -- Antonio Valentino -- Try New Relic Now We'll Send You this Cool Shirt New Relic is the only SaaS-based application performance monitoring service that delivers powerful full stack analytics. Optimize and monitor your browser, app, servers with just a few lines of code. Try New Relic and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] ANN: numexpr 2.1 RC1 available!
Hi Francesc, Il 14/04/2013 22:19, Francesc Alted ha scritto: Announcing Numexpr 2.1RC1 Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like 3*a+4*b) are accelerated and use less memory than doing the same calculation in Python. It wears multi-threaded capabilities, as well as support for Intel's VML library, which allows for squeezing the last drop of performance out of your multi-core processors. What's new == This version adds compatibility for Python 3. A bunch of thanks to Antonio Valentino for his excelent work on this.I apologize for taking so long in releasing his contributions. In case you want to know more in detail what has changed in this version, see: http://code.google.com/p/numexpr/wiki/ReleaseNotes or have a look at RELEASE_NOTES.txt in the tarball. Where I can find Numexpr? = The project is hosted at Google code in: http://code.google.com/p/numexpr/ This is a release candidate 1, so it will not be available on the PyPi repository. I'll post it there when the final version will released. Share your experience = Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy! -- Francesc Alted probably it is a little bit late now but it would be nice if you could consider to include the patches (patch 0002 and 0003) used in debian [1]. Patch 0001 can probably be avoided by adding setup.cfg to the MANIFEST.in file. thanks in advance [1] http://anonscm.debian.org/gitweb/?p=debian-science/packages/numexpr.git;a=tree;f=debian/patches;h=73937039da00e4ecbd9318a856be4aa39325e55f;hb=HEAD -- Antonio Valentino -- Try New Relic Now We'll Send You this Cool Shirt New Relic is the only SaaS-based application performance monitoring service that delivers powerful full stack analytics. Optimize and monitor your browser, app, servers with just a few lines of code. Try New Relic and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] sourceforge downloads corrupted?
Hi Matt, Il 24/04/2013 21:09, Matt Terry ha scritto: Hello, The source tarball for pytables 2.4 on sourceforge appears to be broken. The file size is suspiciously small (800 kB vs 8.5MB on PyPI), the tarball doesn't untar, and the md5 doesn't match. -matt Thanks for reporting. It should be fixed now. ciao -- Antonio Valentino -- Try New Relic Now We'll Send You this Cool Shirt New Relic is the only SaaS-based application performance monitoring service that delivers powerful full stack analytics. Optimize and monitor your browser, app, servers with just a few lines of code. Try New Relic and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Documentation for stable releases?
Hi Gaëtan, Il 22/04/2013 13:09, Gaëtan de Menten ha scritto: Hello all, TL;DR: It would be nice to have online documentation for stable versions and have pytables.github.io point to the doc for the latest stable release by default. I just tried to use the new out= argument to table.read, only to find out it did not work in my version (2.3.1). please file a bug report for this. The out= parameter should be marked as .. versionadded:: 3.0 Then I tried to update my version to 2.4 since I thought it was implemented in that version because of the 2.4.0+1.dev name at the top of the page which I thought meant dev version leading to 2.4, or maybe to 2.4.1, but certainly not the next major release. I got even more confused because, after the initial failure with my 2.3.1 release, I checked the release notes... which I thought were for 2.4 because the title of the release notes page is Release notes for PyTables 2.4 series when it is in fact for the next major version... Gaëtan, we follow guidelines described in [1] for managing the development cycle. We choose a version number only when it is actually time to make a release. VERSION+1.dev just means the development version (.dev) of the next stable release. Here are a couple suggestions: * doc for stable releases (default to latest stable), bonus points to be able to switch easily from one version to another, a-la Python stdlib. I agree, in the future we will pay more attention to publish doc for the latest stable release on the main site * change 2.4.0+1.dev to 3.0-dev or 3.0-pre, and all mentions of 2.4.x * have new arguments to functions documented in the docstring for the functions (like in Python stdlib): new in pytables 3.0 in the docstring for table.read() would have made wonders. please see comments above best regards [1] http://nvie.com/posts/a-successful-git-branching-model -- Antonio Valentino -- Precog is a next-generation analytics platform capable of advanced analytics on semi-structured data. The platform includes APIs for building apps and a phenomenal toolset for data science. Developers can use our toolset for easy data analysis visualization. Get a free account! http://www2.precog.com/precogplatform/slashdotnewsletter ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Tutorial at PyData Conference New York
Hi Francesc, congratulations! Il 27/10/2012 13:16, Francesc Alted ha scritto: Hi, You may be interested on my IPython notebooks and slides for the conference: http://pytables.org/download/PyData2012-NYC.tar.gz PyData-NYC-2012-v3.pptx http://www.pytables.org/docs/PyData2012-NYC.pdf [BTW this time I felt in love with IPython notebook: it is great!] yes, the IPython notebuok is fantastic! ... and the idea of saving tutorials into notebook files is very very nice :)) Maybe we could provide notebook files for all tutorials in the official doc. ciao -- Antonio Valentino -- WINDOWS 8 is here. Millions of people. Your app in 30 days. Visit The Windows 8 Center at Sourceforge for all your go to resources. http://windows8center.sourceforge.net/ join-generation-app-and-make-money-coding-fast/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] PyTables broke in Ubuntu 12.10
Hi Jason, Il giorno 26/ott/2012, alle ore 07:28, Jason Moore moorepa...@gmail.com ha scritto: So it looks like python-tables in Ubuntu 12.10 requires libhdf5-7 and libhdf5-7 has /usr/lib/libhdf5.so.7 not libhdf5.so.6. Correct, the hdf5 package has been updated in ubuntu 12.10. If you use the standard python-tables package from ubuntu repositories please file a bug report on launchpad. Jason On Thu, Oct 25, 2012 at 4:10 PM, Aquil Abdullah aquil.abdul...@gmail.com wrote: Can you check to see if libhdf5.so is in your path? If not, you can add it to the path where it resides to your PATH variable. Hopefully, that helps. Aquil H. Abdullah On Oct 25, 2012, at 18:42, Jason Moore moorepa...@gmail.com wrote: I just updated to Ubuntu 12.10 and my pytables install is broken. I reinstalled and it seems like I have hdf5 1.8.4 installed but I get this error: moorepants@moorepants-LT:BicycleDataProcessor(master)$ vitables InstrumentedBicycleData.h5 Traceback (most recent call last): File /usr/bin/vitables, line 105, in module main(sys.argv) File /usr/bin/vitables, line 48, in main from vitables.vtapp import VTApp File /usr/share/vitables/vitables/vtapp.py, line 35, in module import tables File /usr/local/lib/python2.7/dist-packages/tables/__init__.py, line 30, in module from tables.utilsExtension import getPyTablesVersion, getHDF5Version ImportError: libhdf5.so.6: cannot open shared object file: No such file or directory What am I missing? Jason Anyway you can also use the eotools PPA [1] that has a more updated version of pytables: 2.4 against 2.3.1 of the official universe repo. [1] https://launchpad.net/~a.valentino/+archive/eotools?field.series_filter=quantal best regards -- Antonio Valentino -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] PyTables broke in Ubuntu 12.10
Hi Jason, Il 26/10/2012 18:44, Jason Moore ha scritto: Where exactly do I submit the bug report? There doesn't seem to be a bug option here: https://launchpad.net/ubuntu/quantal/+package/python-tables Jason of course you need a launchpad account, than you can follow instruction of the ReportingBugs page on the ubuntu wiki [1] [1] https://help.ubuntu.com/community/ReportingBugs ciao On Thu, Oct 25, 2012 at 11:27 PM, Antonio Valentino antonio.valent...@tiscali.it wrote: Hi Jason, Il giorno 26/ott/2012, alle ore 07:28, Jason Moore moorepa...@gmail.com ha scritto: So it looks like python-tables in Ubuntu 12.10 requires libhdf5-7 and libhdf5-7 has /usr/lib/libhdf5.so.7 not libhdf5.so.6. Correct, the hdf5 package has been updated in ubuntu 12.10. If you use the standard python-tables package from ubuntu repositories please file a bug report on launchpad. Jason On Thu, Oct 25, 2012 at 4:10 PM, Aquil Abdullah aquil.abdul...@gmail.com wrote: Can you check to see if libhdf5.so is in your path? If not, you can add it to the path where it resides to your PATH variable. Hopefully, that helps. Aquil H. Abdullah On Oct 25, 2012, at 18:42, Jason Moore moorepa...@gmail.com wrote: I just updated to Ubuntu 12.10 and my pytables install is broken. I reinstalled and it seems like I have hdf5 1.8.4 installed but I get this error: moorepants@moorepants-LT:BicycleDataProcessor(master)$ vitables InstrumentedBicycleData.h5 Traceback (most recent call last): File /usr/bin/vitables, line 105, in module main(sys.argv) File /usr/bin/vitables, line 48, in main from vitables.vtapp import VTApp File /usr/share/vitables/vitables/vtapp.py, line 35, in module import tables File /usr/local/lib/python2.7/dist-packages/tables/__init__.py, line 30, in module from tables.utilsExtension import getPyTablesVersion, getHDF5Version ImportError: libhdf5.so.6: cannot open shared object file: No such file or directory What am I missing? Jason Anyway you can also use the eotools PPA [1] that has a more updated version of pytables: 2.4 against 2.3.1 of the official universe repo. [1] https://launchpad.net/~a.valentino/+archive/eotools?field.series_filter=quantal best regards -- Antonio Valentino -- Antonio Valentino -- The Windows 8 Center In partnership with Sourceforge Your idea - your app - 30 days. Get started! http://windows8center.sourceforge.net/ what-html-developers-need-to-know-about-coding-windows-8-metro-style-apps/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] PyTables broke in Ubuntu 12.10
Hi Jason, Il 26/10/2012 21:37, Jason Moore ha scritto: I'll post the bug report, but I'd like to get this working on my system. I've always had trouble compiling pytables from source due to the dependencies. Right now I just need to get this working because I can no longer use my software now that PyTabes is broken. If you use ubuntu 12.04 you can install build dependencies as follows: $ sudo apt-get install libhdf5-dev python-dev cython python-numexpr libbz2-dev zlib1g-dev, liblzo2-dev or simply $ sudo apt-get build-dep pytables Then you can build pytables typing python setup.py build Question 1: What are the exact command for installing for source (including all flags)? I can't find this explicitly in the documentation especially how to use the --hdf5 flag and other flags to point to where the dependencies are installed. I'm trying: sudo python setup.py build_ext --inplace --hdf5=/usr/lib/libhdf5.so.7 But having little luck. It still can't find my hdf5 library. The use of the --hdf5 flag is explained in [1], anyway you should not need it on ubuntu. [1] http://pytables.github.com/usersguide/installation.html Question 2: I tried your pytables2.4 ppa but it also can't find the hdf5 library. How can I install the old /usr/lib/lbhdf5.so.6 file? I also remember it being painful to install the HDF5 libraries from source. Is this file available in the Ubuntu repositories? This is very strange. It seems to me a misconfiguration of the apt system. Are you sure that all you apt sources point to quantal? Maybe some of them still points to precise cheers -- Antonio Valentino -- WINDOWS 8 is here. Millions of people. Your app in 30 days. Visit The Windows 8 Center at Sourceforge for all your go to resources. http://windows8center.sourceforge.net/ join-generation-app-and-make-money-coding-fast/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] PyTables broke in Ubuntu 12.10
Hi Jason, Il 26/10/2012 21:59, Jason Moore ha scritto: Solution was simple once I found it. Here is the workaround: https://bugs.launchpad.net/ubuntu/+source/octave/+bug/1005243 Just make a symlink to the new file. Jason Honestly I don't think it is a good idea. ciao -- Antonio Valentino -- WINDOWS 8 is here. Millions of people. Your app in 30 days. Visit The Windows 8 Center at Sourceforge for all your go to resources. http://windows8center.sourceforge.net/ join-generation-app-and-make-money-coding-fast/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] 10 years of PyTables
Il 21/10/2012 17:26, Francesc Alted ha scritto: Hi!, This month PyTables celebrates the 10th anniversary of its first public release: http://osdir.com/ml/python.scientific.user/2002-10/msg00043.html Happy birthday PyTables!!! :)) ciao -- Antonio Valentino -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] multiprocessing and pytables
Hi Anthony, Il giorno 16/ott/2012, alle ore 02:04, Anthony Scopatz scop...@gmail.com ha scritto: Hello Ernesto, So you are actually asking two different questions, one on reading and the other on writing. In general reading, or querying, with multiprocessing works very well. Writing to a single file with multiple processes is destined to failure though. So the strategy that many people have adopted is to have multiple processes create the data and then have a master process which acts as a queue for writing out the data. Please see the example here for more inspiration [1]. Note that we have been having problems recently with multiprocess writing out to multiple files, but that is not what you want to do. Be Well Anthony 1. https://github.com/PyTables/PyTables/blob/develop/examples/multiprocess_access_queues.py It seems that the topic PyTables + multiprocessing became very popular since some time. Probably we should add a FAQ entry and provide a more extended tutorial based on the example provided by Josh. cheers -- Antonio Valentino -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] [ANN] Blosc 1.1.4 released
Hi Francesc, thank you. Just pushed updates into pytables. ciao Il 16/09/2012 12:07, Francesc Alted ha scritto: === Announcing Blosc 1.1.4 A blocking, shuffling and lossless compression library === What is new? - Redefinition of the BLOSC_MAX_BUFFERSIZE constant as (INT_MAX - BLOSC_MAX_OVERHEAD) instead of just INT_MAX. This prevents to produce outputs larger than INT_MAX, which is not supported. - `exit()` call has been replaced by a ``return -1`` in blosc_compress() when checking for buffer sizes. Now programs will not just exit when the buffer is too large, but return a negative code. - Improvements in explicit casts. Blosc compiles without warnings (with GCC) now. - Lots of improvements in docs, in particular a nice ascii-art diagram of the Blosc format (Valentin Haenel). - [HDF5 filter] Adapted HDF5 filter to use HDF5 1.8 by default (Antonio Valentino). For more info, please see the release notes in: https://github.com/FrancescAlted/blosc/wiki/Release-notes -- Antonio Valentino -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://ad.doubleclick.net/clk;258768047;13503038;j? http://info.appdynamics.com/FreeJavaPerformanceDownload.html ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] New find_library('hdf5dll.dll') call fails in frozen app
Hi Stuart,
Il 27/08/2012 17:43, Stuart Mentzer ha scritto:
Hello,
I upgraded to PyTables 2.4.0 and I was freezing an application on Windows
with PyInstaller. The frozen app fails at this new find_library call in
__init__.py:
if not ctypes.util.find_library('hdf5dll.dll'):
raise ImportError('Could not load hdf5dll.dll, please ensure' +
' that it can be found in the system path')
PyInstaller correctly places this DLL in the same directory as the
application .exe where standard Windows DLL search logic will find it.
Apparently the find_library doesn't do that in a frozen application. That is
a big problem. I had to comment this code out to get a working frozen app.
That code was added in revision e9f6919.
It is mainly a sanity check added under request of one of our users:
https://github.com/PyTables/PyTables/pull/146
This is on Windows 7 64-bit with a 32-bit Python toolchain. Trying both
PyInstaller 1.5.1 and 2.0.
Should I file a bug report? Any easy work-around?
Thanks,
Stuart
Yes please file a pull request with your patch.
It would be nice to preserve the sanity check in standard case so,
maybe, a good solution could be adding some check on sys.frozen or
something like that.
Thank you
--
Antonio Valentino
--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] New find_library('hdf5dll.dll') call fails in frozen app
Hi Christoph,
thank you very much for your hints.
Il 27/08/2012 19:22, Christoph Gohlke ha scritto:
On 8/27/2012 9:42 AM, Antonio Valentino wrote:
Hi Stuart,
Il 27/08/2012 17:43, Stuart Mentzer ha scritto:
Hello,
I upgraded to PyTables 2.4.0 and I was freezing an application on Windows
with PyInstaller. The frozen app fails at this new find_library call in
__init__.py:
if not ctypes.util.find_library('hdf5dll.dll'):
raise ImportError('Could not load hdf5dll.dll, please ensure'
+
' that it can be found in the system path')
PyInstaller correctly places this DLL in the same directory as the
application .exe where standard Windows DLL search logic will find it.
Apparently the find_library doesn't do that in a frozen application. That
is a big problem. I had to comment this code out to get a working frozen
app.
That code was added in revision e9f6919.
It is mainly a sanity check added under request of one of our users:
https://github.com/PyTables/PyTables/pull/146
This is on Windows 7 64-bit with a 32-bit Python toolchain. Trying both
PyInstaller 1.5.1 and 2.0.
Should I file a bug report? Any easy work-around?
Thanks,
Stuart
Yes please file a pull request with your patch.
It would be nice to preserve the sanity check in standard case so,
maybe, a good solution could be adding some check on sys.frozen or
something like that.
Thank you
Hello,
As a workaround for frozen distributions, try to add the sys.executable
directory to os.environ['PATH'] before importing tables.
Christoph, I suppose it can also be done at this point too:
https://github.com/PyTables/PyTables/blob/develop/tables/__init__.py#L24
Isn't it?
Please Stuart, can you try this fix as well.
Ctypes only tries to find a library in the os.environ['PATH']
directories, not the current directory or the sys.executable directory
as one could expect.
http://hg.python.org/cpython/file/64640a02b0ca/Lib/ctypes/util.py#l48
As a workaround, for distributions that place the HDF5 and other DLLs in
the tables package directory, tables.__init__.py adds the tables package
directory to os.environ['PATH']. This also makes sure that the DLLs are
found when loading the hdf5Extension.pyd and other C extension modules
(another common problem). The use of __file__ to get the tables
directory should better be wrapped in a try..except statement.
https://github.com/PyTables/PyTables/blob/develop/tables/__init__.py#L24
Christoph
Thanks, I'll fix it ASAP.
P.S.: please Christoph, do you have some hint for gh-175 [1]?
There is something we can do in PyTables?
[1] https://github.com/PyTables/PyTables/issues/175
--
Antonio Valentino
--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Runtime warning and import error after installation
Hi Serena Rhie, Il giorno 20/ago/2012, alle ore 08:40, PyTables Org pytab...@googlemail.com ha scritto: Forwarding bounced message. ~Josh From: Arang Rhie arr...@gmail.com Date: August 20, 2012 3:59:06 AM GMT+01:00 To: pytables-users@lists.sourceforge.net Subject: Runtime warning and import error after installation Hi, I'm new to PyTable, and now getting really frustrating with the installation process. 1. What is the HDF5 runtime? I've installed HDF5 following the instruction given by HDF5. What file is PyTable looking for? And how can I set the path to it? I've installed HDF5 under /usr/local/hdf5 and set the environment variable $HDF5_DIR on /usr/local/hdf5. this is very strange. Can you please check it again and, in case, file a new issue on [1] But still setup.py is not recognizing it without the --hdf5=/usr/local/hdf5 option. Moreover, I don't understand the 'shared library' of HDF5. does that mean to install hdf5-1.8.9-linux-x86_64-shared.tar.gz from HDF5? I've untar the ~shared.tar.gz under /usr/local/hdf5-shared and set the $LD_LIBRARY_PATH as /usr/local/hdf5-shared. If you maintained the layout of the official tar ball then probably you should set LD_LIBRARY_PATH to point to /usr/local/hdf5-shared/lib (please note the ending /lib). What else do I need to do in addition? 2. I've configured LZO with --enable-shared option, and followed the instruction (make, make check, make test, make install). I've completed without any error, and the shared libraries where set under /usr/local/lib. Do I need to explicitly set other variables for running setup.py? No, it should be OK. If the /usr/local/lib/liblzo2.so actually exists the setup script should not complain about a missing runtime. Can you please check and let us know. [root@gmi-student tables-2.4.0]# python setup.py install --hdf5=/usr/local/hdf5 * Found numpy 1.6.2 package installed. * Found numexpr 2.0.1 package installed. * Found Cython 0.16 package installed. * Found HDF5 headers at ``/usr/local/hdf5/include``, library at ``/usr/local/hdf5/lib``. .. WARNING:: Could not find the HDF5 runtime. The HDF5 shared library was *not* found in the default library paths. In case of runtime problems, please remember to install it. * Found LZO 2 headers at ``/usr/local/include``, library at ``/usr/local/lib``. .. WARNING:: Could not find the LZO 2 runtime. The LZO 2 shared library was *not* found in the default library paths. In case of runtime problems, please remember to install it. * Skipping detection of LZO 1 since LZO 2 has already been found. * Found bzip2 headers at ``/usr/include``, library at ``/usr/lib64``. running install running build running build_py running build_ext running build_scripts running install_lib running install_scripts changing mode of /usr/bin/ptdump to 755 changing mode of /usr/bin/ptrepack to 755 changing mode of /usr/bin/nctoh5 to 755 running install_egg_info Removing /usr/lib64/python2.7/site-packages/tables-2.4.0-py2.7.egg-info Writing /usr/lib64/python2.7/site-packages/tables-2.4.0-py2.7.egg-info [root@gmi-student tables-2.4.0]# python Python 2.7 (r27:82500, Sep 16 2010, 18:02:00) [GCC 4.5.1 20100907 (Red Hat 4.5.1-3)] on linux2 Type help, copyright, credits or license for more information. import tables Traceback (most recent call last): File stdin, line 1, in module File tables/__init__.py, line 30, in module from tables.utilsExtension import getPyTablesVersion, getHDF5Version ImportError: libhdf5.so.7: cannot open shared object file: No such file or directory Serena Rhie best regards [1] https://github.com/PyTables/PyTables/issues -- Antonio Valentino -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Pytables file reading
Hi Juan Manuel, Il 04/08/2012 01:55, Juan Manuel Vázquez Tovar ha scritto: Hello all, I´m managing a file close to 26 Gb size. It´s main structure is a table with a bit more than 8 million rows. The table is made by four columns, the first two columns store names, the 3rd one has a 53 items array in each cell and the last column has a 133x6 matrix in each cell. I use to work with a Linux workstation with 24 Gb. My usual way of working with the file is to retrieve, from each cell in the 4th column of the table, the same row from the 133x6 matrix. I store the information in a bumpy array with shape 8e6x6. In this process I almost use the whole workstation memory. Is there anyway to optimize the memory usage? I'm not sure to understand. My impression is that you do not actually need to have the entire 8e6x6 matrix in memory at once, is it correct? In that case you could simply try to load less data using something like data = table.read(0, 5e7, field='name of the 4-th field') process(data) data = table.read(5e7, 1e8, field='name of the 4-th field') process(data) See also [1] and [2]. Does it make sense for you? [1] http://pytables.github.com/usersguide/libref.html#table-methods-reading [2] http://pytables.github.com/usersguide/libref.html#tables.Table.read If not, I have been thinking about splitting the file. Thank you, Juanma cheers -- Antonio Valentino -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Pytables file reading
Hi Juan Manuel, Il 05/08/2012 22:28, Juan Manuel Vázquez Tovar ha scritto: Hi Antonio, You are right, I don´t need to load the entire table into memory. The fourth column has multidimensional cells and when I read a single row from every cell in the column, I almost fill the workstation memory. I didn´t expect that process to use so much memory, but the fact is that it uses it. May be I didn´t explain very well last time. Thank you, Juanma Sorry, still don't understand. Can you please post a short code snipped that shows how exactly do you read data into your program? My impression is that somewhere you use some instruction that triggers loading of unnecessary data into memory. 2012/8/5 Antonio Valentino antonio.valent...@tiscali.it Hi Juan Manuel, Il 04/08/2012 01:55, Juan Manuel Vázquez Tovar ha scritto: Hello all, I´m managing a file close to 26 Gb size. It´s main structure is a table with a bit more than 8 million rows. The table is made by four columns, the first two columns store names, the 3rd one has a 53 items array in each cell and the last column has a 133x6 matrix in each cell. I use to work with a Linux workstation with 24 Gb. My usual way of working with the file is to retrieve, from each cell in the 4th column of the table, the same row from the 133x6 matrix. I store the information in a bumpy array with shape 8e6x6. In this process I almost use the whole workstation memory. Is there anyway to optimize the memory usage? I'm not sure to understand. My impression is that you do not actually need to have the entire 8e6x6 matrix in memory at once, is it correct? In that case you could simply try to load less data using something like data = table.read(0, 5e7, field='name of the 4-th field') process(data) data = table.read(5e7, 1e8, field='name of the 4-th field') process(data) See also [1] and [2]. Does it make sense for you? [1] http://pytables.github.com/usersguide/libref.html#table-methods-reading [2] http://pytables.github.com/usersguide/libref.html#tables.Table.read If not, I have been thinking about splitting the file. Thank you, Juanma cheers -- Antonio Valentino -- Antonio Valentino -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Pytables file reading
Hi Juan Manuel, Il 05/08/2012 22:52, Juan Manuel Vázquez Tovar ha scritto: Hi Antonio, This is the piece of code I use to read the part of the table I need: data = [case[´loads´][i] for case in table] where i is the index of the row that I need to read from the matrix (133x6) stored in each cell of the column loads. Juanma that looks perfectly fine to me. No idea about what could be the issue :/ You can perfform patrial reads using Table.iterrows: data = [case[´loads´][i] for case in table.iterrows(start, stop)] Please also consider that using a single np.array with 1e8 rows instead of a list of arrays will allows you to save the memory overhead of 1e8 array objects. Considering that 6 doubles are 48 bytes while an empty np.array takes 80 bytes In [64]: sys.getsizeof(np.zeros((0,))) Out[64]: 80 you should be able to reduce the memory footprint by far more than an half. cheers 2012/8/5 Antonio Valentino antonio.valent...@tiscali.it Hi Juan Manuel, Il 05/08/2012 22:28, Juan Manuel Vázquez Tovar ha scritto: Hi Antonio, You are right, I don´t need to load the entire table into memory. The fourth column has multidimensional cells and when I read a single row from every cell in the column, I almost fill the workstation memory. I didn´t expect that process to use so much memory, but the fact is that it uses it. May be I didn´t explain very well last time. Thank you, Juanma Sorry, still don't understand. Can you please post a short code snipped that shows how exactly do you read data into your program? My impression is that somewhere you use some instruction that triggers loading of unnecessary data into memory. -- Antonio Valentino -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] PyTables Simultaneous Read Write from Current File
Hi all, Il 14/07/2012 00:44, Josh Ayers ha scritto: My first instinct would be to handle all access (read and write) to that file from a single process. You could create two multiprocessing.Queue objects, one for data to write and one for read requests. Then the process would check the queues in a loop and handle each request serially. The data read from the file could be sent back to the originating process using another queue or pipe. You should be able to do the same thing with sockets if the other parts of your application are in languages other than Python. I do something similar to handle writing to a log file from multiple processes and it works well. In that case the file is write-only - and just a simple text file rather than HDF5 - but I don't see any reason why it wouldn't work for read and write as well. Hope that helps, Josh I totally agree with Josh. I don't have a test code to demonstrate it but IMHO parallelizing I/O to/from a single file on a single disk do not makes too much sense unless you have special HW. Is this your case Jacob? IMHO with standard SATA devices you could have a marginal speedup (in the best case), but if your bottleneck is the I/O this will not solve your problem. If someone finds the time to implement a toy example of what Josh suggested we could put it on the cookbook :) regards On Fri, Jul 13, 2012 at 12:18 PM, Anthony Scopatz scop...@gmail.com wrote: On Fri, Jul 13, 2012 at 2:09 PM, Jacob Bennett jacob.bennet...@gmail.com wrote: [snip] My first implementation was to have a set of current files stay in write mode and have an overall lock over these files for the current day, but (stupidly) I forgot that lock instances cannot be shared over separate processes, only threads. So could you give me any advice in this situation? I'm sure it has come up before. ;) Hello All, I previously suggested to Jacob a setup where only one proc would have a write handle and all of the other processes would be in read-only mode. I am not sure that this would work. Francesc, Antonio, Josh, etc or anyone else, how would you solve this problem where you may want many processors to query the file, while something else may be writing to it? I defer to people with more experience... Thanks for your help! Be Well Anthony Thanks, Jacob Bennett -- Antonio Valentino -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] [Pytables-announce] ANN: PyTables 2.4.0 beta1
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Francesc, Il 11/07/2012 11:33, Francesc Alted ha scritto: Hey Antonio, this looks great. Thanks :) BTW, which is the status of the 3.x support? I vaguely remember you asking me for some help on this, but I don't remember well. Not that I have a lot of time to spend on it, but perhaps I can use some hours in the next days. well, in PyTables 2.4 we made some job in preparation of the porting to python3, but the porting itself is still not started. One of the main issues is that numexpr still not supports python3 so we have a missing dependency. I started a porting of numexpr to python3 (see [1]) but it is still incomplete. I hope it is good enough to let us start working on the porting of PyTables. Of course if you would like to give a look to numexpr for python3 it would be of great help. After the PyTables 2.4 final I plan to publish a wiki page with my roadmap proposal. IMHO main points are: * open a new branch in the repo * remove al deprecated code (Numeric, numarray, netcdf3, etc). This breaks the API and, IMHO, we will also need to bump the format version * ensure that all the required SW work (enough) on python3 * handle str/unicode issues * full support to unicode HDF5 object names * start working an a good setup for 2to3 (needs some investigation) * ... Please let me know if you think there are other point that are important for python3 support [1] https://groups.google.com/forum/?fromgroups#!topic/numexpr/M2MVjXsBR0c cheers Thanks, Francesc On 7/7/12 8:47 PM, Antonio Valentino wrote: === Announcing PyTables 2.4.0b1 === We are happy to announce PyTables 2.4.0b1. [CUT] - -- Antonio Valentino -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk/9soUACgkQ1JUs2CS3bP4u1ACeJKnMQRFF1hATXFMG3lPH2xyU 9DwAoJNPp6L8gHf+s5hA2Jhj4JLyl3jr =AYqd -END PGP SIGNATURE- -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] ANN: PyTables 2.4.0 beta1
Hi Christoph, Il 08/07/2012 18:21, Christoph Gohlke ha scritto: Hi Antonio, here's the stderr output: HDF5-DIAG: Error detected in HDF5 (1.8.8) thread 0: #000: ..\..\hdf5-1.8.8\src\H5F.c line 1522 in H5Fopen(): unable to open file major: File accessability minor: Unable to open file #001: ..\..\hdf5-1.8.8\src\H5F.c line 1313 in H5F_open(): unable to read superblock major: File accessability minor: Read failed #002: ..\..\hdf5-1.8.8\src\H5Fsuper.c line 334 in H5F_super_read(): unable to find file signature major: File accessability minor: Not an HDF5 file #003: ..\..\hdf5-1.8.8\src\H5Fsuper.c line 155 in H5F_locate_signature(): unable to find a valid file signature major: Low-level I/O minor: Unable to initialize object Christoph thank you. This is strange, HDF5-DIAG actually is in the output so you should not have the reported error: == FAIL: test_enable_messages (tables.tests.test_basics.HDF5ErrorHandling) -- Traceback (most recent call last): File X:\Python27-x64\lib\site-packages\tables\tests\common.py, line 259, in newmethod return oldmethod(self, *args, **kwargs) File X:\Python27-x64\lib\site-packages\tables\tests\test_basics.py, line 2445, in test_enable_messages self.assertTrue(HDF5-DIAG in stderr) AssertionError: False is not true Looking at the doc is seems that there is no particular issue at subprocess level. Umm, can you please check if the message actually comes form the stderr and not form the stdout? thanks On 7/8/2012 3:55 AM, Antonio Valentino wrote: Hi Christoph, thank you for reporting. Can you please tell us which is the output of the attached script on your machine? thanks in advance Il 07/07/2012 21:18, Christoph Gohlke ha scritto: Looks good. Only one test failure on win-amd64-py2.7 (attached). Christoph On 7/7/2012 11:47 AM, Antonio Valentino wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 === Announcing PyTables 2.4.0b1 === [CUT] -- Antonio Valentino -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] ANN: PyTables 2.4.0 beta1
Thank you very much Christoph. Il 08/07/2012 19:41, Christoph Gohlke ha scritto: I submitted a PR at https://github.com/PyTables/PyTables/pull/161 Christoph On 7/8/2012 10:06 AM, Antonio Valentino wrote: Hi Christoph, Il 08/07/2012 18:21, Christoph Gohlke ha scritto: Hi Antonio, here's the stderr output: HDF5-DIAG: Error detected in HDF5 (1.8.8) thread 0: #000: ..\..\hdf5-1.8.8\src\H5F.c line 1522 in H5Fopen(): unable to open file major: File accessability minor: Unable to open file #001: ..\..\hdf5-1.8.8\src\H5F.c line 1313 in H5F_open(): unable to read superblock major: File accessability minor: Read failed #002: ..\..\hdf5-1.8.8\src\H5Fsuper.c line 334 in H5F_super_read(): unable to find file signature major: File accessability minor: Not an HDF5 file #003: ..\..\hdf5-1.8.8\src\H5Fsuper.c line 155 in H5F_locate_signature(): unable to find a valid file signature major: Low-level I/O minor: Unable to initialize object Christoph thank you. This is strange, HDF5-DIAG actually is in the output so you should not have the reported error: == FAIL: test_enable_messages (tables.tests.test_basics.HDF5ErrorHandling) -- Traceback (most recent call last): File X:\Python27-x64\lib\site-packages\tables\tests\common.py, line 259, in newmethod return oldmethod(self, *args, **kwargs) File X:\Python27-x64\lib\site-packages\tables\tests\test_basics.py, line 2445, in test_enable_messages self.assertTrue(HDF5-DIAG in stderr) AssertionError: False is not true Looking at the doc is seems that there is no particular issue at subprocess level. Umm, can you please check if the message actually comes form the stderr and not form the stdout? thanks On 7/8/2012 3:55 AM, Antonio Valentino wrote: Hi Christoph, thank you for reporting. Can you please tell us which is the output of the attached script on your machine? thanks in advance Il 07/07/2012 21:18, Christoph Gohlke ha scritto: Looks good. Only one test failure on win-amd64-py2.7 (attached). Christoph On 7/7/2012 11:47 AM, Antonio Valentino wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 === Announcing PyTables 2.4.0b1 === [CUT] -- Antonio Valentino -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] closing an erroneous file
Ciao Daniele, Il giorno Mon, 25 Jun 2012 09:17:00 +0200 Mythsmith s...@modena1.it ha scritto: Hi Anthony, Shouldn't the close() method also clear the cache? I think a file should be either opened or closed... Should I file a bug report? Best regards, Daniele The close method also remover the file from the cache if there are no more references to it https://github.com/PyTables/PyTables/blob/6fccb7495ba1bc758c7b04960fe1cd392abe9b96/tables/file.py#L2098 Anyway yes, if you have some problem with the file caching system please file a bug report on github. Of course test scripts or patches are very welcome. ciao Il 21/06/2012 19:23, Anthony Scopatz ha scritto: Hi Daniele, This is probably because of the way PyTables caches its file objects. As a temporary work around, why don't you try clearing the cache or at least removing this file. The cache is just a dictionary and it is located at tables.file._open_files. ie try: tables.file._open_files.clear() -or- del tables.file._open_files.pop[touch.h5] Be Well Anthony On Thu, Jun 21, 2012 at 10:43 AM, Mythsmith s...@modena1.it mailto:s...@modena1.it wrote: Hi, I noticed that if I open an erroneous file (eg: empty), then it seems not possible to completely close it and reopen the same path, even if a valid file was created in the meanwhile. The error is: ValueError: The file 'touch.h5' is already opened. Please close it before reopening in write mode. You find a complete example attached. Regards, daniele -- Antonio Valentino -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] createTable chokes on rich-dtype structured array
Hi Alvaro,
thank you for reporting.
I filed an issue on GitHub to track the problem:
https://github.com/PyTables/PyTables/issues/160
ciao
Il 25/06/2012 12:03, Alvaro Tejero Cantero ha scritto:
Hi,
In view of the upcoming release I thought I'd report this because at
the time I cannot fix it myself:
I am using a structured array with a dtype specified with the
following numpy-accepted
format (quotation follows from
http://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html):
[(field_name, field_dtype, field_shape), ...]
obj should be a list of fields where each field is described by a
tuple of length 2 or 3. (Equivalent to the descr item in the
__array_interface__ attribute.)
The first element, field_name, is the field name (if this is '' then a
standard field name, 'f#', is assigned). The field name may also be a
2-tuple of strings where the first string is either a “title” (which
may be any string or unicode string) or meta-data for the field which
can be any object, and the second string is the “name” which must be a
valid Python identifier.
This is my concrete example:
header = [(('timestamp', 't'), 'u4'),
(('unit (cluster) id', 'unit'),'u2')]
This is what PyTables says upon passing either the structured array or
np.dtype(header) to the createTables function:
test.createTable('/', 'spike', s, 'test')
---
ValueErrorTraceback (most recent call last)
/home/tejero/Dropbox/O/ridge/doc/ipython-input-40-5fdbd9feb41d in module()
1 test.createTable('/', 'spike', s, 'test')
/home/tejero/Local/Envs/test/lib/python2.7/site-packages/tables/file.pyc
in createTable(self, where, name, description, title, filters,
expectedrows, chunkshape, byteorder, createparents)
768 description=description, title=title,
769 filters=filters, expectedrows=expectedrows,
-- 770 chunkshape=chunkshape, byteorder=byteorder)
771
772
/home/tejero/Local/Envs/test/lib/python2.7/site-packages/tables/table.pyc
in __init__(self, parentNode, name, description, title, filters,
expectedrows, chunkshape, byteorder, _log)
805 self._v_recarray = nparray
806 self.description, self._rabyteorder = \
-- 807 descr_from_dtype(nparray.dtype)
808
809 # No description yet?
/home/tejero/Local/Envs/test/lib/python2.7/site-packages/tables/description.pyc
in descr_from_dtype(dtype_)
723 fields = {}
724 fbyteorder = '|'
-- 725 for (name, (dtype, pos)) in dtype_.fields.items():
726 kind = dtype.base.kind
727 byteorder = dtype.base.byteorder
ValueError: too many values to unpack
-á.
--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
--
Antonio Valentino
--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Inheritance in
Hi Danid, hi Francesc, Il 31/03/2012 03:08, Francesc Alted ha scritto: On 3/30/12 7:57 PM, Daπid wrote: Hello, I have several different kinds of data tables, absolutely independent, defined as in the tutorial (http://pytables.github.com/usersguide/tutorials.html): [CUT] One, naively, saw the repetition and would want to do something like: class Pie(IsDescription): # All kind of pies dough=Int64Col() baking=Float64Col() class SaltedPie(Pie): anchovy=Float32Col() class SweetPie(Pie): apple=UInt32Col() but, when I try to set 'dough', I get: KeyError: 'no such column: dough' Of course, my approach is not correct. Is there a valid way of doing it? Right, subclassing IsDescription is not supported. Sorry, but I think that the only way is to do the repetition explicitly. Or, maybe you can use a NumPy dtype instead, that allows you to create table schemas more succinctly. Some work on in broken inheritance of IsDescriptor has been done in the development branch (see [#65]). Thanks to Andrea Bedini for the patch. Which version of PyTables are you using David? cheers [#65] https://github.com/PyTables/PyTables/issues/65 -- Antonio Valentino -- This SF email is sponsosred by: Try Windows Azure free for 90 days Click Here http://p.sf.net/sfu/sfd2d-msazure ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Merging multiple DB
Hi Danid, Il 11/03/2012 01:43, Daπid ha scritto: Thank you, Anthony. I will have a deep look and come back if I find more problems. On Sat, Mar 10, 2012 at 10:50 AM, Antonio Valentino antonio.valent...@tiscali.it wrote: http://pytables.github.com/usersguide/ch04.html#Table.append May I ask you where did you found that broken link? Yes. Here: http://www.pytables.org/moin/HintsForSQLUsers For appending a block of rows in a single shot, Table.append() is more adequate. But it looks like all the links to the reference are broken. OK, all links in that page should be fixed now. Thank you -- Antonio Valentino -- Virtualization Cloud Management Using Capacity Planning Cloud computing makes use of virtualization - but cloud computing also focuses on allowing computing to be delivered as a service. http://www.accelacomm.com/jaw/sfnl/114/51521223/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Merging multiple DB
Hi danid, Il 09/03/2012 10:40, Daπid ha scritto: The documentation page appears to be broken: http://pytables.github.com/usersguide/ch04.html#Table.append May I ask you where did you found that broken link? Maybe it is something we can fix. Best regards -- Antonio Valentino -- Virtualization Cloud Management Using Capacity Planning Cloud computing makes use of virtualization - but cloud computing also focuses on allowing computing to be delivered as a service. http://www.accelacomm.com/jaw/sfnl/114/51521223/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] ptrepack and windows drive letters
Hi Brad,
Il 24/02/2012 20:16, Brad Buran ha scritto:
Apologies if this is a repost. The original email was sent from an
unsubscribed address and I don't believe it went through.
When I type the command:
ptrepack g:/data/bnb/1016/physiology/111206_12101.hd5: c:/test.hd5:
I get the following error:
Traceback (most recent call last):
File C:\Python27\Scripts\ptrepack-script.py, line 10, in module
sys.exit(main())
File C:\Python27\lib\site-packages\tables\scripts\ptrepack.py, line 395,
in
main
srcfile, srcnode = src
ValueError: too many values to unpack
This appears to be due to the fact that the code assumes that : can
only be used as a separator between the file name and node path. May
I propose changing lines 390 391 in scripts/ptrepack.py from:
# Catch the files passed as the last arguments
src = pargs[0].split(':')
dst = pargs[1].split(':')
To:
# Catch the files passed as the last arguments
src = pargs[0].rsplit(':', 1)
dst = pargs[1].rsplit(':', 1)
This should alleviate the issue.
Brad
Thank you for reporting and for the patch.
I have opened a new issue on github:
https://github.com/PyTables/PyTables/issues/133
best regards
--
Antonio Valentino
--
Virtualization Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/
___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] User block in HDF5 files
Hi Brad, Il 19/01/2012 04:13, Brad Buran ha scritto: Hi Antonio: The user block I am referring to is a region at the beginning of the file that is reserved for user metadata. I pasted the description from the HDF5 documentation below (http://www.hdfgroup.org/HDF5/doc1.6/UG/08_TheFile.html). I believe H5Py added this feature (http://code.google.com/p/h5py/source/detail?r=8a3010d07e14); however, I don't see a similar way to do this via PyTables. I suppose I could use h5py to create a blank HDF5 file with the required user block then reopen it in PyTables, however, I'd like to minimize code dependencies and everything I've written is built around PyTables rather than h5py. Sorry, for the misunderstanding, I was not aware of this feature. Please file a feature request on https://github.com/PyTables/PyTables best regards Thanks! Brad Documentation from HDF5: User-block size herr_t H5Pset_userblock (hid_t plist, hsize_t size) herr_t H5Pget_userblock (hid_t plist, hsize_t *size) The user-block is a fixed-length block of data located at the beginning of the file and which is ignored by the HDF5 library. This block is specifically set aside for any data or information that developers determine to be useful to their application but that will not be used by the HDF5 library. The size of the user-block is defined in bytes and may be set to any power of two, with a minimum size of 512 bytes (i.e. 512, 1024, 2048, etc). This property is set with H5Pset_userblock and queried via H5Pget_userblock. For example, if an application was thought to reqire a 4K user-block, that could be set with the following function call: status = H5Pset_userblock(fcpl_id, 4096) The property list could later be queried with status = H5Pget_userblock(fcpl_id, size) and the value 4096 would be returned in the parameter size. On Wed, Jan 18, 2012 at 4:52 PM, Antonio Valentino antonio.valent...@tiscali.it wrote: Hi Brad, Il 18/01/2012 21:49, Brad Buran ha scritto: Is there a way to set the size of the user block when creating a file in PyTables? This would be useful for using PyTables to generate Matlab 7.3 compatible files. Brad Try to use CArrays and set the chunkshape explicitly http://pytables.github.com/usersguide/libref.html#tables.File.createCArray Does it answers your question? best regards -- Antonio Valentino -- Antonio Valentino -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] User block in HDF5 files
Hi Brad, Il 18/01/2012 21:49, Brad Buran ha scritto: Is there a way to set the size of the user block when creating a file in PyTables? This would be useful for using PyTables to generate Matlab 7.3 compatible files. Brad Try to use CArrays and set the chunkshape explicitly http://pytables.github.com/usersguide/libref.html#tables.File.createCArray Does it answers your question? best regards -- Antonio Valentino -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Blocked access using PyTables and matrix multiplication
Hi Brad,
Il 10/12/2011 20:36, Brad Buran ha scritto:
I am trying to speed up some analysis routines on arrays that are
approximately 16 x 200,000,000 elements (stored in HDF5 arrays that
were originally created with PyTables). I was looking into whether I
could speed up the analysis using tricks such as memmap and numexpr;
however, since I need to perform row-wise operations (e.g. computing
the dot product with a 16x16 array followed by a scipy.signal.filter
operation) which requires indexing, I do not believe I can use
numexpr. This leaves memmaping, but I understand that PyTables
offers something similar. I found a very old discussion on this
mailing list
(http://www.mail-archive.com/pytables-users@lists.sourceforge.net/msg01295.html),
but the link Francesc provided to the slides from Euro Scipy
describing how to use a blocking technique with PyTables no longer
works. Does anyone have access to the original slides?
Probably material you are looking for is at
http://www.pytables.org/moin/HowToUse#Presentations
I'm assuming that the blocking technique is as simple as determining a
chunk size to operate on and then looping through the PyTables Array,
loading the chunk into memory, running np.dot and scipy.signal.filter
and saving the result to a new PyTables array, but I was curious to
see if the slides point out any subtleties of this approach that I
should be aware of.
If I understand correctly, the blocking approach is as simple as the
following:
# note that diff is a 16x16 array
source = f_in.root.data
dest = f_in.createCArray('/', 'result', atom=tables.Float32Atom(),
size=source.size)
temp = np.empty((16, chunksize))
for chunk in range(n_chunks):
block = source[:, chunk*chunksize:chunk*chunksize+chunksize]
result = np.dot(diff, block, out=temp)
result = scipy.signal.filtfilt(b, a, result)
dest[:, chunk*chunksize:chunk*chunksize+chunksize] = result
Thanks!
Brad
Yes, this is the idea.
Surely Francesc can provide very useful hints about this topic.
On my part I can suggest you to choose very carefully the chunk shape
when you generate your datasets.
Best regards
--
Antonio Valentino
--
Learn Windows Azure Live! Tuesday, Dec 13, 2011
Microsoft is holding a special Learn Windows Azure training event for
developers. It will provide a great way to learn Windows Azure and what it
provides. You can attend the event by watching it streamed LIVE online.
Learn more at http://p.sf.net/sfu/ms-windowsazure
___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Importing from Matlab's sparse format
Hi Tom, Il 09/12/2011 14:12, Tom Diethe ha scritto: I have files stored using Matlab's sparse format (HDF5, csc I believe), and I'm trying to use Pytables to operate on them directly, but haven't succeeded yet. Using h5py I can do the following: # Method 1: uses h5py (WORKS) f1 = h5py.File(fname) data = f1['M']['data] ir = f1['M]['ir'] jc = f1['M']['jc'] M = scipy.sparse.csc_matrix( (data,ir,jc) ) but if I try to do the equivalent in Pytables: # Method 2: uses pyTables (DOESN'T WORK) f2 = tables.openFile(fname) data = f2.root.M.data ir = f2.root.M.ir jc = f2.root.M.jc M = scipy.sparse.csc_matrix( (data,ir,jc) ) If my understanding is correct before calling csc_matrix you should actually read data from disk data = f2.root.M.data[...] ir = f2.root.M.ir[...] jc = f2.root.M.jc[...] Please note that f3.root.M.data in a pytables object, and not a numpy array In [23]: f2.root.M.data Out[23]: /M/data (Array(20,)) '' atom := Float64Atom(shape=(), dflt=0.0) maindim := 0 flavor := 'numpy' byteorder := 'little' chunkshape := None this fails (after a long wait) with the error: TypeError Traceback (most recent call last) /home/tdiethe/BMJ/ipython console in module() /usr/lib/python2.6/dist-packages/scipy/sparse/compressed.pyc in __init__(self, arg1, shape, dtype, copy, dims, nzmax) 56 self.indices = np.array(indices, copy=copy) 57 self.indptr = np.array(indptr, copy=copy) --- 58 self.data= np.array(data, copy=copy, dtype=getdtype(dtype, data)) 59 else: 60 raise ValueError, unrecognized %s_matrix constructor usage %\ /usr/lib/python2.6/dist-packages/scipy/sparse/sputils.pyc in getdtype(dtype, a, default) 69 canCast = False 70 else: --- 71 raise TypeError, could not interpret data type 72 else: 73 newdtype = np.dtype(dtype) TypeError: could not interpret data type I have two questions: - how do I load this file in - do I need to perform the conversion to a scipy sparse matrix in order to be able to perform operations on it, or can I perform operations directly on the disk files (matrix multiplication etc)? Not sure to understand the second question but I guess the answer is yes, unless it is a trivial element-by-element operation. Best regards -- Antonio Valentino -- Learn Windows Azure Live! Tuesday, Dec 13, 2011 Microsoft is holding a special Learn Windows Azure training event for developers. It will provide a great way to learn Windows Azure and what it provides. You can attend the event by watching it streamed LIVE online. Learn more at http://p.sf.net/sfu/ms-windowsazure ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Ubuntu 11.10: blosc is not supported?
Hi Francesc, hi Martin, Il 06/12/2011 14:55, Francesc Alted ha scritto: 2011/12/6 PyTables Org pytab...@googlemail.com Forwarding to the list. ~Josh Begin forwarded message: *From: *pytables-users-boun...@lists.sourceforge.net *Date: *December 6, 2011 2:20:35 PM GMT+01:00 *To: *pytables-users-ow...@lists.sourceforge.net *Subject: **Auto-discard notification* The attached message has been automatically discarded. *From: *Martin Felder martin.fel...@zsw-bw.de *Date: *December 6, 2011 2:05:15 PM GMT+01:00 *To: *pytables-users@lists.sourceforge.net *Subject: **Ubuntu 11.10: blosc is not supported?* Hi, I installed pytables via the Ubuntu package manager (currently version 2.1.2-3.1build1), and we use it a lot for production work. Thanks for this great package! So far I haven't tried enabling compression, but since it says in the documentation BLOSC comes with it, I created a filter with complib=blosc, only to get: ValueError: compression library ``blosc`` is not supported; it must be one of: zlib, lzo, bzip2 Do I have to compile a newer version from source to enable BLOSC? Yes, you need at least PyTables 2.2 for using Blosc. Antonio has recently released PyTables binaries for Debian in: http://sourceforge.net/projects/pytables/files/pytables/2.3.1/ that might be useful for Ubuntu too. Yes it should work but it is only for amd64. Users of Ubuntu 11.10 can use the following PPA: https://launchpad.net/~a.valentino/+archive/eotools I'm trying to ush it in the official debian/ubuntu archives but I have serious problems to contact current maintainers. cheers -- Antonio Valentino -- Cloud Services Checklist: Pricing and Packaging Optimization This white paper is intended to serve as a reference, checklist and point of discussion for anyone considering optimizing the pricing and packaging model of a cloud services business. Read Now! http://www.accelacomm.com/jaw/sfnl/114/51491232/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Problem with new PyTables installation on Debian - ImportError: No module named utilsExtension
Hi Alok, Il giorno Tue, 29 Nov 2011 17:31:01 -0500 Alok Mohindra alok.mohin...@gmail.com ha scritto: Hi, I am excited about the potential of PyTables for a couple of big data projects I am working on but I have run into an issue which I don't know how to resolve. This is my first PyTables installation and I am installing it on the latest version of Linux Mint 64-bit based on Debian/Ubuntu. I have tried installing PyTables in two ways, first in a virtualenv with pip and easy_install along with manual installation of hdf5, numpy and numexpr, Two questions: * which is the HDF5 version: 1.6.x or 1.8.x? * do you have a C compiler installed? and secondly as a system-wide installation using apt-get build-dep python-tables. I recently released debian packages for amd64: http://sourceforge.net/projects/pytables/files/pytables/2.3.1/ why don't you give it a try? With either approach, I am getting the same error when I try to 'from tables import *' or 'import tables' from either python or ipython. The error is ImportError: No module named utilsExtension While there are some older comments on the mailing list discussing this error ( http://blog.gmane.org/gmane.comp.python.pytables.user/month=20100401), they do not appear to apply to my problem. Can anyone suggest the simplest path to a successful installation of PyTables on a debian-derived Linux? I am anxious to get started if I can get this resolved quickly. thanks, --Alok You could try to install pytables manually from the source tarball without using pip or similar: $ python setup.py install or, if you are not in a virtual env $ python setup.py install --prefix=INSTALL_PREFIX Please also ensure that you have cython correctly installed. best regards -- Antonio Valentino -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-novd2d ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
[Pytables-users] ANN: PyTables 2.3.1
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 === Announcing PyTables 2.3.1 === This is a bugfix release. Upgrading is recommended for users that are running PyTables in production environments. What's new == This release includes a small number of changes. It only fixes a couple of bugs that are considered serious even if they should not impact a large number of users: - - :issue:`113` caused installation of PyTables 2.3 to fail on hosts with multiple python versions installed. - - :issue:`111` prevented to read scalar datasets of UnImplemented types. In case you want to know more in detail what has changed in this version, have a look at: http://pytables.github.com/release_notes.html You can download a source package with generated PDF and HTML docs, as well as binaries for Windows, from: http://sourceforge.net/projects/pytables/files/pytables/@VERSION@ For an on-line version of the manual, visit: http://pytables.github.com/usersguide/index.html What it is? === PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. PyTables includes OPSI, a new indexing technology, allowing to perform data lookups in tables exceeding 10 gigarows (10**10 rows) in less than 1 tenth of a second. Resources = About PyTables: http://www.pytables.org About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ Acknowledgments === Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Most specially, a lot of kudos go to the HDF5 and NumPy (and numarray!) makers. Without them, PyTables simply would not exist. Share your experience = Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. - **Enjoy data!** - -- The PyTables Team -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6rHAkACgkQ1JUs2CS3bP70gACgpdlVdblRzwgKtNZyWWFjUtf3 GgAAoL0/ji/6NMTVJeRxYCm4FXZJ8vpd =3kQm -END PGP SIGNATURE- -- The demand for IT networking professionals continues to grow, and the demand for specialized networking skills is growing even more rapidly. Take a complimentary Learning@Cisco Self-Assessment and learn about Cisco certifications, training, and career opportunities. http://p.sf.net/sfu/cisco-dev2dev ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
[Pytables-users] ANN: PyTables 2.3 released
=== Announcing PyTables 2.3 === We are happy to announce PyTables 2.3. This release comes after about 10 months of development and after that Francesc Altet, the creator of PyTables, ceased activities with the project. Thank you Francesc. Also the project has been moved to GitHub: http://github.com/PyTables/PyTables. What's new == The main new features in 2.3 series are: * PyTables now includes the codebase of PyTables Pro (now release under open source license) gaining a lot of performance improvements and some new features like: - the new and powerful indexing engine: OPSI - a fine-tuned LRU cache for both metadata (nodes) and regular data * The entire documentation set has been converted to ReStructuredTest and Sphinx As always, a large amount of bugs have been addressed and squashed too. In case you want to know more in detail what has changed in this version, have a look at: http://pytables.github.com/release_notes.html You can download a source package with generated PDF and HTML docs, as well as binaries for Windows, from: http://sourceforge.net/projects/pytables/files/pytables/2.3 For an on-line version of the manual, visit: http://pytables.github.com/usersguide/index.html What it is? === PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. PyTables includes OPSI, a new indexing technology, allowing to perform data lookups in tables exceeding 10 gigarows (10**10 rows) in less than 1 tenth of a second. Resources = About PyTables: http://www.pytables.org About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ Acknowledgments === Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Most specially, a lot of kudos go to the HDF5 and NumPy (and numarray!) makers. Without them, PyTables simply would not exist. Share your experience = Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. **Enjoy data!** -- The PyTables Team -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
[Pytables-users] ANN: PyTables 2.3 released
=== Announcing PyTables 2.3 === We are happy to announce PyTables 2.3. This release comes after about 10 months of development and after that Francesc Altet, the creator of PyTables, ceased activities with the project. Thank you Francesc. Also the project has been moved to GitHub: http://github.com/PyTables/PyTables. What's new == The main new features in 2.3 series are: * PyTables now includes the codebase of PyTables Pro (now release under open source license) gaining a lot of performance improvements and some new features like: - the new and powerful indexing engine: OPSI - a fine-tuned LRU cache for both metadata (nodes) and regular data * The entire documentation set has been converted to ReStructuredTest and Sphinx As always, a large amount of bugs have been addressed and squashed too. In case you want to know more in detail what has changed in this version, have a look at: http://pytables.github.com/release_notes.html You can download a source package with generated PDF and HTML docs, as well as binaries for Windows, from: http://sourceforge.net/projects/pytables/files/pytables/2.3 For an on-line version of the manual, visit: http://pytables.github.com/usersguide/index.html What it is? === PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. PyTables includes OPSI, a new indexing technology, allowing to perform data lookups in tables exceeding 10 gigarows (10**10 rows) in less than 1 tenth of a second. Resources = About PyTables: http://www.pytables.org About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ Acknowledgments === Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Most specially, a lot of kudos go to the HDF5 and NumPy (and numarray!) makers. Without them, PyTables simply would not exist. Share your experience = Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. **Enjoy data!** -- The PyTables Team -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
[Pytables-users] ANN: PyTables 2.3rc1 released
=== Announcing PyTables 2.3rc1 === We are happy to announce PyTables 2.3. This release comes after abour 10 months of development and after that Francesc Altet, the creator of PyTables, ceased activities with the project. Thank you Francesc. Also the project has been moved to GitHub: http://github.com/PyTables/PyTables. What's new == The main new features in 2.3 series are: * PyTables now includes the codebase of PyTables Pro (now release under open source license) gaining a lot of performance improvements and some new features like: - the new and powerful indexing engine: OPSI - a fine-tuned LRU cache for both metadata (nodes) and regular data * The entire documentation set has been converted to ReStructuredTest and Sphinx As always, a large amount of bugs have been addressed and squashed too. In case you want to know more in detail what has changed in this version, have a look at: http://pytables.github.com/release_notes.html You can download a source package with generated PDF and HTML docs, as well as binaries for Windows, from: http://sourceforge.net/projects/pytables/files/pytables/2.3rc1 For an on-line version of the manual, visit: http://pytables.github.com/usersguide/index.html What it is? === PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. PyTables includes OPSI, a new indexing technology, allowing to perform data lookups in tables exceeding 10 gigarows (10**10 rows) in less than 1 tenth of a second. Resources = About PyTables: http://www.pytables.org About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ Acknowledgments === Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Most specially, a lot of kudos go to the HDF5 and NumPy (and numarray!) makers. Without them, PyTables simply would not exist. Share your experience = Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. **Enjoy data!** -- The PyTables Team -- Using storage to extend the benefits of virtualization and iSCSI Virtualization increases hardware utilization and delivers a new level of agility. Learn what those decisions are and how to modernize your storage and backup environments for virtualization. http://www.accelacomm.com/jaw/sfnl/114/51434361/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] setup.py detection of hdf5 runtim
Hi Chris, Il 02/09/2011 16:47, Chris Kees ha scritto: Hi, Would it be possible to patch the setup.py so that it searches for the runtime libraries in the provided path for hdf5 rather than just a set of default directories? We frequently have to install on systems where we don't have access to the default directories and have to append to LD_LIBRARY_PATH/DYLD_LIBRARY_PATH. The current version setup.py always generates a warning for us (and is actually finding the wrong hdf5 runtime when it DOESN'T generate a warning). Here's what I'm proposing: With The current version setup.py do you mean the one from the PyTables 2.2 package tarball? In case, it would be nice if you could give a try to the latest version currently in master on github https://github.com/PyTables/PyTables/ There is another small patch on my personal area https://github.com/avalentino/PyTables/commit/6e6c9ad7a09bfd97310a2ef8ed1ea526b11acc1e that could be useful in this case. Also it would be useful if you could give us more details (e.g. the exact command line, setup.py output, relevant environment and so on). --- a/setup.py +++ b/setup.py @@ -155,7 +155,7 @@ class Package(object): # An explicit path can not be provided for runtime libraries. # (The argument is accepted for compatibility with previous methods.) return _find_file_path( -self.runtime_name, default_runtime_dirs, +self.runtime_name, locations, self._runtime_prefixes, self._runtime_suffixes ) -Chris Thanks -- Antonio Valentino -- Special Offer -- Download ArcSight Logger for FREE! Finally, a world-class log management solution at an even better price-free! And you'll get a free Love Thy Logs t-shirt when you download Logger. Secure your free ArcSight Logger TODAY! http://p.sf.net/sfu/arcsisghtdev2dev ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] HDF5-DIAG errors that don't raise exceptions
Hi Stuart, Il 22/08/2011 05:59, Stuart Mentzer ha scritto: Hi Antonio, Thanks for the quick response. Hi Stuart, Il giorno 21/ago/2011, alle ore 02.38, Stuart Mentzer ha scritto: [CUT] Yes dontPrint is part of the C++ API, the C one has H5Eset_auto to should perform a similar task. Well I suppose we could provide a function to suppress HDF5 messages but IMHO it is more important to fix the issue regarding exception. With the proper error handling using HDF5ExtError I am able to stop the code quickly when the first problem arises but I still get about a page of HDF error messages. It would be great to be able to turn that off for a production release, so I would vote for adding a dontPrint call. I have just opened a ticket on GitHub https://github.com/PyTables/PyTables/issues/87 regards -- Antonio Valentino -- uberSVN's rich system and user administration capabilities and model configuration take the hassle out of deploying and managing Subversion and the tools developers use with it. Learn more about uberSVN and get a free download at: http://p.sf.net/sfu/wandisco-dev2dev ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] HDF5-DIAG errors that don't raise exceptions
Hi Stuart, Il giorno 21/ago/2011, alle ore 02.38, Stuart Mentzer ha scritto: Hello, Some kinds of HDF5-DIAG errors don't generate any exceptions making it difficult to do proper error handling. For example: HDF5-DIAG: Error detected in HDF5 (1.8.5) thread 0: #000: ..\..\..\src\H5Dio.c line 174 in H5Dread(): can't read data major: Dataset minor: Read failed doesn't raise an HDF5ExtError or any other exception. Is this something that can be fixed in PyTables? I just performed a quick check on all calls to H5Dread and it seems to me that in all cases there is a check for failures with relative HDF5ExtError exception raising. Can you please be a little more specific on how to reproduce the issue? I encourage you to open an issue on https://github.com/PyTables/PyTables Also, can the HDF5-DIAG messages be suppressed? I know that the HDF5 API has a dontPrint call but I don't see a PyTables interface to that. If we can't detect these exceptions I at least want to mask them from end users. Thanks! Yes dontPrint is part of the C++ API, the C one has H5Eset_auto to should perform a similar task. Well I suppose we could provide a function to suppress HDF5 messages but IMHO it is more important to fix the issue regarding exception. regards -- Antonio Valentino -- Get a FREE DOWNLOAD! and learn more about uberSVN rich system, user administration capabilities and model configuration. Take the hassle out of deploying and managing Subversion and the tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] git strategy to move forward with
Hi Chris, Il giorno 20/ago/2011, alle ore 03.57, Chris Kees ha scritto: Hi, We use pytables and numexpr, and I'd like to add them as submodules to our project so that I can easily keep updated with pytables improvements and pass changes back up stream if I can be of any help (we do a lot of HPC work with pytables). Is there a plan to add 2.3 as a release branch fairly soon or some other stable point in the git repository that I could point my submodule to? I could also stick with 2.2 for the time being but noticed below that 2.3beta had been used in EPD so I thought there must be a branch or fork somewhere that is The master branch should be beta quality at the moment. We plan to pack a beta release soon. If you look at https://github.com/PyTables/PyTables/issues?milestone=2state=open there are still a few bugs to close and they are mainly related to the setup process which seems to be your problem too. relatively stable. I would just go with the head of the master branch, but compiling with Cython 0.15 and numpy 1.6.1 gave me the error below. Thanks, Chris Chris, I'm setting up a virtual environment to reproduce your issue. Can you confirm that the cython executable is in your path? Looking at the error message below it seems that the setup script is unable to find the HDF5 runtime: .. WARNING:: Could not find the HDF5 runtime. Can you please post more detail on your system setup and the exact command line you used? * Found numpy 1.6.2.dev-f21011a package installed. * Found numexpr 1.5.dev package installed. * Found Cython 0.15 package installed. * Found HDF5 headers at ``/Users/cekees/src/proteus-git/darwin/include``, library at ``/Users/cekees/src/proteus-git/darwin/lib``. .. WARNING:: Could not find the HDF5 runtime. The HDF5 shared library was *not* found in the default library paths. In case of runtime problems, please remember to install it. * Could not find LZO 2 headers and library; disabling support for it. * Could not find LZO 1 headers and library; disabling support for it. * Found bzip2 headers at ``/usr/include``, library at ``/usr/lib``. cythoning tables/utilsExtension.pyx to tables/utilsExtension.c Traceback (most recent call last): File setup.py, line 464, in module cython_extfiles = get_cython_extfiles(cython_extnames) File setup.py, line 455, in get_cython_extfiles retcode = subprocess.call([cython, extpfile]) File /Users/cekees/src/proteus-git/darwin/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py, line 493, in call return Popen(*popenargs, **kwargs).wait() File /Users/cekees/src/proteus-git/darwin/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py, line 679, in __init__ errread, errwrite) File /Users/cekees/src/proteus-git/darwin/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py, line 1228, in _execute_child raise child_exception regards -- Antonio Valentino -- Get a FREE DOWNLOAD! and learn more about uberSVN rich system, user administration capabilities and model configuration. Take the hassle out of deploying and managing Subversion and the tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2 ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] 2.3 roadmap (proposal)
Hi Anthony, hi Josh, Il 06/08/2011 03:01, Anthony Scopatz ha scritto: Hello All, On Fri, Aug 5, 2011 at 2:51 PM, Josh Moore josh.mo...@gmx.de wrote: Chiming in late Also chiming in late. Oh, sorry guys if I left few time to respond. On Aug 4, 2011, at 10:17 AM, Antonio Valentino wrote: Hi list, hi developers, [CUT] All sounds fine. I'll certainly test on my Mac 10.6, and can do so in a virtual environment depending on what kind of coverage there is. Thank you Josh. I can perform tests on linux (64 bit) and, with a little ore effort, on Mac. No windows until September and, in any case, only win XP. It would be nice to have a wiki page (maybe Dev/Test) to track all test performed on the current development branch including details on the platform, library versions and so on. What do you think about? I can try to work on the docs this weekend. I have been deep in docs on Thank you Anthony. another project. Sphinx FTW! I made some thought about proposing to use sphinx for the user guide in the future. I'm a little bit dubious, it is a big task. [CUT] Cheers, ~Josh. P.S. Do we want to rename/close/merge the 2.3b1 milestone? https://github.com/PyTables/PyTables/issues?milestone=1state=open Maybe we could rename it. No idea about the new title, maybe something related to project organization. Also IMHO #4 can be closed (please check) and I would move #5 to milestone 2.3. [#4] https://github.com/PyTables/PyTables/issues/4 [#5] https://github.com/PyTables/PyTables/issues/5 Best regards -- Antonio Valentino -- BlackBerryreg; DevCon Americas, Oct. 18-20, San Francisco, CA The must-attend event for mobile developers. Connect with experts. Get tools for creating Super Apps. See the latest technologies. Sessions, hands-on labs, demos much more. Register early save! http://p.sf.net/sfu/rim-blackberry-1 ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] 2.3 roadmap (proposal)
Hola Francesc, Il 05/08/2011 08:42, Francesc Alted ha scritto: 2011/8/4, Antonio Valentino a_valent...@users.sourceforge.net: [CUT] * leave out, for the moment, advanced packaging #70 and #71 (unless anyone has already the job done). The debian directory sould be removed from the master branch IMHO. OK. And yes, debian directory should be removed (it is a responsibility of the debian maintainer only). Francesc some time ago I tried to contact the maintained of the debian package (Wen Heping if I'm not wrong) but without success. Do you know him? Since PyTables version currently in debian is outdated (2.1.2) my idea was to maintain a temporary debian branch (on PyTabes or on my personal area) to update the package an migrate it to dh7 and dh_python2. In this way it should be easier for debian developers to push 2.3 in the main archive when it will be ready. [CUT] * perform some testing with numpy 1.6.1 Would be great. Well, I performed a quick test (not --heavy) with numpy 1.6.1 on ubuntu 11.04 amd64 and all seems to work. * update docs for release Yes, specially ANNOUNCE.txt and RELEASE_NOTES.txt.in I remember I've seen somewhere a summary of additional features provided by PyTables Pro but I can't find it anymore. Do you have some pointer? * beta release Beta? Hmm, the PyTables sources are based on a previous stable PyTables Pro 2.2.1, and you only made slight changes, so I'd go directly with 2.3rc1 (but this is up to you). Ok for me, +1 for rc1 * full test on all platforms Yeah, at least with Windows and Linux (MacOSX is similar enough to Linux to not have to check it too frequently). Also, the purpose of a release candidate will allow people to check it before a final release. Do we also need to test various combination of python version and libraries? python 2.6/2.7 hdf5 1.8.x/1.6.10 (maybe only 1.8) numpy 1.4.x/1.5.x/1.6.x No more testing for numarry and Numeric since they are deprecated. OK, PyTables Pro was already well tested so maybe we don't have to worry too much in this case but it is useful to know for future releases. regards -- Antonio Valentino -- BlackBerryreg; DevCon Americas, Oct. 18-20, San Francisco, CA The must-attend event for mobile developers. Connect with experts. Get tools for creating Super Apps. See the latest technologies. Sessions, hands-on labs, demos much more. Register early save! http://p.sf.net/sfu/rim-blackberry-1 ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] 2.3 roadmap (proposal)
Hi Francesc, Il 05/08/2011 17:52, Francesc Alted ha scritto: 2011/8/5, Antonio Valentino a_valent...@users.sourceforge.net: [CUT] Since PyTables version currently in debian is outdated (2.1.2) my idea was to maintain a temporary debian branch (on PyTabes or on my personal area) to update the package an migrate it to dh7 and dh_python2. In this way it should be easier for debian developers to push 2.3 in the main archive when it will be ready. Another possibility is to keep the debian directory in the repo, but remove it from the source tarball (i.e. remove debian/ from MANIFEST.in). Wait... I've just check this and the debian directory is not included anymore. I suppose this should be fine, right? oh sorry, I've just pushed the change. I can revert it if you prefer. Anyway my idea was to point the wach file to github so to start packaging from a pure source distribution (no pre-built doc and no cythonized files) an make packaging easier. In this case having a debian directory around is not a good thing. thanks -- Antonio Valentino -- BlackBerryreg; DevCon Americas, Oct. 18-20, San Francisco, CA The must-attend event for mobile developers. Connect with experts. Get tools for creating Super Apps. See the latest technologies. Sessions, hands-on labs, demos much more. Register early save! http://p.sf.net/sfu/rim-blackberry-1 ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
[Pytables-users] 2.3 roadmap (proposal)
Hi list, hi developers, it seems to me that the PyTables development is a little stalled at the moment. I think that it is important to have PyTables 2.3 as soon as possible. Some job has already been done and new features coming from the PyTables Pro IMHO justify a new release. So my proposal is to: * merge the pro_liberation branch #1 * leave out, for the moment, advanced packaging #70 and #71 (unless anyone has already the job done). The debian directory sould be removed from the master branch IMHO. * proceed with deprecations #68 and #76 * merge all the work already done (#66, #77 TBC, ...) * perform some testing with numpy 1.6.1 * update docs for release * beta release * full test on all platforms * final release I think I have addressed issues #66, #68 and #77 in branches on my personal fork (https://github.com/avalentino/PyTables). It is only needed review and merge. I'm going to start working on #76. Job on #1 is almost complete. IMHO pro_liberation branch is ready for merge even if XSL stylesheets still need some cleanup (see #1). Unfortunately I don't know XSL enough to put my hands in. OK, any feedback is welcome and, most of all, I would appreciate a lot an OK from someone before starting to merge the various changes. [#1] https://github.com/PyTables/PyTables/issues/1 [#66] https://github.com/PyTables/PyTables/issues/66 [#68] https://github.com/PyTables/PyTables/issues/68 [#70] https://github.com/PyTables/PyTables/issues/70 [#71] https://github.com/PyTables/PyTables/issues/71 [#76] https://github.com/PyTables/PyTables/issues/76 [#77] https://github.com/PyTables/PyTables/issues/77 best regards -- Antonio Valentino -- BlackBerryreg; DevCon Americas, Oct. 18-20, San Francisco, CA The must-attend event for mobile developers. Connect with experts. Get tools for creating Super Apps. See the latest technologies. Sessions, hands-on labs, demos much more. Register early save! http://p.sf.net/sfu/rim-blackberry-1 ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
[Pytables-users] Fwd: Windows installation problems
Here it is a message that bounced (unsubscribed address, probably). Messaggio originale Oggetto: [Pytables-users] Windows installation problems Data: Tue, 26 Jul 2011 15:40:26 +0200 Mittente: PyTables Org pytab...@googlemail.com Rispondi-a: Discussion list for PyTables pytables-users@lists.sourceforge.net A: pytables-users pytables-users@lists.sourceforge.net Begin forwarded message: From: pytables-users-boun...@lists.sourceforge.net Date: July 26, 2011 3:18:45 PM GMT+02:00 To: pytables-users-ow...@lists.sourceforge.net Subject: Auto-discard notification The attached message has been automatically discarded. From: José María García Pérez josemaria.alk...@gmail.com Date: July 26, 2011 3:18:38 PM GMT+02:00 To: pytables-users@lists.sourceforge.net Subject: Windows installation problems Dear all, I have been using PyTables from time to time from long ago. I am trying to use it again, but this time I am not managing to make it work in Windows (I am using it in Linux). I used the binary installation: tables-2.2.1.win32-py2.7.exe I am using a laptop, 32bits with Windows XP, no admin rights and python 2.7. The installation finished without any complain. But then I get the following: Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win32 Type help, copyright, credits or license for more information. import tables Traceback (most recent call last): File stdin, line 1, in module File c:\Home\C08937\Python27\Lib\site-packages\tables\__init__.py, line 63, in module from tables.utilsExtension import getPyTablesVersion, getHDF5Version ImportError: DLL load failed: No se pudo iniciar la aplicaci≤n porque su configu raci≤n es incorrecta. Reinstalar la aplicaci≤n puede solucionar el problema. I have investigated as much as a I could. I tried Dependency Walker and Resource Hacker, but I get nowhere. I believe I have the required DLLs in the system and accesible in the path. When I open hd5dll.dll with Dependency Walker it complains about: Error: The Side-by-Side configuration information for \python27\lib\site-packages\tables\HDF5DLL.DLL contains errors. No se pudo iniciar la aplicacin porque su configuracin es incorrecta. Reinstalar la aplicacin puede solucionar el problema (14001). Error: The Side-by-Side configuration information for \python27\lib\site-packages\tables\SZIP.DLL contains errors. No se pudo iniciar la aplicacin porque su configuracin es incorrecta. Reinstalar la aplicacin puede solucionar el problema (14001). Any clue? Could anybody state a systematic approach to get it working? Cheers, José M. -- Magic Quadrant for Content-Aware Data Loss Prevention Research study explores the data loss prevention market. Includes in-depth analysis on the changes within the DLP market, and the criteria used to evaluate the strengths and weaknesses of these DLP solutions. http://www.accelacomm.com/jaw/sfnl/114/51385063/ ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Magic Quadrant for Content-Aware Data Loss Prevention Research study explores the data loss prevention market. Includes in-depth analysis on the changes within the DLP market, and the criteria used to evaluate the strengths and weaknesses of these DLP solutions. http://www.accelacomm.com/jaw/sfnl/114/51385063/___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
[Pytables-users] PyTables 2.3beta in EDP 7.1
Hi list, PyTables 2.3beta has been included in in EDP 7.1. http://www.enthought.com/products/changelog.php nice! ciao -- Antonio Valentino -- All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2 ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] garbage collecting HDF5 files?
Hi Ben, Il 05/06/2011 11:51, Ben Elliston ha scritto: Hi all. I have a large CArray that I needed to extend, so I created a new array the right size, and copied the data elements into the new array. I then used removeNode to remove the old array, but the HDF5 file has not shrunk. How do I vacuum up the deleted items? Thanks, Ben maybe the ptrepack utility may help http://www.pytables.org/docs/manual/ape.html#ptrepackDescr regards -- Antonio Valentino -- Simplify data backup and recovery for your virtual environment with vRanger. Installation's a snap, and flexible recovery options mean your data is safe, secure and there when you need it. Discover what all the cheering's about. Get your free trial download today. http://p.sf.net/sfu/quest-dev2dev2 ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] PyTables Pro has been liberated
Hi Josh, Il 03/06/2011 23:48, Josh Moore ha scritto: Hi all, On Jun 3, 2011, at 11:37 PM, Francesc Alted wrote: Hey Josh, 2011/6/3 Josh Moore josh.mo...@gmx.de [CUT] Instead, I created a PyTables organization: https://github.com/PyTables and pushed the trunk branch created via: git svn clone --authors authors.txt -s http://www.pytables.org/svn/pytables/PyTablesPro/ to: https://github.com/PyTables/PyTables Everyone who's expressed interest in maintenance duties should be added to the organization, and I'll turn over ownership to Francesc until we've decided how things move forward. I like your solution very much. Thanks for moving forward so fast :) I'm fine to have ownership for the time being, but I prefer if other person can take it over. Or, cannot we have several owners at the same time? Definitely. There can be any number of owners, and you need not be one of them! :) I tried to find the people who had mentioned wanting to take an active role on github and found these: • avalentino (Antonio Valentino) • bamford • scopatz (Anthony Scopatz) • wesm (Wes McKinney) Can you confirm that you are who I thought you were, not that some poor githubber is wondering what this pytables thing is. And if there are any other github users, please speak up. Yes, avalentino it's me. Thank you for your effort Cheers -- Antonio Valentino -- Simplify data backup and recovery for your virtual environment with vRanger. Installation's a snap, and flexible recovery options mean your data is safe, secure and there when you need it. Discover what all the cheering's about. Get your free trial download today. http://p.sf.net/sfu/quest-dev2dev2 ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Francesc Alted ceasing activities with PyTables
Hi list, Il 03/06/2011 15:12, Batuhan Osmanoglu ha scritto: Hi all, I would like to see Pytables continue, and I am glad to see that there are others interested. I think it would be a great idea to meet with everyone who is interested with Francesc's lead and we can discuss how the governance body functions. As Ben suggested, one way to do it may be to have a few people to revise and commit the bug-fixes and improvements from the community. And as Anthony pointed out meeting every few months is also a good idea I think. I just finished my Ph.D. and I don't really know how much time I will be able to spare, but I am using PyTables for storing and manipulating large geospatial datasets (think satellite images and numpy). So I have a natural interest in improving and maintaining PyTables in that respect. But there is a lot more that can be done with PyTables so I hope others can help with what they are interested in. I think it is a great idea to get together somehow with everyone involved, and see what can be done. Is the users list the right place to continue discussing this? I would like all users to be informed, but I don't want to upset anyone by sending lots of emails. ;) Take care, batu. Please guys, include me in the list of volunteers. I don't have too much time to dedicate to it (especially in this period) but I would like to contribute to keep PyTables alive. regards -- Antonio Valentino -- Simplify data backup and recovery for your virtual environment with vRanger. Installation's a snap, and flexible recovery options mean your data is safe, secure and there when you need it. Discover what all the cheering's about. Get your free trial download today. http://p.sf.net/sfu/quest-dev2dev2 ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] ANN: PyTables 2.2 (final) released
Hi Francesc, Il giorno Thu, 1 Jul 2010 20:39:03 +0200 Francesc Alted fal...@pytables.org ha scritto: Hi List, This is to announce the release of PyTables 2.2 final. I've finally declared Blosc stable and I'm very happy to say that Blosc is showing very good results on every benchmark that I'm doing with PyTables. I'd like to thank all of those that have collaborated on its intensive testing. Well, I hope you will like the new release too! Congratulations. Wonderful job, as usual. Looking at the changelog it seems to me that there is a good exchange of solutions between pytables and the h5py project. Which is the exact relationship between projects? There are plans for a closer collaboration? I wonder if there is somewhere a detailed features comparison between pytables and h5py. ciao -- Antonio Valentino -- This SF.net email is sponsored by Sprint What will you do first with EVO, the first 4G phone? Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
[Pytables-users] A little improvenet to the migration guide
Hi Francesc, hi Ivan, maybe it could be useful to add the following to the migration guide: == Migrating from PyTables 1.x to 2.x == Other changes = - ``CArray._v_chunksize`` -- ``CArray._v_chunkshape`` - ``EArray._v_chunksize`` -- ``EArray._v_chunkshape`` -- Antonio Valentino [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] PyTables 2.0a1a released
Alle 14:51, lunedì 19 febbraio 2007, Francesc Altet ha scritto: El dl 19 de 02 del 2007 a les 14:43 +0100, en/na Antonio Valentino va escriure: Alle 14:18, lunedì 19 febbraio 2007, Francesc Altet ha scritto: Hola Antonio, Yes. I've no tested alpha1 with no numarray and/or Numeric installed. Now, I've committed the patches (hopefully the correct ones) in trunk. Please, can you confirm that your tests runs well in your platform? If all goes well, I'll do a new micro-release later today. Ciao, OK, the test suite runs for rev:2401. There are some failures (see the attached file) but I don't know if they are due to the numpy version (1.0.1). Nope. These should be due to your Pyrex version. Please upgrade Pyrex to at least 0.9.5, *regenerate* all .c files and rerun again. Ciao, you are right :) python test_all.py --heavy 21 |tee ~/tables20_testlog.txt /home/valentino/projects/extern/pytables/tables/leaf.py:149: UserWarning: lzo compression library is not available. Using zlib instead!. %s compression library is not available. Using zlib instead!. %(complib)) [cut] Ran 2167 tests in 93.669s OK *Warning*: NumPy version is lower than recommended: 1.0.1 1.0.2 ***Python2.5 in 64-bit platform detected: disabling numarray/Numeric tests*** -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= PyTables version: 2.0a1 HDF5 version: 1.6.5 NumPy version: 1.0.1 Zlib version: 1.2.3 BZIP2 version: 1.0.3 (15-Feb-2005) Python version:2.5 (r25:51908, Jan 9 2007, 17:00:50) [GCC 4.1.2 20061115 (prerelease) (SUSE Linux)] Platform: linux2-x86_64 Byte-ordering: little -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Performing the complete test suite! -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Skipping Numeric test suite. Skipping numarray test suite. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= -- Antonio Valentino Consorzio Innova Contrada Chiatamura lotto 188 Zona Industriale La Martella 75100 Matera (MT) Italy - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] test failed on openSUSE
Il giorno Thu, 01 Feb 2007 16:35:22 +0100 Francesc Altet [EMAIL PROTECTED] ha scritto: Hola Antonio, El dj 01 de 02 del 2007 a les 16:19 +0100, en/na Antonio Valentino va escriure: Hi list, I'm trying to get PyTables working on a HP xw4400 workstation (Intel Core 2 Duo 2.66GHz) with openSUSE 10.2 (2.6.18.2-34-default #1 SMP x86_64), python 2.5. Running the test suite on PyTables from the svn/trunk I get a Segmentation fault (see the attached log). Any idea on how to solve the problem? Did you read the section Special Warning for Python 2.5 and 64-bit platforms users in the latests PyTables' announcements?: http://www.mail-archive.com/pytables-users@lists.sourceforge.net/msg00375.html ;) Uh, I'm sorry. I completely missed that warning. For more info about the issue, see the thread: http://comments.gmane.org/gmane.comp.python.numeric.general/9441 Using Python2.4 should solve the problems until PyTables 2.0 would appear (very soon now). Do you mean that a *stable version* will be released soon? I can't find PyTables 2.0 on the svn repository. :( [BTW, if you want to successfully post to the PyTables list, please, be sure that you send the message from a *subscribed* address.] HTH, OK -- Antonio Valentino [EMAIL PROTECTED] - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier. Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
