Re: [Pytables-users] Compression and Indexing in PyTables
More input from Jeff Reback. Hey Jeff, I see that you try to post here from time to time, but your messages bounce because the address that you use as sender is not subscribed. Please make sure that you post from a subscribed address. Thanks! Francesc On 7/28/13 11:35 AM, pytables-users-boun...@lists.sourceforge.net wrote: The attached message has been automatically discarded. Re: [Pytables-users] Compression and Indexing in PyTables.eml Subject: Re: [Pytables-users] Compression and Indexing in PyTables From: Jeff Reback jreb...@yahoo.com Date: 7/28/13 11:35 AM To: Discussion list for PyTables pytables-users@lists.sourceforge.net pandas stores using Pytables and embeds extra meta data in the attributes to enable deserialization to the original pandas structure On Jul 28, 2013, at 11:23 AM, Francesc Altedfal...@gmail.com wrote: On 7/28/13 10:21 AM, David Reed wrote: maybe I wasn't aware of this, but has PANDAS completely wrapped PyTables, or is PyTables something I should still be using for storing and accessing scientific data, and PANDAS has an access point to it? Yeah, more the later than the former. PyTables is an standalone library, but Pandas uses it as another storage backend. -- Francesc Alted -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Francesc Alted -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
[Pytables-users] Tables vs Arrays
I'm really trying to become more productive using PyTables, but am struggling with what I should be using. Whats the difference between a table and an array? -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Tables vs Arrays
On Sun, Jul 28, 2013 at 8:38 PM, David Reed david.ree...@gmail.com wrote: I'm really trying to become more productive using PyTables, but am struggling with what I should be using. Whats the difference between a table and an array? Hi David, The difference between Arrays and Tables, conceptually is the same as the different between numpy arrays and numpy structured arrays. The plain old [Aa]rray is a continuous block of a single data type. Tables and structured arrays have a more complex data type that is composed of a continuous sequence of other data types (ie the fields / columns). Which data structure you use really depends a lot of the type of problem you are trying to solve and what kinds of questions you want to answer with that data structure. That said, the implementation of Tables is far more similar to EArrays than Arrays. So a lot of the performance trade offs that you see are similar. You should watch my HDF5 is for Lovers talk for more generic advice [1]. I hope this helps! Be Well Anthony 1. http://www.youtube.com/watch?v=Nzx0HAd3FiI -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Tables vs Arrays
On 7/28/13 9:58 PM, Anthony Scopatz wrote: On Sun, Jul 28, 2013 at 8:38 PM, David Reed david.ree...@gmail.com mailto:david.ree...@gmail.com wrote: I'm really trying to become more productive using PyTables, but am struggling with what I should be using. Whats the difference between a table and an array? Hi David, The difference between Arrays and Tables, conceptually is the same as the different between numpy arrays and numpy structured arrays. The plain old [Aa]rray is a continuous block of a single data type. Tables and structured arrays have a more complex data type that is composed of a continuous sequence of other data types (ie the fields / columns). Which data structure you use really depends a lot of the type of problem you are trying to solve and what kinds of questions you want to answer with that data structure. That said, the implementation of Tables is far more similar to EArrays than Arrays. So a lot of the performance trade offs that you see are similar. Besides this, another interesting difference is that Tables allow queries to be performed in a similar way to relational databases (but using a more NumPy-esque syntax). Here it is some examples: http://pytables.github.io/cookbook/hints_for_sql_users.html?highlight=query#selecting-data and you can index columns too: http://pytables.github.io/cookbook/hints_for_sql_users.html?highlight=query#creating-an-index so that you can accelerate queries involving indexed columns. -- Francesc Alted -- See everything from the browser to the database with AppDynamics Get end-to-end visibility with application monitoring from AppDynamics Isolate bottlenecks and diagnose root cause in seconds. Start your free trial of AppDynamics Pro today! http://pubads.g.doubleclick.net/gampad/clk?id=48808831iu=/4140/ostg.clktrk ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
