Re: [Pytables-users] PyTables data files for a tutorial

2012-10-21 Thread Jason Moore 
This is a PyTables generated file with data collected from vehicle
(bicycle) dynamics measurements. Meta data are in tables and time series
are stored in array objects.

http://mae.ucdavis.edu/~biosport/InstrumentedBicycleData/InstrumentedBicycleData.h5.bz2

It is about 308 mb compressed and 610 mb uncompressed.

Jason

On Sun, Oct 21, 2012 at 1:01 PM, Andy Wilson wrote:

> On Sun, Oct 21, 2012 at 10:41 AM, Francesc Alted 
> wrote:
>
> > Hi,
> >
> > I'm going to give a tutorial on PyTables next Thursday during the PyData
> > conference in New York (http://nyc2012.pydata.org/) and I'd like to use
> > some real life data files.  So, if you have some public repository with
> > data generated with PyTables, please tell me.  I'm looking for files
> > that are not very large (< 1GB), and that use the Table object
> > significantly.  A small description of the data included will be more
> > that welcome too!
> >
> > Thanks!
> >
> > --
> > Francesc Alted
>
>
>
> Hi Francesc.
>
> I've been working on a library for accessing climatology data that
> uses pytables to cache data from the USGS. It could easily be used to
> create a sample dataset for some area of interest. File size is
> determined by how much data gets queried.
>
>
> The general layout is:
>
> /usgs/sites
> - the sites table contains information and metadata about a site
>
>
> /usgs/values///
> - a table containing all the timeseries data for each site and
> parameter is created as data are queried
> - parameter codes are a bit obscure but a dict with descriptive
> metadata stashed at table.attrs.variable
> - the datetime column has a CSIndex on it and stored as as a string
> because some sites have data prior to the year 1901
> - pretty inefficient in terms of disk space (lots of large-ish string
> columns) because it handles a very general class of data types
>
>
> Here's what the code would look like to download and create the hdf5
> file for 10 random sites in New York:
>
> import ulmo
>
> # the default location for the hdf5 file is OS dependent, so provide
> the path you want to use
> hdf5_file_path = './usgs_data.h5'
>
> # get list of sites in NY
> ulmo.usgs.pytables.update_site_list(state_code='NY', path=hdf5_file_path)
> sites = ulmo.usgs.pytables.get_sites(path=hdf5_file_path)
>
> # download data for a few random sites
> for site in sites.keys()[:10]:
> ulmo.usgs.pytables.update_site_data(site, path=hdf5_file_path)
>
>
>
> The project is on github: https://github.com/swtools/ulmo
> and the code that does all the pytables stuff (including the table
> descriptions) is here:
> https://github.com/swtools/ulmo/blob/master/ulmo/usgs/pytables.py
>
> -andy
>
>
> --
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_sfd2d_oct
> ___
> Pytables-users mailing list
> Pytables-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>



-- 
Jason K. Moore, Ph.D.
Personal Website 
Sports Biomechanics Lab , UC Davis
Davis Open Science 
Google Voice: +01 530-601-9791
--
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users

Re: [Pytables-users] PyTables data files for a tutorial

2012-10-21 Thread Andy Wilson 
On Sun, Oct 21, 2012 at 10:41 AM, Francesc Alted  wrote:

> Hi,
>
> I'm going to give a tutorial on PyTables next Thursday during the PyData
> conference in New York (http://nyc2012.pydata.org/) and I'd like to use
> some real life data files.  So, if you have some public repository with
> data generated with PyTables, please tell me.  I'm looking for files
> that are not very large (< 1GB), and that use the Table object
> significantly.  A small description of the data included will be more
> that welcome too!
>
> Thanks!
>
> --
> Francesc Alted



Hi Francesc.

I've been working on a library for accessing climatology data that
uses pytables to cache data from the USGS. It could easily be used to
create a sample dataset for some area of interest. File size is
determined by how much data gets queried.


The general layout is:

/usgs/sites
- the sites table contains information and metadata about a site


/usgs/values///
- a table containing all the timeseries data for each site and
parameter is created as data are queried
- parameter codes are a bit obscure but a dict with descriptive
metadata stashed at table.attrs.variable
- the datetime column has a CSIndex on it and stored as as a string
because some sites have data prior to the year 1901
- pretty inefficient in terms of disk space (lots of large-ish string
columns) because it handles a very general class of data types


Here's what the code would look like to download and create the hdf5
file for 10 random sites in New York:

import ulmo

# the default location for the hdf5 file is OS dependent, so provide
the path you want to use
hdf5_file_path = './usgs_data.h5'

# get list of sites in NY
ulmo.usgs.pytables.update_site_list(state_code='NY', path=hdf5_file_path)
sites = ulmo.usgs.pytables.get_sites(path=hdf5_file_path)

# download data for a few random sites
for site in sites.keys()[:10]:
ulmo.usgs.pytables.update_site_data(site, path=hdf5_file_path)



The project is on github: https://github.com/swtools/ulmo
and the code that does all the pytables stuff (including the table
descriptions) is here:
https://github.com/swtools/ulmo/blob/master/ulmo/usgs/pytables.py

-andy

--
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users

Re: [Pytables-users] 10 years of PyTables

2012-10-21 Thread Antonio Valentino 
Il 21/10/2012 17:26, Francesc Alted ha scritto:
> Hi!,
>
> This month PyTables celebrates the 10th anniversary of its first public
> release:
>
> http://osdir.com/ml/python.scientific.user/2002-10/msg00043.html
>

Happy birthday PyTables!!! :))


ciao

-- 
Antonio Valentino

--
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users

Re: [Pytables-users] 10 years of PyTables

2012-10-21 Thread Anthony Scopatz 
Congrats Francesc!

It is a testament to how useful PyTables is that it is still around and
going strong!

Personally, I know that your fast, polite, and in-depth responses on the
mailing list have made PyTables the great resources that it is.
 Additionally,
it has served as a model to me for how open source projects *should* be run!

I'd also really like to thank Antonio for driving new features into the
code base!

If only we were all on the same continent, we could have a PyTables birthday
party or something...

Be Well
Anthony

On Sun, Oct 21, 2012 at 10:26 AM, Francesc Alted wrote:

> Hi!,
>
> This month PyTables celebrates the 10th anniversary of its first public
> release:
>
> http://osdir.com/ml/python.scientific.user/2002-10/msg00043.html
>
> There one can read that very new features of Python like generators and
> metaclasses were leveraged.  Even that a nascent Pyrex (the antecessor
> of Cython) was used for the extensions.  Oh, what memories!
>
> The original text below:
>
> -
> Hi!,
>
> PyTables is a Python package which allows dealing with HDF5 tables. Such
> a table is defined as a collection of records whose values are stored in
> fixed-length fields. PyTables is intended to be easy-to-use, and tries to
> be a high-performance interface to HDF5. To achieve this, the newest
> improvements introduced in Python 2.2 (like generators or slots and
> metaclasses in new-brand classes) has been used. Pyrex creation extension
> tool has been chosen to access the HDF5 library.
>
> This package should be platform independent, but until now I've tested it
> only with Linux. It's the first public release (v 0.1), and it is in
> alpha state.
>
> You can get it from:
>
> http://sourceforge.net/projects/pytables/
>
> There is still not a project home page. Perhaps in next days.
>
> Feedback welcome.!
>
> --
> Francesc Alted PGP KeyID: 0x61C8C11F
> Scientific aplications developer
> Public PGP key available:http://www.openlc.org/falted_at_openlc.asc
> Key fingerprint = 1518 38FE 3A3D 8BE8 24A0 3E5B 1328 32CC 61C8 C11F
>
>
> --
> Francesc Alted
>
>
>
> --
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_sfd2d_oct
> ___
> Pytables-users mailing list
> Pytables-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
--
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users

Re: [Pytables-users] PyTables data files for a tutorial

2012-10-21 Thread Anthony Scopatz 
Hello Francesc,

I look forward to your pydata hearing how your tutorial goes!

Here [1] is a file that stores some basic nuclear data that is freely
redistributable.  It stores atomic weights, bound neutron scattering
lengths, and pre-compiled neutron cross sections (xs) for 5 different
energy regimes.  Everything in here is a table.  The file is rather
(at about 165 kb).  There are integer, float, and complex columns.

I hope that this helps!

Be Well
Anthony

1. https://s3.amazonaws.com/pyne/prebuilt_nuc_data.h5

On Sun, Oct 21, 2012 at 10:41 AM, Francesc Alted wrote:

> Hi,
>
> I'm going to give a tutorial on PyTables next Thursday during the PyData
> conference in New York (http://nyc2012.pydata.org/) and I'd like to use
> some real life data files.  So, if you have some public repository with
> data generated with PyTables, please tell me.  I'm looking for files
> that are not very large (< 1GB), and that use the Table object
> significantly.  A small description of the data included will be more
> that welcome too!
>
> Thanks!
>
> --
> Francesc Alted
>
>
>
> --
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_sfd2d_oct
> ___
> Pytables-users mailing list
> Pytables-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
--
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users

[Pytables-users] PyTables data files for a tutorial

2012-10-21 Thread Francesc Alted 
Hi,

I'm going to give a tutorial on PyTables next Thursday during the PyData 
conference in New York (http://nyc2012.pydata.org/) and I'd like to use 
some real life data files.  So, if you have some public repository with 
data generated with PyTables, please tell me.  I'm looking for files 
that are not very large (< 1GB), and that use the Table object 
significantly.  A small description of the data included will be more 
that welcome too!

Thanks!

-- 
Francesc Alted


--
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users

[Pytables-users] 10 years of PyTables

2012-10-21 Thread Francesc Alted 
Hi!,

This month PyTables celebrates the 10th anniversary of its first public 
release:

http://osdir.com/ml/python.scientific.user/2002-10/msg00043.html

There one can read that very new features of Python like generators and 
metaclasses were leveraged.  Even that a nascent Pyrex (the antecessor 
of Cython) was used for the extensions.  Oh, what memories!

The original text below:

-
Hi!,

PyTables is a Python package which allows dealing with HDF5 tables. Such
a table is defined as a collection of records whose values are stored in
fixed-length fields. PyTables is intended to be easy-to-use, and tries to
be a high-performance interface to HDF5. To achieve this, the newest
improvements introduced in Python 2.2 (like generators or slots and
metaclasses in new-brand classes) has been used. Pyrex creation extension
tool has been chosen to access the HDF5 library.

This package should be platform independent, but until now I've tested it
only with Linux. It's the first public release (v 0.1), and it is in
alpha state.

You can get it from:

http://sourceforge.net/projects/pytables/

There is still not a project home page. Perhaps in next days.

Feedback welcome.!

--
Francesc Alted PGP KeyID: 0x61C8C11F
Scientific aplications developer
Public PGP key available:http://www.openlc.org/falted_at_openlc.asc
Key fingerprint = 1518 38FE 3A3D 8BE8 24A0 3E5B 1328 32CC 61C8 C11F


-- 
Francesc Alted


--
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users