Anthony:
No thanks, think there is a bug, and so I was offering up code and files for
testing. The files are an order of magnitude larger than I thought. The base
netCDF4 files, compressed, are 1.4 GB. I dont have time to pare down the files
and see if that is the problem - my guess is, since I have played with this
conversion before on smaller files is that it does not.
I am uploading 2 files and the code to dropbox (I have a big space up there) if
anyone wants to try this on their machine. Let me know and I will send a link
to the files, but keep in mind the tar file is ~ 3 GB. The code only take
I can work around this by simply "hand" processing the files one by one. But
something is weird here, so I feel there is a bug to squash. Since the files
are so large, perhaps that is the issue. The write times for the first file
are 3x faster than the second, because (obviously) 3x more data is being
written. But the processing on my Mac pro only takes about 45 sec for the
first file, and 2 min for the second, if that much.
Thanks for any consideration!
Lou Wicker
On Jun 24, 2011, at 5:11 PM, Anthony Scopatz wrote:
>
>
> On Fri, Jun 24, 2011 at 2:14 PM, Dr. Louis Wicker <louis.wic...@noaa.gov>
> wrote:
> Thanks. Except that when I only do one data set, it works. E.g,
>
> If I start on a different file - it then compresses. Let me check, and if
> someone (not you, I guess) wants, I can tar 2 files up and the code and put
> it on dropbox.
>
> These are big files, the uncompressed files are 0.5Gb
>
> Do you just need someone to run your original script on these files?
>
> Be Well
> Anthony
>
>
> L
>
> On Jun 24, 2011, at 12:22 PM, Francesc Alted wrote:
>
> > A Friday 24 June 2011 18:31:28 Dr. Louis Wicker escrigué:
> >> sorry - that was test code to see that the uncompressed files were
> >> the same size as what I was seeng. I have tried complevel 1-5, and
> >> get the same result.
> >
> > It is possible that Blosc cannot compress your datasets, and as
> > consequence, it leaves them uncompressed. You may want to try with
> > level 9 just to see if Blosc can actually compress them.
> >
> > --
> > Francesc Alted
> >
> > ------------------------------------------------------------------------------
> > All the data continuously generated in your IT infrastructure contains a
> > definitive record of customers, application performance, security
> > threats, fraudulent activity and more. Splunk takes this data and makes
> > sense of it. Business sense. IT sense. Common sense..
> > http://p.sf.net/sfu/splunk-d2d-c1
> > _______________________________________________
> > Pytables-users mailing list
> > Pytables-users@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/pytables-users
>
> ----------------------------------------------------------------------------
> | Dr. Louis J. Wicker
> | NSSL/FRDD Rm 4368
> | National Weather Center
> | 120 David L. Boren Boulevard, Norman, OK 73072
> |
> | E-mail: louis.wic...@noaa.gov
> | HTTP: www.nssl.noaa.gov/~lwicker
> | Phone: (405) 325-6340
> | Fax: (405) 325-6780
> |
> | If you start a day without a clear plan about how you’re
> | going to spend your attention, you’ll end up wasting most
> | of it.—Martha Beck (2002)
> |
> | Assemby of Japanese bicycle requires great peace of mind.
> | Robert Pirsig, ZAMM
> |
> ----------------------------------------------------------------------------
> |
> | "The contents of this message are mine personally and
> | do not reflect any position of the Government or NOAA."
> |
> ----------------------------------------------------------------------------
>
>
>
>
>
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> All the data continuously generated in your IT infrastructure contains a
> definitive record of customers, application performance, security
> threats, fraudulent activity and more. Splunk takes this data and makes
> sense of it. Business sense. IT sense. Common sense..
> http://p.sf.net/sfu/splunk-d2d-c1
> _______________________________________________
> Pytables-users mailing list
> Pytables-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
> ------------------------------------------------------------------------------
> All the data continuously generated in your IT infrastructure contains a
> definitive record of customers, application performance, security
> threats, fraudulent activity and more. Splunk takes this data and makes
> sense of it. Business sense. IT sense. Common sense..
> http://p.sf.net/sfu/splunk-d2d-c1_______________________________________________
> Pytables-users mailing list
> Pytables-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/pytables-users
----------------------------------------------------------------------------
| Dr. Louis J. Wicker
| NSSL/WRDD Rm 4366
| National Weather Center
| 120 David L. Boren Boulevard, Norman, OK 73072
|
| E-mail: louis.wic...@noaa.gov
| HTTP: www.nssl.noaa.gov/~lwicker
| Phone: (405) 325-6340
| Fax: (405) 325-6780
|
| If you want to do the impossible, don't hire an expert
| because he knows | it can't be done.
|
| -- Henry Ford
|
| NSSL VORTEX2 Page:
| http://www.nssl.noaa.gov/projects/vortex2/
----------------------------------------------------------------------------
|
| "The contents of this message are mine personally and
| do not reflect any position of the Government or NOAA."
|
----------------------------------------------------------------------------
------------------------------------------------------------------------------
All the data continuously generated in your IT infrastructure contains a
definitive record of customers, application performance, security
threats, fraudulent activity and more. Splunk takes this data and makes
sense of it. Business sense. IT sense. Common sense..
http://p.sf.net/sfu/splunk-d2d-c1
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
