Hi Francesc, The challenge I face is getting it off my machine - I've ordered an encrypted USB stick that I'll post to you in due course.
Currently, it appears that memory is consumed in the numpy loadtxt code (whilst loading the csv) and not released until I do a file close() PyTables statement. See the task manage screen grabs enclosed. I haven't been able to find any memory leaks yet in the application code but I haven't been able to reproduce the error in a standalone script yet. I've also managed to corrupt another file - I had to end process with the file open. I can't even reopen that file. I'll post more when I've a better sense of where the problems lie. Thanks David -----Original Message----- From: Francesc Alted [mailto:fal...@pytables.org] Sent: Thu 9-Dec-2010 08:01 To: pytables-users@lists.sourceforge.net Subject: Re: [Pytables-users] MemoryError on reading attributes A Wednesday 08 December 2010 19:07:24 david.bri...@ubs.com escrigué: > Hi Francesc > > It's 1.2GByte so I'll have to burn a DVD and post it to you - first I > need to find out how to do that. Well, 1.2 GB is not that much for nowadays ADSL lines, so if you can upload it to an accessible place through Internet I can download it without much problem. > I had a MemoryError loading the > data in the first place. Is there a way to check that the file > format is good? I don't know what you mean exactly. If the file has been created with PyTables, then the format should be good. > Basically the current file has about 1200 tables (float64, float32, > float32, float32, float32, int64) with 500k rows. I see. Well, another option would be to modify your current script so that it can generate the tables using synthetic (faked) data. That way you won't need to send the original file. Cheers, -- Francesc Alted ------------------------------------------------------------------------------ This SF Dev2Dev email is sponsored by: WikiLeaks The End of the Free Internet http://p.sf.net/sfu/therealnews-com _______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users Visit our website at http://www.ubs.com This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mails are not encrypted and cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. This message is provided for informational purposes and should not be construed as a solicitation or offer to buy or sell any securities or related financial instruments. UBS Limited is a company limited by shares incorporated in the United Kingdom registered in England and Wales with number 2035362. Registered office: 1 Finsbury Avenue, London EC2M 2PP. UBS Limited is authorised and regulated by the Financial Services Authority. UBS AG is a public company incorporated with limited liability in Switzerland domiciled in the Canton of Basel-City and the Canton of Zurich respectively registered at the Commercial Registry offices in those Cantons with Identification No: CH-270.3.004.646-4 and having respective head offices at Aeschenvorstadt 1, 4051 Basel and Bahnhofstrasse 45, 8001 Zurich, Switzerland. Registered in the United Kingdom as a foreign company with No: FC021146 and having a UK Establishment registered at Companies House, Cardiff, with No: BR 004507. The principal office of UK Establishment: 1 Finsbury Avenue, London EC2M 2PP. In the United Kingdom, UBS AG is authorised and regulated by the Financial Services Authority. UBS reserves the right to retain all messages. Messages are protected and accessed only in legally justified cases. ------------------------------------------------------------------------------ Oracle to DB2 Conversion Guide: Learn learn about native support for PL/SQL, new data types, scalar functions, improved concurrency, built-in packages, OCI, SQL*Plus, data movement tools, best practices and more. http://p.sf.net/sfu/oracle-sfdev2dev _______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
