Hello Anthony, With the above being said and with more work put on the initial attempt, could you suggest other methods that might improve write performance? The optimization page I feel mostly talks about data retrieval, which will be more important later on, but I have to make a writing bound before then. The only thing that I have done to improve performance is to use table.append(rows).
My updated code is attached to this email, thanks again! Thanks, Jacob On Tue, Jun 26, 2012 at 12:56 AM, Anthony Scopatz <scop...@gmail.com> wrote: > Hello Jacob, > > This is not surprising. The HDF5 parallel library requires MPI and comes > with some special restrictions (no compression on write). As such, the > pain of implementing a parallel write version of PyTables has not been > worth it. We certainly welcome pull requests and further discussion on > this issue ;). Often times it is easier (and faster...writing is > expensive) to do the computation in parallel followed by a single write. > Or you could have a dedicated thread which queues and executes write > commands as they come in. Just some thoughts on how to avoid this > problem. > > Parallel reads are supported. > > Let me know if you have further questions or really want to dive into this > issue. > > Be Well > Anthony > > On Mon, Jun 25, 2012 at 1:33 PM, Jacob Bennett > <jacob.bennet...@gmail.com>wrote: > >> Hello PyTables Users, >> >> I am very new to pytables and if you all could help me out, that would be >> splendid. >> >> I'm currently having trouble with writing to two separate HDF5 files >> using pytables. Each file itself is only accessible by a single thread, so >> there really shouldn't be any threading issues. When I run my python script >> however, it just seems to crash at random time intervals without any error >> messages received or exceptions thrown. >> >> I write data to the HDF5 files as follows. I have two HDF5 files that >> represent book snapshots and trade snapshots. The data of these snapshots >> come in the form of python dictionaries whose values are the data itself in >> an array. Two threads run on each file. One thread controls when to create >> new files and close others based upon the time of day while the other >> thread iterates over each key value pair in the dictionary and loads data >> to the file. When a thread has access to the file, the file is locked. >> >> I have my two datawrappers attached to the email. Please take a look at >> them. One thread runs acceptDict in a loop while the other runs changeFile >> in a loop. This is really frustrating when I don't get any errors and >> python just crashes unexpectedly. >> >> Thanks, >> Jacob >> >> -- >> Jacob Bennett >> Massachusetts Institute of Technology >> Department of Electrical Engineering and Computer Science >> Class of 2014| benne...@mit.edu >> >> >> >> ------------------------------------------------------------------------------ >> Live Security Virtual Conference >> Exclusive live event will cover all the ways today's security and >> threat landscape has changed and how IT managers can respond. Discussions >> will include endpoint security, mobile security and the latest in malware >> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >> _______________________________________________ >> Pytables-users mailing list >> Pytables-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/pytables-users >> >> > > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > Pytables-users mailing list > Pytables-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/pytables-users > > -- Jacob Bennett Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science Class of 2014| benne...@mit.edu
BookDataWrapper.py
Description: Binary data
------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
