Hi Jacob,
On Tue, Jun 26, 2012 at 5:35 PM, Jacob Bennett <jacob.bennet...@gmail.com>wrote:
> Hello Anthony,
>
> With the above being said and with more work put on the initial attempt,
> could you suggest other methods that might improve write performance? The
> optimization page I feel mostly talks about data retrieval, which will be
> more important later on, but I have to make a writing bound before then.
> The only thing that I have done to improve performance is to use
> table.append(rows).
>
Typically also having larger chunksizes will increase performance.
Additionally, adding compression via filters may also increase write
speeds. However, the exact strategy you take will depend on the size of
data that you are writing and the number of processors that you have. Out
of curiosity, what are your data and nproc sizes?
> My updated code is attached to this email, thanks again!
>
I tried running this but there was no TimeHandler module.... However, I
will note that this doesn't look like the most efficient code with all of
try-except blocks. I think that hasattr() will work in a lot of these
cases for you.
Be Well
Anthony
>
> Thanks,
> Jacob
>
>
> On Tue, Jun 26, 2012 at 12:56 AM, Anthony Scopatz <scop...@gmail.com>wrote:
>
>> Hello Jacob,
>>
>> This is not surprising. The HDF5 parallel library requires MPI and comes
>> with some special restrictions (no compression on write). As such, the
>> pain of implementing a parallel write version of PyTables has not been
>> worth it. We certainly welcome pull requests and further discussion on
>> this issue ;). Often times it is easier (and faster...writing is
>> expensive) to do the computation in parallel followed by a single write.
>> Or you could have a dedicated thread which queues and executes write
>> commands as they come in. Just some thoughts on how to avoid this
>> problem.
>>
>> Parallel reads are supported.
>>
>> Let me know if you have further questions or really want to dive into
>> this issue.
>>
>> Be Well
>> Anthony
>>
>> On Mon, Jun 25, 2012 at 1:33 PM, Jacob Bennett <
>> jacob.bennet...@gmail.com> wrote:
>>
>>> Hello PyTables Users,
>>>
>>> I am very new to pytables and if you all could help me out, that would
>>> be splendid.
>>>
>>> I'm currently having trouble with writing to two separate HDF5 files
>>> using pytables. Each file itself is only accessible by a single thread, so
>>> there really shouldn't be any threading issues. When I run my python script
>>> however, it just seems to crash at random time intervals without any error
>>> messages received or exceptions thrown.
>>>
>>> I write data to the HDF5 files as follows. I have two HDF5 files that
>>> represent book snapshots and trade snapshots. The data of these snapshots
>>> come in the form of python dictionaries whose values are the data itself in
>>> an array. Two threads run on each file. One thread controls when to create
>>> new files and close others based upon the time of day while the other
>>> thread iterates over each key value pair in the dictionary and loads data
>>> to the file. When a thread has access to the file, the file is locked.
>>>
>>> I have my two datawrappers attached to the email. Please take a look at
>>> them. One thread runs acceptDict in a loop while the other runs changeFile
>>> in a loop. This is really frustrating when I don't get any errors and
>>> python just crashes unexpectedly.
>>>
>>> Thanks,
>>> Jacob
>>>
>>> --
>>> Jacob Bennett
>>> Massachusetts Institute of Technology
>>> Department of Electrical Engineering and Computer Science
>>> Class of 2014| benne...@mit.edu
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and
>>> threat landscape has changed and how IT managers can respond. Discussions
>>> will include endpoint security, mobile security and the latest in malware
>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> _______________________________________________
>>> Pytables-users mailing list
>>> Pytables-users@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Pytables-users mailing list
>> Pytables-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>
>>
>
>
> --
> Jacob Bennett
> Massachusetts Institute of Technology
> Department of Electrical Engineering and Computer Science
> Class of 2014| benne...@mit.edu
>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pytables-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
