Hi everyone, I've developed a parser that read csv files and save them on a hdf5 file (on python using pytables). The initial problem it was that pytables doesn't support simultaneous write (on multiple threads/processes) so I've decided to create multiple processes so each one of them read a whole csv file on a array and then lock the hdf5 object, write the table and release it.
The problem with that it's that writing the array to the table is quite slow. There is any option to speed up this ? Do more pre-processing on the threads (like an special kind of array) so when they keep locked the hdf5 file less time and avoid/reduce the bottleneck ? pytables pro will improve this situation ? Regards, -- Felipe Barriga Richards Algometrics SpA ------------------------------------------------------------------------------ The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE: Pinpoint memory and threading errors before they happen. Find and fix more than 250 security defects in the development cycle. Locate bottlenecks in serial and parallel code that limit performance. http://p.sf.net/sfu/intel-dev2devfeb _______________________________________________ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
