Re: [Pytables-users] multiprocessing and pytables

2012-10-16 Thread Antonio Valentino 
Hi Anthony,

Il giorno 16/ott/2012, alle ore 02:04, Anthony Scopatz scop...@gmail.com ha 
scritto:

 Hello Ernesto, 
 
 So you are actually asking two different questions, one on reading and the 
 other on writing.  In general reading, or querying, with multiprocessing 
 works very well.  Writing to a single file with multiple processes is 
 destined to failure though.  So the strategy that many people have adopted is 
 to have multiple processes create the data and then have a master process 
 which acts as a queue for writing out the data.  Please see the example here 
 for more inspiration [1].  Note that we have been having problems recently 
 with multiprocess writing out to multiple files, but that is not what you 
 want to do. 
 
 Be Well
 Anthony
 
 1. 
 https://github.com/PyTables/PyTables/blob/develop/examples/multiprocess_access_queues.py

It seems that the topic PyTables + multiprocessing became very popular since 
some time.

Probably we should add a FAQ entry and provide a more extended tutorial based 
on the example provided by Josh.


cheers

--
Antonio Valentino

--
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users

Re: [Pytables-users] multiprocessing and pytables

2012-10-15 Thread Anthony Scopatz 
Hello Ernesto,

So you are actually asking two different questions, one on reading and the
other on writing.  In general reading, or querying,
with multiprocessing works very well.  Writing to a single file with
multiple processes is destined to failure though.  So the strategy that
many people have adopted is to have multiple processes create the data and
then have a master process which acts as a queue for writing out the data.
 Please see the example here for more inspiration [1].  Note that we have
been having problems recently with multiprocess writing out to multiple
files, but that is not what you want to do.

Be Well
Anthony

1.
https://github.com/PyTables/PyTables/blob/develop/examples/multiprocess_access_queues.py

On Mon, Oct 15, 2012 at 11:45 AM, Ernesto Picardi e.pica...@unical.itwrote:

 Dear all,

 I have a hdf5 file including several tables. To speed up the creation of
 all tables, could I create each individual table by independent processes
 launched by multiprocessing module? Could I employ independent processes to
 query diverse tables of the same hdf5 file?

 Thank you very much in advance for whatever answer.

 Regards,

 Ernesto







  Riservatezza / Confidentiality 
 In ottemperanza al D.Lgs. n. 196 del 30/6/2003 in materia di protezione
 dei dati personali, le informazioni contenute in questo messaggio sono
 strettamente riservate ed esclusivamente indirizzate al destinatario
 indicato (oppure alla persona responsabile di  rimetterlo al destinatario).
 Vogliate tener presente che qualsiasi uso, riproduzione o divulgazione di
 questo messaggio e' vietato. Nel caso in cui aveste ricevuto questo
 messaggio per errore, vogliate cortesemente avvertire il mittente e
 distruggere il presente  messaggio.



 --
 Don't let slow site performance ruin your business. Deploy New Relic APM
 Deploy New Relic app performance management and know exactly
 what is happening inside your Ruby, Python, PHP, Java, and .NET app
 Try New Relic at no cost today and get our sweet Data Nerd shirt too!
 http://p.sf.net/sfu/newrelic-dev2dev
 ___
 Pytables-users mailing list
 Pytables-users@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/pytables-users

--
Don't let slow site performance ruin your business. Deploy New Relic APM
Deploy New Relic app performance management and know exactly
what is happening inside your Ruby, Python, PHP, Java, and .NET app
Try New Relic at no cost today and get our sweet Data Nerd shirt too!
http://p.sf.net/sfu/newrelic-dev2dev___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users