Re: [Pytables-users] multiprocessing and pytables
Hi Anthony, Il giorno 16/ott/2012, alle ore 02:04, Anthony Scopatz scop...@gmail.com ha scritto: Hello Ernesto, So you are actually asking two different questions, one on reading and the other on writing. In general reading, or querying, with multiprocessing works very well. Writing to a single file with multiple processes is destined to failure though. So the strategy that many people have adopted is to have multiple processes create the data and then have a master process which acts as a queue for writing out the data. Please see the example here for more inspiration [1]. Note that we have been having problems recently with multiprocess writing out to multiple files, but that is not what you want to do. Be Well Anthony 1. https://github.com/PyTables/PyTables/blob/develop/examples/multiprocess_access_queues.py It seems that the topic PyTables + multiprocessing became very popular since some time. Probably we should add a FAQ entry and provide a more extended tutorial based on the example provided by Josh. cheers -- Antonio Valentino -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
[Pytables-users] multiprocessing and pytables
Dear all, I have a hdf5 file including several tables. To speed up the creation of all tables, could I create each individual table by independent processes launched by multiprocessing module? Could I employ independent processes to query diverse tables of the same hdf5 file? Thank you very much in advance for whatever answer. Regards, Ernesto Riservatezza / Confidentiality In ottemperanza al D.Lgs. n. 196 del 30/6/2003 in materia di protezione dei dati personali, le informazioni contenute in questo messaggio sono strettamente riservate ed esclusivamente indirizzate al destinatario indicato (oppure alla persona responsabile di rimetterlo al destinatario). Vogliate tener presente che qualsiasi uso, riproduzione o divulgazione di questo messaggio e' vietato. Nel caso in cui aveste ricevuto questo messaggio per errore, vogliate cortesemente avvertire il mittente e distruggere il presente messaggio. -- Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] multiprocessing and pytables
Hello Ernesto, So you are actually asking two different questions, one on reading and the other on writing. In general reading, or querying, with multiprocessing works very well. Writing to a single file with multiple processes is destined to failure though. So the strategy that many people have adopted is to have multiple processes create the data and then have a master process which acts as a queue for writing out the data. Please see the example here for more inspiration [1]. Note that we have been having problems recently with multiprocess writing out to multiple files, but that is not what you want to do. Be Well Anthony 1. https://github.com/PyTables/PyTables/blob/develop/examples/multiprocess_access_queues.py On Mon, Oct 15, 2012 at 11:45 AM, Ernesto Picardi e.pica...@unical.itwrote: Dear all, I have a hdf5 file including several tables. To speed up the creation of all tables, could I create each individual table by independent processes launched by multiprocessing module? Could I employ independent processes to query diverse tables of the same hdf5 file? Thank you very much in advance for whatever answer. Regards, Ernesto Riservatezza / Confidentiality In ottemperanza al D.Lgs. n. 196 del 30/6/2003 in materia di protezione dei dati personali, le informazioni contenute in questo messaggio sono strettamente riservate ed esclusivamente indirizzate al destinatario indicato (oppure alla persona responsabile di rimetterlo al destinatario). Vogliate tener presente che qualsiasi uso, riproduzione o divulgazione di questo messaggio e' vietato. Nel caso in cui aveste ricevuto questo messaggio per errore, vogliate cortesemente avvertire il mittente e distruggere il presente messaggio. -- Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users