Hi Daπid,
So in general there are a couple of different ways of tackling this issue.
The one that you choose will depend on your desired scalablity and existing
architecture. I'll outline some options now:
1) What you described. Every process writes out its own library and then
a post-process sweeps through and combines them all later. This is
probably the easiest to implement. You wouldn't even need to dump them
to ASCII. Tables have an append() method you would find useful.
2) Have one library per node (ie 10 total libraries, 4 processes per
library).
If the writing is done in a thread safe way, then you only have to sweep
through and post-process 10 files. Naturally, the individual file sizes
are
larger.
3) Have one master process whose sole job it is to write the single library.
All other 'compute' processes communicate with this process. The compute
processes will calculate a row of the table and send it back over the wire
to the master process as a tuple. The master process will take this row,
put it on a stack and crank through the stack, adding rows to the table when
it has free time. For communication, you could use something like JSON RPC
(in the Python standard library) or ZeroMQ / pyzmq (which is easy to use and
has a lot of nice features) or MPI / mpi4py (which is meant for high
performance
computing concerns). No post-processing is needed for this strategy.
None of this should every require your writing a plain text file ever. I
hope that
this helps!
Be Well
Anthony
On Tue, Mar 6, 2012 at 6:08 PM, Daπid davidmen...@gmail.com wrote:
It was me, at that moment I hadn't confirmed my subscription. Sorry!
On Wed, Mar 7, 2012 at 1:06 AM, Francesc Alted fal...@pytables.org
wrote:
This has been probably sent from an unsubscribed address.
Begin forwarded message:
From: pytables-users-boun...@lists.sourceforge.net
Subject: Auto-discard notification
Date: March 6, 2012 2:55:54 PM PST
To: pytables-users-ow...@lists.sourceforge.net
The attached message has been automatically discarded.
From: Daπid davidmen...@gmail.com
Subject: Merging multiple DB
Date: March 6, 2012 2:55:27 PM PST
To: pytables-users@lists.sourceforge.net
Hello.
First of all, I have to warn I am an absolute newbie to PyTables and
DB, so please forgive my conceptual holes.
I am running a Monte Carlo simulation of an embarrassingly
parallelizable problem. The calculations are being done on a grid of
10 computers QuadCore, running each one four independent processes. I
assume the safer is to generate one DB per process, ending up with
forty different (but equivalent) DB. My question is: is there any easy
way of merging all of them?
The final size of the DB will be around ten columns of numbers by a
few million rows, relatively small, so I compression is not required
and reading optimization is not vital.
The simplest -and maybe shabby- way I can think of is to output every
thread on different ASCII, read them all and insert them in a master
DB, but this looks inefficient and cumbersome to me.
Thank you very much,
David.
-- Francesc Alted
--
Virtualization Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/
___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
--
Virtualization Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/
___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
--
Virtualization Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
