Where is the thread safe array? I tried recently with IPC::Shareable and it blows up very easily... I shipped the code to the author and have not heard from him yet (1 weeks now).
What I did was: I tied a hash with a shared memory and use one process to push an unique member into the hash and have another process to pop up data, and I easily run out of memory and tie function failed... (1000 members with 20 bytes strings) concurrently... Any comments? -----Original Message----- From: CHEN SEI-LIM [mailto:[EMAIL PROTECTED]] Sent: Sunday, November 11, 2001 10:53 PM To: [EMAIL PROTECTED]; [EMAIL PROTECTED] Subject: Re: forking and DBI [EMAIL PROTECTED] wrote: > -- Tim Bunce <[EMAIL PROTECTED]> on 11/09/01 22:48:07 +0000 > > > On Fri, Nov 09, 2001 at 11:35:09AM -0500, [EMAIL PROTECTED] wrote: > >> So, if I've got a massively parallel database, and I want to run > >> a batch of queries against it in parallel--let's say five hundred, > >> six at a time, what would be the best way to do that? My first thought > >> was a fork/connect for every query, but that's way too much overhead. > > > > You could start six subprocesses and feed the queries to them, perhaps via a > > pipe. > > Put an array of SQL into shared memory using a shared-safe > tied hash and shift, fork however many procs you like and > have them shift (or pop) the queries off of the list. The > IPC::Sharable or threadsafe array module (or both maybe?) > implement proc-safe updte operations. > > -- > Steven Lembark 2930 W. Palmer > Workhorse Computing Chicago, IL 60647 > +1 800 762 1582 Put those SQLs into a temp file, fork processes you like and read SQL from temp file. For safety, use lock to make sure exclusive reading and current offset marking. After SQL querying write the results back to file as log. Then you can check log records of this file if every querys succeed?
