Re: [PHP] Fork and zombies
Waynn Lue wrote:
Here's pseudo code for what I'm trying to do:
foreach ($things as $thing) {
info = getInfo($thing); // uses a db connection
makeApiCall(info);
}
makeApiCall(info) {
$pid = pcntl_fork();
if ($pid == -1) {
die(could not fork);
} else if ($pid) {
// parent, return the child pid
echo child pid $pid\n;
return;
} else {
// do some api calls
exit;
}
}
But after I spawn off the process, getInfo($thing) errors out sometime
later on with an invalid query error, because I think the db
connection is gone. I thought adding exit in the child process
would be enough, but that doesn't seem to work, I still get the same
error. Why would the child process affect the query in the parent
process, especially if I exit in the child process?
First things first - I would add a pcntl_wait like this:
foreach ($things as $thing) {
info = getInfo($thing); // uses a db connection
makeApiCall(info);
switch( $p=pcntl_wait( $stat, WNOHANG ) )
{
case -1: echo some sort of error in pcntl_wait()\n; break;
case 0: break;
default:
echo child $p finished\n;
}
}
Second, it sounds like you're expecting to reuse your database
connection from getInfo() in the child you're forking in makeAPIcall()?
I don't think that really works - I think you need a new connection per
child.
/Per
--
Per Jessen, Zürich (3.9°C)
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Fork and zombies
Here's pseudo code for what I'm trying to do:
foreach ($things as $thing) {
info = getInfo($thing); // uses a db connection
makeApiCall(info);
}
makeApiCall(info) {
$pid = pcntl_fork();
if ($pid == -1) {
die(could not fork);
} else if ($pid) {
// parent, return the child pid
echo child pid $pid\n;
return;
} else {
// do some api calls
exit;
}
}
But after I spawn off the process, getInfo($thing) errors out sometime
later on with an invalid query error, because I think the db
connection is gone. I thought adding exit in the child process
would be enough, but that doesn't seem to work, I still get the same
error. Why would the child process affect the query in the parent
process, especially if I exit in the child process?
First things first - I would add a pcntl_wait like this:
foreach ($things as $thing) {
info = getInfo($thing); // uses a db connection
makeApiCall(info);
switch( $p=pcntl_wait( $stat, WNOHANG ) )
{
case -1: echo some sort of error in pcntl_wait()\n; break;
case 0: break;
default:
echo child $p finished\n;
}
}
I actually tried this in the meantime:
$pid = pcntl_fork();
if ($pid == -1) {
die(could not fork);
} else if ($pid) {
// parent, return the child pid
echo child pid $pid waiting\n;
pcntl_waitpid($pid, $status);
if (pcntl_wifexited($status)) {
echo finished [$status] waiting\n;
return;
} else {
echo ERROR\n;
}
But it still has the same problem, and I'm also trying to avoid pcntl_wait
or pcntl_waitpid at all because I still want to do it asynchronously. I
even tried rewriting it do return $pid in the parent thread, and then
aggregate them at the calling function level, and then loop through them all
with pcntl_waitpid, but that didn't work either.
Second, it sounds like you're expecting to reuse your database
connection from getInfo() in the child you're forking in makeAPIcall()?
I don't think that really works - I think you need a new connection per
child.
Oh, in this case, I don't do anything with the database at all in the child
thread. I was worried there would be some errors there so I actually
commented out all db accesses in the child thread, but it still somehow
closes the parent's db connection (or at least causes the query not to work,
somehow).
Re: [PHP] Fork and zombies
Waynn Lue wrote:
I actually tried this in the meantime:
$pid = pcntl_fork();
if ($pid == -1) {
die(could not fork);
} else if ($pid) {
// parent, return the child pid
echo child pid $pid waiting\n;
pcntl_waitpid($pid, $status);
if (pcntl_wifexited($status)) {
echo finished [$status] waiting\n;
return;
} else {
echo ERROR\n;
}
I think your waitpid() is in the wrong place, and at least you need use
WNOHANG - unless you specifically want to wait for each child to finish
before starting another one.
But it still has the same problem, and I'm also trying to avoid
pcntl_wait or pcntl_waitpid at all because I still want to do it
asynchronously.
pcntl_wait(WNOHANG) will make everything work asynchronously.
/Per
--
Per Jessen, Zürich (5.8°C)
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Fork and zombies
Waynn Lue wrote: Ah, I was changing it to waiting for each child to finish in order to see if I could narrow down my db problem, because I figure this should be more or less equivalent to running it synchronously. Even like this, though, it still causes the db problem. I think the database problem is caused by your child inheriting the connection, and then closing it - whilst the parent is still using it. Your parent needs to open a new connection. /Per -- Per Jessen, Zürich (8.8°C) -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Fork and zombies
I think your waitpid() is in the wrong place, and at least you need use WNOHANG - unless you specifically want to wait for each child to finish before starting another one. But it still has the same problem, and I'm also trying to avoid pcntl_wait or pcntl_waitpid at all because I still want to do it asynchronously. pcntl_wait(WNOHANG) will make everything work asynchronously. Ah, I was changing it to waiting for each child to finish in order to see if I could narrow down my db problem, because I figure this should be more or less equivalent to running it synchronously. Even like this, though, it still causes the db problem.
Re: [PHP] Fork and zombies
Waynn Lue wrote:
I periodically run a script that makes a call against a remote API,
which
takes some time to return. In an attempt to increase thoroughput, I
decided to investigate using pnctl_fork to spawn off multiple
processes to make the call, since the slowest part is the network part
of it (and waiting for the
server response). I ended up writing a script that did this:
$pid = pnctl_fork();
if ($pid == -1) {
die('could not fork');
} else if ($pid) {
echo parent $pid\n;
return;
} else {
// make API call
}
While this works, it unfortunately leaves behind a zombie process
every single time.
You need to call pcntl_wait() or pcntl_waitpid().
/Per
--
Per Jessen, Zürich (4.2°C)
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Fork and zombies
Waynn Lue wrote:
I periodically run a script that makes a call against a remote API,
which
takes some time to return. In an attempt to increase thoroughput, I
decided to investigate using pnctl_fork to spawn off multiple
processes to make the call, since the slowest part is the network part
of it (and waiting for the
server response). I ended up writing a script that did this:
$pid = pnctl_fork();
if ($pid == -1) {
die('could not fork');
} else if ($pid) {
echo parent $pid\n;
return;
} else {
// make API call
}
While this works, it unfortunately leaves behind a zombie process
every single time.
You need to call pcntl_wait() or pcntl_waitpid().
Right, but if I do that, then the parent has to wait until the child
completes before it exits. Is there any way to have the parent exit before
the child but prevent the zombie process for existing?
Re: [PHP] Fork and zombies
Waynn Lue wrote: While this works, it unfortunately leaves behind a zombie process every single time. You need to call pcntl_wait() or pcntl_waitpid(). Right, but if I do that, then the parent has to wait until the child completes before it exits. No it doesn't - just call pcntl_wait() with WNOHANG to check the status of a child. /Per -- Per Jessen, Zürich (6.5°C) -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Fork and zombies
While this works, it unfortunately leaves behind a zombie process
every single time.
You need to call pcntl_wait() or pcntl_waitpid().
Right, but if I do that, then the parent has to wait until the child
completes before it exits.
No it doesn't - just call pcntl_wait() with WNOHANG to check the status
of a child.
Hm, so now I'm not so sure what's happening. Before trying your suggestion,
I ran my script again without pcntl_wait so that I'd have a baseline for how
many zombie processes are being created, and it turns out that I don't think
any are. Previously, I was doing
ps aux | awk '{ print $8 $2 }' | grep -w Z
to find zombie processes, and it turns out that they existed from before, as
the count didn't change after I ran my script. That leads to another
problem, though, in that the resource for the database doesn't seem to be
available for the parent after the child exits. Here's pseudo code for what
I'm trying to do:
foreach ($things as $thing) {
info = getInfo($thing); // uses a db connection
makeApiCall(info);
}
makeApiCall(info) {
$pid = pcntl_fork();
if ($pid == -1) {
die(could not fork);
} else if ($pid) {
// parent, return the child pid
echo child pid $pid\n;
return;
} else {
// do some api calls
exit;
}
}
But after I spawn off the process, getInfo($thing) errors out sometime later
on with an invalid query error, because I think the db connection is
gone. I thought adding exit in the child process would be enough, but
that doesn't seem to work, I still get the same error. Why would the child
process affect the query in the parent process, especially if I exit in the
child process?
Following your suggestion, I also tried pcntl_wait in the upperlevel
function, but still had the same problem.
Thanks for any insight,
Waynn
Re: [PHP] fork/spawnzombie question
bruce wrote: Hi Nathan/Torok... Hey guys... got a bit of a question. I'm playing around with the php/for/pcntl_exec functions and I've got a process that spawns off a bunch of child processes. Unfortunately, I'm getting to where I have 100's of zombie child processes that I can see from the linux/processTBL. I don't want to have my master loop do a waitpid() call, as it would block on the wait for one of the child processes to exit. No, it wouldn't - just use WNOHANG: pcntl_waitpid( -1, $status, WNOHANG ); /Per Jessen -- Per Jessen, Zürich (-0.9°C) -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Fork PHP script X at a time.
Not sure if anyone is interested in what I have come up with, its busy
running just now but it seems to be performing ok.
$do =0;
$ok =0;
$count=0;
while($do!=1){
if($ok==0){
$sql = SELECT * FROM locations WHERE parent = '0' AND doing=0 and
done !=1 ORDER BY num DESC LIMIT 0 , 1;
$res = mysql_query($sql) or die ($sql.mysql_error());
while($_data = mysql_fetch_array($res)){
$id = $county_data[id];
$sql2 = UPDATE locations SET doing = 1 WHERE id= '$id' AND
siteid=1 AND parent = '0';
$res3 = mysql_query($sql2);
exec(/usr/local/bin/php fork.php .$data[id]. /dev/null
);
}
}
$sql = SELECT * FROM locations WHERE parent = '0' AND doing=1 AND done=0;
$res = mysql_query($sql);
$oknumber = mysql_num_rows($res);
if($oknumber == 5) $ok=1;
else $ok=0;
$sql = SELECT * FROM locations WHERE parent = '0' AND done=0;
$res2 = mysql_query($sql);
$donumber = mysql_num_rows($res2);
if($donumber==0) $do=1;
}
In my fork.php script I set it to done=1 and doing = 0 once it has
finished.
This is pretty rough and ready and as my shell account won't let me do a
top i can't tell what sort of drain my loop is having on the system,
but hopefully it will be ok.
Just thought it might be an idea to post up this solution in-case
someone else has the same problem.
John
On Mon, 2004-09-20 at 16:36, John Wards wrote:
Hi, I have a bit of a cold today so I probably would have figured this
out for myself eventually but hey ;-)
Right I have a script that I need to run around 90 times thru a cron job
but passing a different i.d. to it each time.
I have experimented with running all 90 at once and its a no go as it
just makes the server run to slow and the session cookies I use time out
(I am doing some libcurl things).
Also running them in a loop one at a time is no use because it takes
about 8 hours.
What I would like to do is fork off say 5-10 at a time and when one is
done start another one in its place, as they all take different length
of times to compleate.
Any clues at where to start? Here is my fork all 90+ code...
$sql = SELECT * FROM locations WHERE parent = '0';
$res = mysql_query($sql) or die ($sql.mysql_error());
while($data = mysql_fetch_array($res)){
exec(/usr/local/bin/php fork.php .$_data[id]. /dev/null );
}
Cheers
John Wards
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Fork PHP script X at a time.
On Tue, 2004-09-21 at 13:58, John Wards wrote: Not sure if anyone is interested in what I have come up with, its busy running just now but it seems to be performing ok. Again adding to this thread for future reference. Its just compleated all 90 threads in 1 hour 50 minutes and that was running 5 at a time. Very sucessful bit of code, I'm quite cuffed ;-) -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Fork PHP script X at a time.
Hi, I have a bit of a cold today so I probably would have figured this
out for myself eventually but hey ;-)
Right I have a script that I need to run around 90 times thru a cron job
but passing a different i.d. to it each time.
I have experimented with running all 90 at once and its a no go as it
just makes the server run to slow and the session cookies I use time out
(I am doing some libcurl things).
Also running them in a loop one at a time is no use because it takes
about 8 hours.
What I would like to do is fork off say 5-10 at a time and when one is
done start another one in its place, as they all take different length
of times to compleate.
Any clues at where to start? Here is my fork all 90+ code...
$sql = SELECT * FROM locations WHERE parent = '0';
$res = mysql_query($sql) or die ($sql.mysql_error());
while($data = mysql_fetch_array($res)){
exec(/usr/local/bin/php fork.php .$_data[id]. /dev/null );
}
You should be able to find out about the PID's for every forked
process, and use a loop (advisable with a sleep() in it) to see how
many procs are still running.. When the amount drops somewhat, spawn
another ..
Hope will get you somewhere!
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Fork PHP script X at a time.
On Mon, 2004-09-20 at 16:45, Wouter van Vliet wrote:
What I would like to do is fork off say 5-10 at a time and when one is
done start another one in its place, as they all take different length
of times to compleate.
Any clues at where to start? Here is my fork all 90+ code...
$sql = SELECT * FROM locations WHERE parent = '0';
$res = mysql_query($sql) or die ($sql.mysql_error());
while($data = mysql_fetch_array($res)){
exec(/usr/local/bin/php fork.php .$_data[id]. /dev/null );
}
You should be able to find out about the PID's for every forked
process, and use a loop (advisable with a sleep() in it) to see how
many procs are still running.. When the amount drops somewhat, spawn
another ..
Hmm, first off how would I get the PID for each process?
I could store each PID and then sleep for a minute, then check to see if
those PIDs are still valid (How would I do that?!) then fork some more
and loop till fade...
Think I will sleep on it.
John
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Fork PHP script X at a time.
play around with popen() and stream_select()
John Wards wrote:
Hi, I have a bit of a cold today so I probably would have figured this
out for myself eventually but hey ;-)
Right I have a script that I need to run around 90 times thru a cron job
but passing a different i.d. to it each time.
I have experimented with running all 90 at once and its a no go as it
just makes the server run to slow and the session cookies I use time out
(I am doing some libcurl things).
Also running them in a loop one at a time is no use because it takes
about 8 hours.
What I would like to do is fork off say 5-10 at a time and when one is
done start another one in its place, as they all take different length
of times to compleate.
Any clues at where to start? Here is my fork all 90+ code...
$sql = SELECT * FROM locations WHERE parent = '0';
$res = mysql_query($sql) or die ($sql.mysql_error());
while($data = mysql_fetch_array($res)){
exec(/usr/local/bin/php fork.php .$_data[id]. /dev/null );
}
Cheers
John Wards
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Fork PHP script X at a time.
On Mon, 2004-09-20 at 16:56, Marek Kilimajer wrote: play around with popen() and stream_select() Oooh that looks handy. More night time reading I think! Any further ideas to save a man with a cold from using his brain would be great! Cheers John -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Fork PHP script X at a time.
John Wards wrote: Hi, I have a bit of a cold today so I probably would have figured this out for myself eventually but hey ;-) Right I have a script that I need to run around 90 times thru a cron job but passing a different i.d. to it each time. not suprised. I have experimented with running all 90 at once and its a no go as it just makes the server run to slow and the session cookies I use time out (I am doing some libcurl things). Also running them in a loop one at a time is no use because it takes about 8 hours. if your script consumes lots of CPU, lots of memory, lots of bandwidth or any combination of these you may not see a significant improvement by using threads. If you sleep you might find that the total running time will actually increase, so this is something you might want to go about carefully. PHP isn't the easiest language to use when you have to do your own threading but since others have already given good advice on the howto i will refrain from adding my 0.02 -- Raditha Dissanayake. http://www.radinks.com/sftp/ | http://www.raditha.com/megaupload Lean and mean Secure FTP applet with | Mega Upload - PHP file uploader Graphical User Inteface. Just 128 KB | with progress bar. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Fork PHP script X at a time.
On Mon, 20 Sep 2004 16:36:51 +0100, John Wards [EMAIL PROTECTED] wrote:
Right I have a script that I need to run around 90 times thru a cron job
but passing a different i.d. to it each time.
I have experimented with running all 90 at once and its a no go as it
just makes the server run to slow and the session cookies I use time out
(I am doing some libcurl things).
Also running them in a loop one at a time is no use because it takes
about 8 hours.
What I would like to do is fork off say 5-10 at a time and when one is
done start another one in its place, as they all take different length
of times to compleate.
Any clues at where to start? Here is my fork all 90+ code...
$sql = SELECT * FROM locations WHERE parent = '0';
$res = mysql_query($sql) or die ($sql.mysql_error());
while($data = mysql_fetch_array($res)){
exec(/usr/local/bin/php fork.php .$_data[id]. /dev/null );
I'd write a scheduler. Insert the jobs into the database whenever
they need to run. Then have a script that runs via cron that will
monitor system load, and start and stop jobs as required.
http://pear.php.net/package/System_ProcWatch
--
Greg Donald
http://gdconsultants.com/
http://destiney.com/
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Fork PHP script X at a time.
John Wards wrote:
On Mon, 2004-09-20 at 16:56, Marek Kilimajer wrote:
play around with popen() and stream_select()
Oooh that looks handy.
More night time reading I think!
Any further ideas to save a man with a cold from using his brain would
be great!
Cheers
John
Wanted to do something interesting, so here is something to get you
started. It opens a pool of 10 sleep processes that sleep for a random
time. Everytime a sleep process ends its sleep and exits, a new sleep
process is created.
However, it contains some very, very ugly hack. stream_select() breaks
index association of its parameters, so to find out what stream changed
status one has to compare values of the original array ($handles) and
the array passed to stream_select() ($read).
Is it how it's supposed to work or am I missing something?
?php
$handles = array();
for($i = 0; $i 10; $i++) {
$handles[$i] = popen('sleep ' . rand(1, 10), 'r');
echo Opened '$handles[$i]'\n;
}
while(count($handles)) {
$read = $handles;
if (false === ($num_changed_streams = stream_select($read, $write =
NULL, $except = NULL, 10))) { // don't forget to change tv_sec param
/* Error handling */
echo ERROR\n;
} elseif ($num_changed_streams 0) {
foreach($read as $key = $handle) {
echo Closing '$handle'\n;
//echo fread($handle, 2096);
pclose($handle);
if($i 50) {
$handles[$i] = popen('sleep ' . rand(1, 10), 'r');
echo Opened '$handles[$i]'\n;
$i++;
}
}
// UGLY hack here
$handles = array_diff($handles, $read);
}
}
?
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Fork and multi-thread in PHP?
That's one of the main reasons you would want to use the mail() function. I wouldn't advise a fork() here, but if you really insist, see the pcntl extension - http://php.net/pcntl -Rasmus On Wed, 12 Jun 2002, Nathan Cassano wrote: Hi PHP folks, I have a program that sends out email by making socket connection to our email server. The problem is that the program is very slow because it has to finish talking to the email server until it can proceed with the next email (as opposed to the mail() function that just forks off a sendmail process). Is there a way for PHP to fork or multi-thread to do I/O? -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] fork?
Vulcan Logic SRM www.vl-srm.net Every your task is a banana. Regards, Andrey Hristov - Original Message - From: Paul Roberts [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Saturday, February 23, 2002 1:05 PM Subject: [PHP] fork? how do i do two things at the same time, i'm thinking of the equivalent of fork in perl Paul Roberts [EMAIL PROTECTED] -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
