Re: [PHP] Fork and zombies
Waynn Lue wrote:
Here's pseudo code for what I'm trying to do:
foreach ($things as $thing) {
info = getInfo($thing); // uses a db connection
makeApiCall(info);
}
makeApiCall(info) {
$pid = pcntl_fork();
if ($pid == -1) {
die(could not fork);
} else if ($pid) {
// parent, return the child pid
echo child pid $pid\n;
return;
} else {
// do some api calls
exit;
}
}
But after I spawn off the process, getInfo($thing) errors out sometime
later on with an invalid query error, because I think the db
connection is gone. I thought adding exit in the child process
would be enough, but that doesn't seem to work, I still get the same
error. Why would the child process affect the query in the parent
process, especially if I exit in the child process?
First things first - I would add a pcntl_wait like this:
foreach ($things as $thing) {
info = getInfo($thing); // uses a db connection
makeApiCall(info);
switch( $p=pcntl_wait( $stat, WNOHANG ) )
{
case -1: echo some sort of error in pcntl_wait()\n; break;
case 0: break;
default:
echo child $p finished\n;
}
}
Second, it sounds like you're expecting to reuse your database
connection from getInfo() in the child you're forking in makeAPIcall()?
I don't think that really works - I think you need a new connection per
child.
/Per
--
Per Jessen, Zürich (3.9°C)
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Fork and zombies
Here's pseudo code for what I'm trying to do:
foreach ($things as $thing) {
info = getInfo($thing); // uses a db connection
makeApiCall(info);
}
makeApiCall(info) {
$pid = pcntl_fork();
if ($pid == -1) {
die(could not fork);
} else if ($pid) {
// parent, return the child pid
echo child pid $pid\n;
return;
} else {
// do some api calls
exit;
}
}
But after I spawn off the process, getInfo($thing) errors out sometime
later on with an invalid query error, because I think the db
connection is gone. I thought adding exit in the child process
would be enough, but that doesn't seem to work, I still get the same
error. Why would the child process affect the query in the parent
process, especially if I exit in the child process?
First things first - I would add a pcntl_wait like this:
foreach ($things as $thing) {
info = getInfo($thing); // uses a db connection
makeApiCall(info);
switch( $p=pcntl_wait( $stat, WNOHANG ) )
{
case -1: echo some sort of error in pcntl_wait()\n; break;
case 0: break;
default:
echo child $p finished\n;
}
}
I actually tried this in the meantime:
$pid = pcntl_fork();
if ($pid == -1) {
die(could not fork);
} else if ($pid) {
// parent, return the child pid
echo child pid $pid waiting\n;
pcntl_waitpid($pid, $status);
if (pcntl_wifexited($status)) {
echo finished [$status] waiting\n;
return;
} else {
echo ERROR\n;
}
But it still has the same problem, and I'm also trying to avoid pcntl_wait
or pcntl_waitpid at all because I still want to do it asynchronously. I
even tried rewriting it do return $pid in the parent thread, and then
aggregate them at the calling function level, and then loop through them all
with pcntl_waitpid, but that didn't work either.
Second, it sounds like you're expecting to reuse your database
connection from getInfo() in the child you're forking in makeAPIcall()?
I don't think that really works - I think you need a new connection per
child.
Oh, in this case, I don't do anything with the database at all in the child
thread. I was worried there would be some errors there so I actually
commented out all db accesses in the child thread, but it still somehow
closes the parent's db connection (or at least causes the query not to work,
somehow).
Re: [PHP] Fork and zombies
Waynn Lue wrote:
I actually tried this in the meantime:
$pid = pcntl_fork();
if ($pid == -1) {
die(could not fork);
} else if ($pid) {
// parent, return the child pid
echo child pid $pid waiting\n;
pcntl_waitpid($pid, $status);
if (pcntl_wifexited($status)) {
echo finished [$status] waiting\n;
return;
} else {
echo ERROR\n;
}
I think your waitpid() is in the wrong place, and at least you need use
WNOHANG - unless you specifically want to wait for each child to finish
before starting another one.
But it still has the same problem, and I'm also trying to avoid
pcntl_wait or pcntl_waitpid at all because I still want to do it
asynchronously.
pcntl_wait(WNOHANG) will make everything work asynchronously.
/Per
--
Per Jessen, Zürich (5.8°C)
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Fork and zombies
Waynn Lue wrote: Ah, I was changing it to waiting for each child to finish in order to see if I could narrow down my db problem, because I figure this should be more or less equivalent to running it synchronously. Even like this, though, it still causes the db problem. I think the database problem is caused by your child inheriting the connection, and then closing it - whilst the parent is still using it. Your parent needs to open a new connection. /Per -- Per Jessen, Zürich (8.8°C) -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Fork and zombies
I think your waitpid() is in the wrong place, and at least you need use WNOHANG - unless you specifically want to wait for each child to finish before starting another one. But it still has the same problem, and I'm also trying to avoid pcntl_wait or pcntl_waitpid at all because I still want to do it asynchronously. pcntl_wait(WNOHANG) will make everything work asynchronously. Ah, I was changing it to waiting for each child to finish in order to see if I could narrow down my db problem, because I figure this should be more or less equivalent to running it synchronously. Even like this, though, it still causes the db problem.
[PHP] Fork and zombies
I periodically run a script that makes a call against a remote API, which
takes some time to return. In an attempt to increase thoroughput, I decided
to investigate using pnctl_fork to spawn off multiple processes to make the
call, since the slowest part is the network part of it (and waiting for the
server response). I ended up writing a script that did this:
$pid = pnctl_fork();
if ($pid == -1) {
die('could not fork');
} else if ($pid) {
echo parent $pid\n;
return;
} else {
// make API call
}
While this works, it unfortunately leaves behind a zombie process every
single time. I could rewrite this to instead call fork multiple times, then
wait on all of them to return, but as my system is currently architected,
the easiest way would be to fire and forget for this script. Does anyone
have any ideas on the best way to do it? The other way I've done this is to
use exec(php foo.php) and redirecting stdout and stderr to /dev/null.
Waynn
Re: [PHP] Fork and zombies
Waynn Lue wrote:
I periodically run a script that makes a call against a remote API,
which
takes some time to return. In an attempt to increase thoroughput, I
decided to investigate using pnctl_fork to spawn off multiple
processes to make the call, since the slowest part is the network part
of it (and waiting for the
server response). I ended up writing a script that did this:
$pid = pnctl_fork();
if ($pid == -1) {
die('could not fork');
} else if ($pid) {
echo parent $pid\n;
return;
} else {
// make API call
}
While this works, it unfortunately leaves behind a zombie process
every single time.
You need to call pcntl_wait() or pcntl_waitpid().
/Per
--
Per Jessen, Zürich (4.2°C)
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Fork and zombies
Waynn Lue wrote:
I periodically run a script that makes a call against a remote API,
which
takes some time to return. In an attempt to increase thoroughput, I
decided to investigate using pnctl_fork to spawn off multiple
processes to make the call, since the slowest part is the network part
of it (and waiting for the
server response). I ended up writing a script that did this:
$pid = pnctl_fork();
if ($pid == -1) {
die('could not fork');
} else if ($pid) {
echo parent $pid\n;
return;
} else {
// make API call
}
While this works, it unfortunately leaves behind a zombie process
every single time.
You need to call pcntl_wait() or pcntl_waitpid().
Right, but if I do that, then the parent has to wait until the child
completes before it exits. Is there any way to have the parent exit before
the child but prevent the zombie process for existing?
Re: [PHP] Fork and zombies
Waynn Lue wrote: While this works, it unfortunately leaves behind a zombie process every single time. You need to call pcntl_wait() or pcntl_waitpid(). Right, but if I do that, then the parent has to wait until the child completes before it exits. No it doesn't - just call pcntl_wait() with WNOHANG to check the status of a child. /Per -- Per Jessen, Zürich (6.5°C) -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Fork and zombies
While this works, it unfortunately leaves behind a zombie process
every single time.
You need to call pcntl_wait() or pcntl_waitpid().
Right, but if I do that, then the parent has to wait until the child
completes before it exits.
No it doesn't - just call pcntl_wait() with WNOHANG to check the status
of a child.
Hm, so now I'm not so sure what's happening. Before trying your suggestion,
I ran my script again without pcntl_wait so that I'd have a baseline for how
many zombie processes are being created, and it turns out that I don't think
any are. Previously, I was doing
ps aux | awk '{ print $8 $2 }' | grep -w Z
to find zombie processes, and it turns out that they existed from before, as
the count didn't change after I ran my script. That leads to another
problem, though, in that the resource for the database doesn't seem to be
available for the parent after the child exits. Here's pseudo code for what
I'm trying to do:
foreach ($things as $thing) {
info = getInfo($thing); // uses a db connection
makeApiCall(info);
}
makeApiCall(info) {
$pid = pcntl_fork();
if ($pid == -1) {
die(could not fork);
} else if ($pid) {
// parent, return the child pid
echo child pid $pid\n;
return;
} else {
// do some api calls
exit;
}
}
But after I spawn off the process, getInfo($thing) errors out sometime later
on with an invalid query error, because I think the db connection is
gone. I thought adding exit in the child process would be enough, but
that doesn't seem to work, I still get the same error. Why would the child
process affect the query in the parent process, especially if I exit in the
child process?
Following your suggestion, I also tried pcntl_wait in the upperlevel
function, but still had the same problem.
Thanks for any insight,
Waynn
