GitHub user WeichenXu123 opened a pull request:
https://github.com/apache/spark/pull/12978
[SPARK-15203][Deploy]The spark daemon shell script error, daemon process
start successfully but script output fail message.
This bug is because, sbin/spark-daemon.sh script use bin/spark-class shell
to start daemon, then sleep 2s and check whether the daemon process exists,
using shell script like following:
if [[ ! $(ps -p "$newpid" -o comm=) =~ "java" ]]
the problem is, some machine with bad performance may start the daemon
using a long time(exceeding 2s), but still can start daemon successfully, but
in this case, the shell script judgement ! $(ps -p "$newpid" -o comm=) =~
"java" will fail, because at this time, the $newpid process is still shell
process, until the daemon started, it turns into java process. That's the
reason cause the bug.
In order to reproduce the bug more easily, you can change
sleep 2 ==> sleep 0.01 (sbin/spark-daemon.sh, line 165)
to fix this bug,
I replace
! $(ps -p "$newpid" -o comm=) =~ "java" (sbin/spark-daemon.sh, line 167)
with
-z $(ps --no-headers -p "$newpid")
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/WeichenXu123/spark fix_shell_001
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12978.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12978
----
commit 750078c3ab0d34f55db935b9518b3288f98cade1
Author: WeichenXu <[email protected]>
Date: 2016-05-07T13:00:58Z
fix shell start daemon script bug
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
