[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13172


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/13172#issuecomment-220602808
  
Merged to master/2.0


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[WeichenXu123]

Github user WeichenXu123 commented on the pull request:

https://github.com/apache/spark/pull/13172#issuecomment-220260517
  
@srowen 
Modified as you expected.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13172#issuecomment-220205291
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13172#issuecomment-220205292
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58827/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13172#issuecomment-220205168
  
**[Test build #58827 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58827/consoleFull)**
 for PR 13172 at commit 
[`4375c79`](https://github.com/apache/spark/commit/4375c79a0add93292ac21b2950062108754a90f2).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13172#issuecomment-220190063
  
**[Test build #58827 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58827/consoleFull)**
 for PR 13172 at commit 
[`4375c79`](https://github.com/apache/spark/commit/4375c79a0add93292ac21b2950062108754a90f2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/13172#discussion_r63744491
  
--- Diff: sbin/spark-daemon.sh ---
@@ -162,6 +162,16 @@ run_command() {
   esac
 
   echo "$newpid" > "$pid"
+  
+  #here add this loop to check the daemon process turn into java process. 
It takes at most 5 seconds.
--- End diff --

This would be somewhat clearer as something like "Poll for up to 5 seconds 
for the java process to start" Otherwise LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13172#issuecomment-220072583
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13172#issuecomment-220072220
  
**[Test build #58780 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58780/consoleFull)**
 for PR 13172 at commit 
[`55d911f`](https://github.com/apache/spark/commit/55d911f29137441ddaa1e9022bf85edac86f72af).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13172#issuecomment-220072586
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58780/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[WeichenXu123]

Github user WeichenXu123 closed the pull request at:

https://github.com/apache/spark/pull/12978


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[WeichenXu123]

Github user WeichenXu123 commented on a diff in the pull request:

https://github.com/apache/spark/pull/12978#discussion_r63708450
  
--- Diff: sbin/spark-daemon.sh ---
@@ -162,6 +162,15 @@ run_command() {
   esac
 
   echo "$newpid" > "$pid"
+
+  for i in {1..10}
+  do
+if [[ $(ps -p "$newpid" -o comm=) =~ "java" ]]; then
+   break
+fi
+sleep 0.5
--- End diff --

new clean PR is here:
https://github.com/apache/spark/pull/13172
thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13172#issuecomment-220037041
  
**[Test build #58780 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58780/consoleFull)**
 for PR 13172 at commit 
[`55d911f`](https://github.com/apache/spark/commit/55d911f29137441ddaa1e9022bf85edac86f72af).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/12978#discussion_r63701914
  
--- Diff: sbin/spark-daemon.sh ---
@@ -162,6 +162,15 @@ run_command() {
   esac
 
   echo "$newpid" > "$pid"
+
+  for i in {1..10}
+  do
+if [[ $(ps -p "$newpid" -o comm=) =~ "java" ]]; then
+   break
+fi
+sleep 0.5
--- End diff --

OK, that's fair, we still need to see if it dies. I suppose there could be 
a second loop to keep watching for it to die, but in the normal case that still 
requires waiting some number of seconds. This is OK, except for the PR problem 
of course.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[WeichenXu123]

Github user WeichenXu123 commented on a diff in the pull request:

https://github.com/apache/spark/pull/12978#discussion_r63700865
  
--- Diff: sbin/spark-daemon.sh ---
@@ -162,6 +162,15 @@ run_command() {
   esac
 
   echo "$newpid" > "$pid"
+
+  for i in {1..10}
+  do
+if [[ $(ps -p "$newpid" -o comm=) =~ "java" ]]; then
+   break
+fi
+sleep 0.5
--- End diff --

Oh..the sleep 2 statement can't move.

the loop may take at most 5 seconds to check whether the daemon turn into 
java daemon. When it become java daemon, it jump out the loop.

**then must sleep 2 seconds,**
then check whether the java daemon die. Because java daemon may exist for 
some time and then throw an Exception and die. So here need to wait for some 
time.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/12978#issuecomment-220021986
  
OK maybe, though here you show it took 0.5s.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[WeichenXu123]

Github user WeichenXu123 commented on the pull request:

https://github.com/apache/spark/pull/12978#issuecomment-220021066
  
In order to check this potential problem more carefully, We can add the 
following test code like this:

`
echo "$newpid" > "$pid"
  **for i in {1..30};do
echo "check daemon stage $i" `ps -p "$newpid" -o comm=`
sleep 0.1
  done**
  sleep 2
  # Check if the process has died; in that case we'll tail the log so the 
user can see
  if [[ ! $(ps -p "$newpid" -o comm=) =~ "java" ]]; then
echo "failed to launch $command:"
tail -2 "$log" | sed 's/^/  /'
echo "full log in $log"
  fi
`

then run the start daemon script it will print such as:

sbin/start-master.sh 
starting org.apache.spark.deploy.master.Master, logging to 
/diskext/mySpark/spark/logs/spark-root-org.apache.spark.deploy.master.Master-1-n131.out
check daemon stage 1 bash
check daemon stage 2 bash
check daemon stage 3 bash
check daemon stage 4 bash
check daemon stage 5 java
check daemon stage 6 java
check daemon stage 7 java
check daemon stage 8 java


The running result show that the bash status daemon will exist for sometime.
Especially when the machine just startup and OS cache is clean,  the time 
may exceed 2s.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/12978#discussion_r63696847
  
--- Diff: sbin/spark-daemon.sh ---
@@ -162,6 +162,15 @@ run_command() {
   esac
 
   echo "$newpid" > "$pid"
+
+  for i in {1..10}
+  do
+if [[ $(ps -p "$newpid" -o comm=) =~ "java" ]]; then
+   break
+fi
+sleep 0.5
--- End diff --

Maybe once a second for 5 seconds? you need to remove the "sleep 2" below.
Maybe you can use a variable rather than re-test the condition after the 
loop?

However, the PR has also been messed up with some unrelated changes. Can 
you fix or else open a new PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[WeichenXu123]

Github user WeichenXu123 commented on the pull request:

https://github.com/apache/spark/pull/12978#issuecomment-220016724
  
@srowen According to your suggestion, I add a loop to check whether it pass 
STAGE-1 and launch java daemon. And I recover the check statement `! $(ps -p 
"$newpid" -o comm=) =~ "java"`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/12978#issuecomment-219290733
  
I think the way forward is to make a different change, if you're still 
pretty sure this is an issue: try this a small number of times in a loop until 
it succeeds or the max iterations is reached. Same check, just tried once a 
second for 5 seconds or something.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[WeichenXu123]

Github user WeichenXu123 commented on the pull request:

https://github.com/apache/spark/pull/12978#issuecomment-218097327
  
@srowen Er...I am also a little strange while stage 1 may took a long time 
but it really happen several times... If there is time I will do a more 
detailed test.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/12978#issuecomment-218090853
  
At that point the Java process is already running though. I want to make 
sure we are solving the problem you think. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[WeichenXu123]

Github user WeichenXu123 commented on the pull request:

https://github.com/apache/spark/pull/12978#issuecomment-218089784
  
@srowen At my virtual machine, After OS started and start spark daemon, the 
stage 1 describe above will took a long time,often exceeding 2s, I think the 
java daemon start need to load a huge number of jars so it will took several 
time when started at the first time.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/12978#issuecomment-218071565
  
Hm, how would stage 1 take any significant time though? I'm missing what 
could take that much time.
The problem with this change is that it fails to check whether the java 
process started, so we can't do this. At best a loop to check the status every 
second for several seconds or something.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[WeichenXu123]

Github user WeichenXu123 commented on the pull request:

https://github.com/apache/spark/pull/12978#issuecomment-218040874
  
@srowen 
because spark-daemon.sh using exec command to start the java daemon process.
when run script, the spark-daemon.sh process will exists for a little 
time(stage1) and then changed to a java process(stage2), then if java process 
throw exception, it will die.
if stage1 spending more then 2s, then the judgement if [[ ! $(ps -p 
"$newpid" -o comm=) =~ "java" ]]; is fail, it will regard the process as failed 
but in fact it has not come into stag2. In my virtual machine, this thing offen 
happen.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/12978#discussion_r62428386
  
--- Diff: sbin/spark-daemon.sh ---
@@ -164,7 +164,7 @@ run_command() {
   echo "$newpid" > "$pid"
   sleep 2
   # Check if the process has died; in that case we'll tail the log so the 
user can see
-  if [[ ! $(ps -p "$newpid" -o comm=) =~ "java" ]]; then
+  if [[ -z $(ps --no-headers -p "$newpid") ]]; then
--- End diff --

I don't see the option `--no-headers` on OS X, so I suspect it could be a 
BSD vs Gnu thing. It will fail though.

I don't see how this change addresses the problem though. It still occurs 2 
seconds after the start, but it's not checking whether the java process 
started. Do you mean some kind of loop here, like checking 3 times?

What's the scenario where the bash script takes a long time to run?




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12978#issuecomment-217647731
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15203][Deploy]The spark daemon shell sc...

[WeichenXu123]

GitHub user WeichenXu123 opened a pull request:

https://github.com/apache/spark/pull/12978

[SPARK-15203][Deploy]The spark daemon shell script error, daemon process 
start successfully but script output fail message.

This bug is because, sbin/spark-daemon.sh script use bin/spark-class shell 
to start daemon, then sleep  2s and check whether the daemon process exists, 
using shell script like following:
if [[ ! $(ps -p "$newpid" -o comm=) =~ "java" ]]
the problem is, some machine with bad performance may start the daemon 
using a long time(exceeding 2s), but still can start daemon successfully, but 
in this case, the shell script judgement  ! $(ps -p "$newpid" -o comm=) =~ 
"java"  will fail, because at this time, the $newpid process is still shell 
process, until the daemon started, it turns into java process. That's the 
reason cause the bug.
In order to reproduce the bug more easily, you can change 
sleep 2  ==>  sleep 0.01 (sbin/spark-daemon.sh, line 165)

to fix this bug, 
I replace
 ! $(ps -p "$newpid" -o comm=) =~ "java"   (sbin/spark-daemon.sh, line 167)
with 
-z $(ps --no-headers -p "$newpid")



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/WeichenXu123/spark fix_shell_001

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/12978.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #12978


commit 750078c3ab0d34f55db935b9518b3288f98cade1
Author: WeichenXu 
Date:   2016-05-07T13:00:58Z

fix shell start daemon script bug




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org