[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-200374692 It will not impact on actuall result, but a error stacktrace and log showing failure will make user confused and easy to believe that the application is failed and needed to be submitted again. Like I said above, we return 2-tuple in which the last one (`FinalApplicationStatus `) is not used, the first one said application is finished so it is ok. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-199973290 sorry I didn't read the diff close enough thought you were just catching that type. I don't have an RM HA setup to quickly test it. I think there are cases this can return the wrong thing still (success when failure). Like if you control-c out of spark-shell during this same time period. Yes it probably doesn't make much difference, but at the same time I don't see it printing the exception to the log on shutdown as that big of a deal either. Is this actually causing you an issue or just annoying in the log? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/11692#discussion_r57051703
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -975,6 +975,10 @@ private[spark] class Client(
case e: ApplicationNotFoundException =>
logError(s"Application $appId not found.")
return (YarnApplicationState.KILLED,
FinalApplicationStatus.KILLED)
+ case e: Exception if (e.isInstanceOf[InterruptedException]
+|| e.getCause.isInstanceOf[InterruptedException]) =>
+logInfo("The reporter thread is interrupted, we assume app is
finished.")
+return (YarnApplicationState.FINISHED,
FinalApplicationStatus.SUCCEEDED)
--- End diff --
how about we change the status to be UNDEFINED.
Since we really don't know the status it seems like returning the undefined
in this case make more sense. Hopefully the user would then go look more at the
RM or spark job details.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-197342124 So inside of hadoop in the getApplicationReport call, it was in RetryInvocationHandler which was doing a sleep and got an interrupted exception. That ended up throwing a java.lang.reflect.UndeclaredThrowableException up to monitorApplication which is why it was handled by the NonFatal catch. I need to look at it a bit closer. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-198427974 so, how about it guys? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-198742134 have you tried to reproduce the scenaro and see what happend? The `UndeclaredThrowableException` will be caught by `e.getCause.isInstanceOf[InterruptedException]`, i think. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-198469043 I have had time to look further to see what we should be doing, but as I read the exception you listed above, the fix you are proposing here won't work. Its not getting an InterruptedException back to the monitorApplication routine, its getting an UndeclaredThrowableException. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-197188289 `monitorApplication` returns 2-tuple, in which FinalApplicationStatus is not used when sc stops normally. that's why it doesn't matter what status we set it to. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-197154066 I see your point, so the real issue should only be the log issue. But marking the state as `FINISHED` with `SUCCEED` should be open to question, since here we don't know the real state of application, if it is finished with failure, say job is aborted due to continuous stage failure, should we still mark this as `SUCCEED`? So here I'm conservative to this change, because: 1. This issue is happened rarely and does no harm to the result of application. 2. We cannot get the real exit state of application, so marking as `SUCCEED` should be open to question. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-197135484 the application is finished successfully(RM UI also show success state) but log shows it failed, that's the problem i think. yeah you're right sleep method can throw InterruptedException. this pr is trying to fix the problem we find in RM HA switching. what i am trying to say is that interrupting a monitor thread should not print the failed message in log. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-197129360 But from my understanding, this exception does no harm to your application, since your application is about to finish itself, also this may happen occasionally. Also does it relate to RM HA, from my understanding, this `InterruptedException` will be thrown in any case where the code run into `sleep`, no matter in HA or not. Is there any special thing for RM HA? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-197120333 @jerryshao yes, the problem is that client side's log will throw exception and show app is failed. more details are in log i pasted. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-197118357 @tgravescs I reproduce it and the error message like: >> 16/03/16 10:29:33 INFO YarnClientSchedulerBackend: Shutting down all executors 16/03/16 10:29:33 ERROR Client: Failed to contact YARN for application application_1457924833801_0003. java.lang.reflect.UndeclaredThrowableException at com.sun.proxy.$Proxy10.getApplicationReport(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:431) at org.apache.spark.deploy.yarn.Client.getApplicationReport(Client.scala:221) at org.apache.spark.deploy.yarn.Client.monitorApplication(Client.scala:882) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:144) Caused by: java.lang.InterruptedException: sleep interrupted at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:155) ... 5 more 16/03/16 10:29:33 INFO YarnClientSchedulerBackend: Asking each executor to shut down 16/03/16 10:29:33 ERROR YarnClientSchedulerBackend: Yarn application has already exited with state FAILED! 16/03/16 10:29:33 INFO SparkContext: SparkContext already stopped. 16/03/16 10:29:33 INFO YarnClientSchedulerBackend: Stopped There's no "Interrupting monitor thread" and the exception is UndeclaredThrowableException caused by InterruptedException. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-197106393 @WangTaoTheTonic , would you please elaborate specific problem you met when `InterruptedException` is thrown? From my understanding, it will only throw some exceptions mentioned that this application is failed, is that right? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-196908886 I've only observed exception caused by InterruptedException but not itself directly, thought it should be wrapped internally. The status in RM is ok as it is decided by ApplicationMaster not spark client. In my recall i didn't see the message "Interrupting monitor thread" but not 100% sure. I will try to reproduce it and confirm. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-196902816 Also is this just what is being printed by the client or is the YARN final status (in RM) actually failed? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-196901240 So you are saying that if spark context in yarn client mode is cleanly exiting while the RM is switching to the standby node, the call to getApplicationReport in Yarn can internally retry and sleep, since sc.stop() was called it ends up calling the scheduler backend stop interrupting the monitoring thread. The MonitorThread has a catch for interruptedException and should be printing an info message " Interrupting monitor thread", you are seeing this? Interrupted Exception is not a NonFatal error so it shouldn't be catching it: From scaladoc: Extractor of non-fatal Throwables. Will not match fatal errors like VirtualMachineError (for example, OutOfMemoryError and StackOverflowError, subclasses of VirtualMachineError), ThreadDeath, LinkageError, InterruptedException, ControlThrowable. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-196898154 for another concern about the final application status returned, we don't need too much worry as it is barely used by the codes who invoke this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-196889060 hi @tgravescs , it happened when sc stop normally in client mode. As sc.stop will stop dagscheduler -> stop taskscheduler -> stop scheduler backend -> interrupt the monitor thread, the monitor thread will enter into a retry logic in which there is a sleep intervals(which is not the sleep [here](https://github.com/WangTaoTheTonic/spark/blob/SPARK-13852/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L970)) waiting for RM's switching. The sleep methods will throw an InterruptedException when it is interrupted, so we need to catch it because it will log the application failed as treated as NonFatal(e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-196842093 Also switch sleep are you referring to because the place you put the try/catch isn't around theThread.sleep(interval) in monitorApplication, its only around getApplicationReport --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-196834213 @WangTaoTheTonic Can you please clarify exactly what is going on here? Does this happen in client or cluster mode or both? You are saying if YARN RM fails over from active to standby then our client logic can no longer connect to RM and gets an interrupted Exception? Who is interrupting the monitor thread?If its the spark context then how do you know its success and not failure? I'm not sure reporting success is the right thing to do here if we don't know the real status that is why I want to understand exactly what is going on. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-196756805 @vanzin @jerryshao does that sound right? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-196339172 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53072/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-196339704 OK by me --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-196339168 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-196338742 **[Test build #53072 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53072/consoleFull)** for PR 11692 at commit [`27203d8`](https://github.com/apache/spark/commit/27203d8a5d6a8a5ee8431a34d9e48aa0c6a15157). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-196326439 **[Test build #53072 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53072/consoleFull)** for PR 11692 at commit [`27203d8`](https://github.com/apache/spark/commit/27203d8a5d6a8a5ee8431a34d9e48aa0c6a15157). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-196325232 @srowen thanks for your comments. I've changed it, please check. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/11692#discussion_r56002509 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -976,6 +976,11 @@ private[spark] class Client( logError(s"Application $appId not found.") return (YarnApplicationState.KILLED, FinalApplicationStatus.KILLED) case NonFatal(e) => +if (e.isInstanceOf[InterruptedException] --- End diff -- Ah sure, you can almost combine the two conditions with `|` in Scala but not quite in this case, but you can at least do ... ``` case e: Exception if e.isInstanceOf[InterruptedException] ||e.getCause.isInstanceOf[InterruptedException] => ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user WangTaoTheTonic commented on a diff in the pull request: https://github.com/apache/spark/pull/11692#discussion_r55998595 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -976,6 +976,11 @@ private[spark] class Client( logError(s"Application $appId not found.") return (YarnApplicationState.KILLED, FinalApplicationStatus.KILLED) case NonFatal(e) => +if (e.isInstanceOf[InterruptedException] --- End diff -- then the code segments will be seperated into two parts. i am not sure it's better. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/11692#discussion_r55976051 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -976,6 +976,11 @@ private[spark] class Client( logError(s"Application $appId not found.") return (YarnApplicationState.KILLED, FinalApplicationStatus.KILLED) case NonFatal(e) => +if (e.isInstanceOf[InterruptedException] --- End diff -- Hm does this not work? ``` case e: InterruptedException => ... case e: Exception if e.getCause.isInstanceOf[InterruptedException] => ... ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user WangTaoTheTonic commented on a diff in the pull request: https://github.com/apache/spark/pull/11692#discussion_r55973772 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -976,6 +976,11 @@ private[spark] class Client( logError(s"Application $appId not found.") return (YarnApplicationState.KILLED, FinalApplicationStatus.KILLED) case NonFatal(e) => +if (e.isInstanceOf[InterruptedException] --- End diff -- we can only move InterruptedException to above but not exception cause by it. `just move InterruptedException` or `leave these two here`, which option do you think is better? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/11692#discussion_r55970249 --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala --- @@ -976,6 +976,11 @@ private[spark] class Client( logError(s"Application $appId not found.") return (YarnApplicationState.KILLED, FinalApplicationStatus.KILLED) case NonFatal(e) => +if (e.isInstanceOf[InterruptedException] --- End diff -- Shouldn't these just be additional case statements above? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-196209091 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53060/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-196209090 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-196208979 **[Test build #53060 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53060/consoleFull)** for PR 11692 at commit [`9d6de23`](https://github.com/apache/spark/commit/9d6de234b2c54c94b9fa261d78e5471facb3b6cd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13852][YARN]handle the InterruptedExcep...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11692#issuecomment-196202541 **[Test build #53060 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53060/consoleFull)** for PR 11692 at commit [`9d6de23`](https://github.com/apache/spark/commit/9d6de234b2c54c94b9fa261d78e5471facb3b6cd). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
