[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-56650781 @tgravescs Thanks for your notification. I found the issue which causes that scalastyle is not applied to yarn/common. I resolved this issue in #2520 . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-56652661 @tgravescs Sorry, I have something wrong. Please wait a little. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-56484136 test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-56484525 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20694/consoleFull) for PR 2198 at commit [`2696237`](https://github.com/apache/spark/commit/269623707ad207dd5dda1b5c5e1a8cb9cbe54bcd). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-56490061 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20694/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-56490053 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20694/consoleFull) for PR 2198 at commit [`2696237`](https://github.com/apache/spark/commit/269623707ad207dd5dda1b5c5e1a8cb9cbe54bcd). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `logInfo(Interrupting user class to stop.)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-56550485 This looks good. Thanks @sarutak ! I made one small change at checkin to use getInt instead of getLong on the config. It was also over 100 characters, not sure why scalastyle didn't pick it up. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2198 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user sarutak commented on a diff in the pull request: https://github.com/apache/spark/pull/2198#discussion_r17818868 --- Diff: docs/running-on-yarn.md --- @@ -50,6 +50,13 @@ Most of the configs are the same for Spark on YARN as for other deployment modes /td /tr tr + tdcodespark.yarn.scheduler.reporterThread.maxFailures/code/td + td5/td + td +The number of max failures in a row. Reporter thread which checks failed executor give up when the number of attempts reach this number. --- End diff -- O.K, I agree with you. I'll remove the description from the doc. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-56261930 I tested in the 2 way. 1st one is to hard code to raise Exception in the ApplicationMaster. 2nd one is using Debugger and inject Exception. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/2198#discussion_r17803484 --- Diff: docs/running-on-yarn.md --- @@ -50,6 +50,13 @@ Most of the configs are the same for Spark on YARN as for other deployment modes /td /tr tr + tdcodespark.yarn.scheduler.reporterThread.maxFailures/code/td + td5/td + td +The number of max failures in a row. Reporter thread which checks failed executor give up when the number of attempts reach this number. --- End diff -- I'm kind of on the edge of whether we should expose this to the normal user or not. Generally I wouldn't expect this to need to be configured so I'm thinking we should leave it out of the docs and if we find users are actually setting it then expose it. Do you have objections to that or specific use cases where this needs to be configured? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-56221633 How have you tested this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-55895905 @sarutak it is going to be another day or 2 before I can get time to look at this in detail. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-55904701 @tgravescs Thank you so much! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-55678640 Can anyone review this if you have time? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-55513253 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20291/consoleFull) for PR 2198 at commit [`f7538d4`](https://github.com/apache/spark/commit/f7538d4dcbb48f4d7895f9c70ffbd5c78ba16d31). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-55514281 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20291/consoleFull) for PR 2198 at commit [`f7538d4`](https://github.com/apache/spark/commit/f7538d4dcbb48f4d7895f9c70ffbd5c78ba16d31). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class RatingDeserializer(FramedSerializer):` * ` class Encoder[T : NativeType](columnType: NativeColumnType[T]) extends compression.Encoder[T] ` * ` class Encoder[T : NativeType](columnType: NativeColumnType[T]) extends compression.Encoder[T] ` * ` class Encoder[T : NativeType](columnType: NativeColumnType[T]) extends compression.Encoder[T] ` * ` class Encoder extends compression.Encoder[IntegerType.type] ` * ` class Decoder(buffer: ByteBuffer, columnType: NativeColumnType[IntegerType.type])` * ` class Encoder extends compression.Encoder[LongType.type] ` * ` class Decoder(buffer: ByteBuffer, columnType: NativeColumnType[LongType.type])` * `logInfo(Interrupting user class to stop.)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-55366135 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20207/consoleFull) for PR 2198 at commit [`0d138c6`](https://github.com/apache/spark/commit/0d138c66d4accdb452546c10c9c46766b8093bd8). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class CreateTableAsSelect(` * `case class CreateTableAsSelect(` * `logInfo(Interrupting user class to stop.)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-55359286 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20202/consoleFull) for PR 2198 at commit [`da31243`](https://github.com/apache/spark/commit/da312431d6921202027016341d323bfeb7837f1f). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-55361520 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20202/consoleFull) for PR 2198 at commit [`da31243`](https://github.com/apache/spark/commit/da312431d6921202027016341d323bfeb7837f1f). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `logInfo(Interrupting user class to stop.)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-55362811 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20207/consoleFull) for PR 2198 at commit [`0d138c6`](https://github.com/apache/spark/commit/0d138c66d4accdb452546c10c9c46766b8093bd8). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-54695040 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19849/consoleFull) for PR 2198 at commit [`13b2652`](https://github.com/apache/spark/commit/13b265268ad094aa18f5a33fd643e01d59da09fd). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-54698332 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19849/consoleFull) for PR 2198 at commit [`13b2652`](https://github.com/apache/spark/commit/13b265268ad094aa18f5a33fd643e01d59da09fd). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class AddJar(path: String) extends LeafNode with Command ` * `logInfo(Interrupting user class to stop.)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-54151738 Can you add some details about the cases you are seeing this happen in? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-54155111 As I mentioned in JIRA, if uncaught exception is thrown from Reporter thread, finish status is marked as SUCCEEDED. Futhermore, in YARN Cluster mode, if we don't notice Reporter thread is dead, we waits long time until timeout. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-54162723 sorry I should have clarified, I am just curious what particular exception you are seeing that is causing this? I agree the change is a good idea. Its just useful to know what people are seeing and also document in the jira/pr so that if someone goes to change this code in the future, to say handle particular types of exceptions, they know to handle atleast particular exceptions. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-54177193 Sorry too, I didn't understand what you intended. Actually, I didn't see exception causing Reporter thread dead but unexpected exception like OOM or any other exceptions due to communication error with RM during resource negotiation caused by temporary traffic jam can be thrown, I think. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-54025962 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19552/consoleFull) for PR 2198 at commit [`c081f8e`](https://github.com/apache/spark/commit/c081f8e8e81f72643870ff850e19ed7adb17e4e0). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-54031723 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19552/consoleFull) for PR 2198 at commit [`c081f8e`](https://github.com/apache/spark/commit/c081f8e8e81f72643870ff850e19ed7adb17e4e0). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `logInfo(Interrupting user class to stop.)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-53889989 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19477/consoleFull) for PR 2198 at commit [`a6982ad`](https://github.com/apache/spark/commit/a6982ad2c9ff2ba63492ea254b8ef0030d1ccf27). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-53890795 This pr may also fix [SPARK-3293] https://issues.apache.org/jira/browse/SPARK-3293 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-53891161 Thanks for you information @scwf ! I'll take a look at the issue and also try to resolve that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-53897498 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19477/consoleFull) for PR 2198 at commit [`a6982ad`](https://github.com/apache/spark/commit/a6982ad2c9ff2ba63492ea254b8ef0030d1ccf27). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` class ReporterThreadUncaughtExceptionHandler extends Thread.UncaughtExceptionHandler ` * ` logInfo(Interrupting user class to stop.)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2198#discussion_r16906758
--- Diff:
yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
---
@@ -232,6 +232,24 @@ private[spark] class ApplicationMaster(args:
ApplicationMasterArguments,
t
}
+ class ReporterThreadUncaughtExceptionHandler extends
Thread.UncaughtExceptionHandler {
+override def uncaughtException(th: Thread, e: Throwable) {
+ logError(Uncaught exception, e)
+ finish(FinalApplicationStatus.FAILED, Uncaught exception was thrown
from Reporter thread)
--- End diff --
nit: end sentence with a period.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2198#discussion_r16906842
--- Diff:
yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
---
@@ -232,6 +232,24 @@ private[spark] class ApplicationMaster(args:
ApplicationMasterArguments,
t
}
+ class ReporterThreadUncaughtExceptionHandler extends
Thread.UncaughtExceptionHandler {
--- End diff --
I think this would be cleaner if you just put the exception handling code
in the reporter thread code itself (in `launchReporterThread()`).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-53898896 It feels a little odd to give up in the first exception; this might cause some recoverable network hiccup to fail the whole application. Doing something smarter (like cannot hearbeat after x attemps or something) might be a better alternative. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
