GitHub user thomastechs opened a pull request:
https://github.com/apache/spark/pull/16743
[SPARK-19379][CORE] SparkAppHandle.getState not registering FAILED state
upon Spark app failure in Local mode
## What changes were proposed in this pull request?
As per the bug, the state corresponding to app failures are not getting
registered with call back to the launcher.So, the already available isFailed()
check of TaskState.scala, is utilized to send the explicit status of
SparkAppHandle.State.FAILED back to the launcherBackend, which under the hood
updates the status for (LOST == state) || (FAILED == state).
The already available TaskState.isFinished() is used to set the value
SparkAppHandle.State.FINISHED, as under the hood, it considers FINISHED,
FAILED, KILLED, LOST as part of FINISHED state.
Since the isFailed() is checked first, the isFinished is checked in the
else if condition to get the mutually exclusive scenarios.
## How was this patch tested?
This patch was tested manually. The tests included, by simply submitting
local jobs, the states are set correctly. Also, the tests included with some
fail prone codes executed and tested the status set as FAILED. It looks
difficult to write testcase to create an error prone scnario code and check the
status.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/thomastechs/spark thomastechs-19379
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16743.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16743
----
commit 52ac0083f14a2454b1109b5f535c05ac7fd5a2b4
Author: thomastechs <[email protected]>
Date: 2016-02-22T17:16:15Z
Adding new test suite for [SPARK-12941][SQL]
Added this new integration suite for creating the docker based test suite
for the bug fix in SPARK-12941
commit b711688be233ab78fd92f42b22bcfb852a0bb1b9
Author: thomastechs <[email protected]>
Date: 2016-02-22T17:20:19Z
Update pom.xml for [SPARK-12941] docker test suite addition
Updated this pom.xml with the ojdbc jar related dependency to test the
OracleIntegrationSuite related to SPARK-12941 bug fix
commit c2c6d14353cab4e72b66f9125f964b215a977446
Author: thomastechs <[email protected]>
Date: 2016-02-22T17:46:51Z
Update DockerJDBCIntegrationSuite.scala for [SPARK-12941] docker test suite
addition
Updated DockerJDBCIntegrationSuite.scala with the time out interval, as it
is required to be atleast 200 sec for Oracle instance to be up
commit 54b2d4e466e309c8c12e58cefcdf31ca690183f1
Author: thomastechs <[email protected]>
Date: 2016-02-22T17:51:21Z
Update OracleDialect.scala for [SPARK-12941] docker test suite addition
Updated OracleDialect.scala for [SPARK-12941] for the fix of adding a data
type mapping to Oracle
commit bde06fa2189dfe9f9484d86a942eb6cfa50ffad8
Author: thomastechs <[email protected]>
Date: 2016-02-22T17:57:59Z
Update OracleIntegrationSuite.scala for [SPARK-12941] docker test suite
addition
Updated OracleIntegrationSuite.scala for [SPARK-12941] with Oracle Docker
test cases
commit 0861d72c3b3ad377807edb96f4016231291e792a
Author: thomastechs <[email protected]>
Date: 2016-02-23T02:57:32Z
Update OracleIntegrationSuite.scala for [SPARK-12941] Style check fixes and
ignore test case
Updated the file with style check fixes and ignore test cases, as the ojdbc
jar is not present in the maven.
commit ca7ef4e3de03f70c7f37ede0075e31daf17ff4cc
Author: thomastechs <[email protected]>
Date: 2016-02-23T02:59:23Z
Update OracleIntegrationSuite.scala for [SPARK-12941] Style check fixes
Style check fixes
commit 28a9e9fc7793f0ae7dfc0fc0fae17d6ef8acd94e
Author: thomastechs <[email protected]>
Date: 2016-02-23T03:14:09Z
Update OracleIntegrationSuite.scala for [SPARK-12941] Style check fixes
Updated the order of imports
commit 87e3d7f4de872852a3f16bf6a31761a8cfd1d46b
Author: thomastechs <[email protected]>
Date: 2016-02-23T06:35:28Z
Update pom.xml
Updating this pom.xml to comment the changes related oracle ojdbc jar
dependency as the maven repository does not contain this.
commit 7b12418ca7f75bcc17941bb713286573b7209225
Author: thomastechs <[email protected]>
Date: 2016-02-23T09:43:37Z
Update OracleIntegrationSuite.scala
Updated the OracleIntegrationSuite.scala with the doc and override method
commit 250591dfb7c63f446cfb09702587642da2ec026d
Author: thomastechs <[email protected]>
Date: 2016-02-23T09:48:02Z
Update OracleIntegrationSuite.scala
updated doc
commit d77d5852b06c4003661757207868c693723210bd
Author: thomastechs <[email protected]>
Date: 2016-02-23T11:13:24Z
Update OracleIntegrationSuite.scala for [SPARK-12941] Style check fixes
Updated style check fixes
commit 7fcb16cac7e0f32c74db1c5d9e3d4968c9da183a
Author: thomastechs <[email protected]>
Date: 2016-02-24T02:38:26Z
Update OracleIntegrationSuite.scala for [SPARK-12941] Updated Comments
Updated OracleIntegrationSuite.scala for [SPARK-12941] for the edits in the
comments section
commit b3de7f493cc14e258cd699f395bd1241912beb4b
Author: thomastechs <[email protected]>
Date: 2016-02-24T02:38:36Z
Update pom.xml for [SPARK-12941] added comments
Added comments on why the oracle dependency was commented out
commit 345555f777ab09cd6426685938e5b5c935749472
Author: thomastechs <[email protected]>
Date: 2016-02-24T06:11:17Z
Update DockerJDBCIntegrationSuite.scala
Updated DockerJDBCIntegrationSuite.scala- Reverted the timeout and interval
parameter to 60 and 1. This would be required to be increased for Oracle based
testing.
commit 3f0f3b8105b358f580b6225f9cf1592b44e17450
Author: thomastechs <[email protected]>
Date: 2016-02-24T06:11:33Z
Updated OracleIntegrationSuite.scala
Updated the comments about the time interval change required in
DockerJDBCIntegrationSuite.scala when testing with Oracle tests
commit 5683dcabfd283f6008f4c1294ff41a07e3f40b8f
Author: THOMAS SEBASTIAN <[email protected]>
Date: 2016-03-03T03:55:31Z
Merge remote-tracking branch 'upstream/master'
commit 154e10444b4796fd52ff3e0ddf7b6b7a86cc4a87
Author: THOMAS SEBASTIAN <[email protected]>
Date: 2016-11-24T18:52:43Z
Merge remote-tracking branch 'upstream/master'
commit e6d431789b7f390612b82ba8305439de3f047de1
Author: thomastechs <[email protected]>
Date: 2017-01-30T17:30:25Z
Updated LocalSchedulerBackend.scala for fix of JIRA:19379
The setState methods of the launcher backend is being invoked for the
status updates related to Failed and Finished. The isFailed Test explicity
captures the failure scenario and updates the status. and n else if is given to
maintain the as is scenario of choosing isFinished conditions as FINISHED.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
