Hi,
We are experiencing a potential bug with how Livy reports the state of a
batch job. We have infrastructure that polls the state of a batch
application and we've noticed that when a fatal exception occurs within the
application and there is a non-zero exit code in spark-submit, we see that
Livy still returns a 'success' state for that batch job.
We confirm the exit code and the exception through the Livy server logs.
So for example... after an exit code of 1, we would see a response from
the API like...
{
"id": 277,
"name": null,
"state": "success",
"appId": "application_***",
"appInfo": {
"driverLogUrl": null,
"sparkUiUrl": "*********"
},
"log": [
"20/05/07 10:25:04 INFO SparkContext: Successfully stopped SparkContext",
"20/05/07 10:25:04 INFO SparkContext: SparkContext already stopped."
,
......
"20/05/07 10:25:04 INFO ShutdownHookManager: Deleting directory
/mnt/tmp/spark-62127d15-3ffd-4ec9-a8c0-c45a26707c3f"
,
"20/05/07 10:25:04 INFO ShutdownHookManager: Deleting directory
/mnt/tmp/spark-67271639-4c3e-4a13-aff3-082e5705cbed"
,
"\nstderr: ",
"\nYARN Diagnostics: "
]
}
Can you confirm if this is an known issue, expected behaviour or a
misconfiguration? We are running these spark applications on AWS EMR. We
haven't applied any custom configuration to Livy.
Java Spark version 2.4.0
AWS EMR version: emr-5.24.0
Thanks,
Colin
--
This message contains proprietary information from Equifax which may be
confidential. If you are not an intended recipient, please refrain from any
disclosure, copying, distribution or use of this information and note that
such actions are prohibited. If you have received this transmission in
error, please notify by e-mail postmas...@equifax.com
<mailto:postmas...@equifax.com>. Equifax® is a registered trademark of
Equifax Inc. All rights reserved.
