Hi Colin,
Can you tell me more specifically what the fatal error in spark-submit was?
I'd try to reproduce.
As far as I know, the exit code of spark-submit is not really checked in
version 0.6.0 (fixed in LIVY-586). Nonetheless, when spark-submit fails,
the app should transition to "failed" state after
livy.server.yarn.app-lookup-timeout.
Thanks,
Andras
On Thu, May 7, 2020 at 12:41 PM Colin Kirk <colin.k...@equifax.com> wrote:
> Hi,
>
> We are experiencing a potential bug with how Livy reports the state of a
> batch job. We have infrastructure that polls the state of a batch
> application and we've noticed that when a fatal exception occurs within the
> application and there is a non-zero exit code in spark-submit, we see that
> Livy still returns a 'success' state for that batch job.
>
> We confirm the exit code and the exception through the Livy server logs.
>
> So for example... after an exit code of 1, we would see a response from
> the API like...
>
> {
> "id": 277,
> "name": null,
> "state": "success",
> "appId": "application_***",
> "appInfo": {
> "driverLogUrl": null,
> "sparkUiUrl": "*********"
> },
> "log": [
>
> "20/05/07 10:25:04 INFO SparkContext: Successfully stopped SparkContext",
>
> "20/05/07 10:25:04 INFO SparkContext: SparkContext already stopped.",
> ......
>
> "20/05/07 10:25:04 INFO ShutdownHookManager: Deleting directory
> /mnt/tmp/spark-62127d15-3ffd-4ec9-a8c0-c45a26707c3f"
> ,
>
> "20/05/07 10:25:04 INFO ShutdownHookManager: Deleting directory
> /mnt/tmp/spark-67271639-4c3e-4a13-aff3-082e5705cbed"
> ,
> "\nstderr: ",
> "\nYARN Diagnostics: "
> ]
> }
>
> Can you confirm if this is an known issue, expected behaviour or a
> misconfiguration? We are running these spark applications on AWS EMR. We
> haven't applied any custom configuration to Livy.
>
> Java Spark version 2.4.0
> AWS EMR version: emr-5.24.0
>
> Thanks,
>
> Colin
>
>
> This message contains proprietary information from Equifax which may be
> confidential. If you are not an intended recipient, please refrain from any
> disclosure, copying, distribution or use of this information and note that
> such actions are prohibited. If you have received this transmission in
> error, please notify by e-mail postmas...@equifax.com. Equifax® is a
> registered trademark of Equifax Inc. All rights reserved.
>
>
