[
https://issues.apache.org/jira/browse/LIVY-712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16972668#comment-16972668
]
Michal Sankot commented on LIVY-712:
------------------------------------
As I search though the code i see a change in SparkYarnApp.scala line 295 on
between 0.4.0 and 0.5.0 (same in 0.6.0).
0.4.0:
{{ } catch {}}
{{ case e: InterruptedException =>}}
{{ yarnDiagnostics = ArrayBuffer("Session stopped by user.")}}
{{ changeState(SparkApp.State.KILLED)}}
{{ case e: Throwable =>}}
{{ error(s"Error whiling refreshing YARN state: $e")}}
{{ yarnDiagnostics = ArrayBuffer(e.toString, e.getStackTrace().mkString("
"))}}
{{ changeState(SparkApp.State.FAILED)}}
}
0.5.0/0.6.0:
{{ } catch {}}
{{ case _: InterruptedException =>}}
{{ yarnDiagnostics = ArrayBuffer("Session stopped by user.")}}
{{ changeState(SparkApp.State.KILLED)}}
{{ case NonFatal(e) =>}}
{{ error(s"Error whiling refreshing YARN state", e)}}
{{ yarnDiagnostics = ArrayBuffer(e.getMessage)}}
{{ changeState(SparkApp.State.FAILED)}}
}
So it seems that in 0.5.0+ all Fatal exceptions are ignored and don't make App
go into FAILED state. That looks like a bug. Is it so?
> EMR 5.23/5.27 - Livy does not recognise that Spark job failed
> -------------------------------------------------------------
>
> Key: LIVY-712
> URL: https://issues.apache.org/jira/browse/LIVY-712
> Project: Livy
> Issue Type: Bug
> Components: API
> Affects Versions: 0.5.0, 0.6.0
> Environment: AWS EMR 5.23/5.27, Scala
> Reporter: Michal Sankot
> Priority: Major
> Labels: EMR, api, spark
>
> We've upgraded from AWS EMR 5.13 -> 5.23 (Livy 0.4.0 -> 0.5.0, Spark 2.3.0 ->
> 2.4.0) and an issue appears that when there is an exception thrown during
> Spark job execution, Spark shuts down as if there was no problem and job
> appears as Completed in EMR. So we're not notified when system crashes. The
> same problem appears in EMR 5.27 (Livy 0.6.0, Spark 2.4.4).
> Is it something with Spark? Or a known issue with Livy?
> In Livy logs I see that spark-submit exists with error code 1
> {quote}{{05:34:59 WARN BatchSession$: spark-submit exited with code 1}}
> {quote}
> And then Livy API states that batch state is
> {quote}{{"state": "success"}}
> {quote}
> How can it be made work again?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
