[jira] [Resolved] (LIVY-715) The configuration in the livy.conf.template is inconsistent with LivyConf.scala
[ https://issues.apache.org/jira/browse/LIVY-715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao resolved LIVY-715. -- Fix Version/s: 0.7.0 Resolution: Fixed Issue resolved by pull request 261 https://github.com/apache/incubator-livy/pull/261 > The configuration in the livy.conf.template is inconsistent with > LivyConf.scala > --- > > Key: LIVY-715 > URL: https://issues.apache.org/jira/browse/LIVY-715 > Project: Livy > Issue Type: Bug > Components: Docs >Affects Versions: 0.6.0 >Reporter: mingchao zhao >Assignee: mingchao zhao >Priority: Major > Fix For: 0.7.0 > > Time Spent: 40m > Remaining Estimate: 0h > > When I test livy impersonation found that, in livy.conf.template the > value of livy.impersonation.enabled is true. So I thought impersonation was > enabled by default. > However, impersonation was not turned on when we test. I found that the > real configuration in LivyConf. scala is false. This can mislead users. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (LIVY-717) Specify explicit ZooKeeper version in maven
Mate Szalay-Beko created LIVY-717: - Summary: Specify explicit ZooKeeper version in maven Key: LIVY-717 URL: https://issues.apache.org/jira/browse/LIVY-717 Project: Livy Issue Type: Improvement Reporter: Mate Szalay-Beko I want to test Livy in a cluster where we have a new ZooKeeper 3.5 deployed. Currently in Livy we use Curator 2.7.1, which only supports ZooKeeper 3.4. I want to use Curator 4.0.2, as it supports ZooKeeper 3.5. (Also Hadoop trunk was updated recently to use Curator 4.0.2 and ZooKeeper 3.5.6, see [HADOOP-16579|https://issues.apache.org/jira/browse/HADOOP-16579].) When I tried to build Livy with Curator 4.0.2, I found out that Curator 4.x brings a specific 3.5 beta ZooKeeper version as compile time dependency, which is different from the stable 3.5.6 ZooKeeper version used in Hadoop. Also The Hadoop version used in Livy brings different ZooKeeper version. In this patch I don't I want to change the Curator or ZooKeeper versions, but I want to enable the users to specify an explicit ZooKeeper version for Livy by a maven parameter. (the Curator version is already added as maven parameter) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-712) EMR 5.23/5.27 - Livy does not recognise that Spark job failed
[
https://issues.apache.org/jira/browse/LIVY-712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16979272#comment-16979272
]
Yiheng Wang commented on LIVY-712:
--
Can you provide a way to reproduce the issue?
> EMR 5.23/5.27 - Livy does not recognise that Spark job failed
> -
>
> Key: LIVY-712
> URL: https://issues.apache.org/jira/browse/LIVY-712
> Project: Livy
> Issue Type: Bug
> Components: API
>Affects Versions: 0.5.0, 0.6.0
> Environment: AWS EMR 5.23/5.27, Scala
>Reporter: Michal Sankot
>Priority: Major
> Labels: EMR, api, spark
>
> We've upgraded from AWS EMR 5.13 -> 5.23 (Livy 0.4.0 -> 0.5.0, Spark 2.3.0 ->
> 2.4.0) and an issue appears that when there is an exception thrown during
> Spark job execution, Spark shuts down as if there was no problem and job
> appears as Completed in EMR. So we're not notified when system crashes. The
> same problem appears in EMR 5.27 (Livy 0.6.0, Spark 2.4.4).
> Is it something with Spark? Or a known issue with Livy?
> In Livy logs I see that spark-submit exists with error code 1
> {quote}{{05:34:59 WARN BatchSession$: spark-submit exited with code 1}}
> {quote}
> And then Livy API states that batch state is
> {quote}{{"state": "success"}}
> {quote}
> How can it be made work again?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (LIVY-712) EMR 5.23/5.27 - Livy does not recognise that Spark job failed
[
https://issues.apache.org/jira/browse/LIVY-712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16979271#comment-16979271
]
Yiheng Wang commented on LIVY-712:
--
This code is changed in this patch:
https://github.com/apache/incubator-livy/commit/ca4cad22968e1a2f88fa0ec262c1088812e3d251
[~jshao] Any suggestion about this?
> EMR 5.23/5.27 - Livy does not recognise that Spark job failed
> -
>
> Key: LIVY-712
> URL: https://issues.apache.org/jira/browse/LIVY-712
> Project: Livy
> Issue Type: Bug
> Components: API
>Affects Versions: 0.5.0, 0.6.0
> Environment: AWS EMR 5.23/5.27, Scala
>Reporter: Michal Sankot
>Priority: Major
> Labels: EMR, api, spark
>
> We've upgraded from AWS EMR 5.13 -> 5.23 (Livy 0.4.0 -> 0.5.0, Spark 2.3.0 ->
> 2.4.0) and an issue appears that when there is an exception thrown during
> Spark job execution, Spark shuts down as if there was no problem and job
> appears as Completed in EMR. So we're not notified when system crashes. The
> same problem appears in EMR 5.27 (Livy 0.6.0, Spark 2.4.4).
> Is it something with Spark? Or a known issue with Livy?
> In Livy logs I see that spark-submit exists with error code 1
> {quote}{{05:34:59 WARN BatchSession$: spark-submit exited with code 1}}
> {quote}
> And then Livy API states that batch state is
> {quote}{{"state": "success"}}
> {quote}
> How can it be made work again?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
