[jira] [Commented] (LIVY-636) Unable to create interactive session with additional JAR in spark.driver.extraClassPath
[
https://issues.apache.org/jira/browse/LIVY-636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17141791#comment-17141791
]
yuuya commented on LIVY-636:
Hi
Here's the solution
Set config spark`s scala-library-\{version}.jar and livy`s
repl_\{version}-jars/*
{code:java}
livy.repl.jars=/usr/lib/spark/jars/scala-library-2.11.12.jar,/usr/local/lib/livy/repl_2.11-jars/commons-codec-1.9.jar,/usr/local/lib/livy/repl_2.11-jars/livy-core_2.11-0.7.0-incubating.jar,/usr/local/lib/livy/repl_2.11-jars/livy-repl_2.11-0.7.0-incubating.jar{code}
What kind of policy should you handle as a project in the future?
> Unable to create interactive session with additional JAR in
> spark.driver.extraClassPath
> ---
>
> Key: LIVY-636
> URL: https://issues.apache.org/jira/browse/LIVY-636
> Project: Livy
> Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Ishita Virmani
>Priority: Major
> Attachments: applicationmaster.log, container.log, stacktrace.txt,
> test.png
>
>
> Command Run: c{{url -H "Content-Type: application/json" -X POST -d
> '\{"kind":"pyspark","conf":{"spark.driver.extraClassPath":"/data/XXX-0.0.1-SNAPSHOT.jar"}}'
> -i http:///session}}
> {{The above command fails to create a Spark Session on YARN with Null pointer
> exception. Stack trace for the same has been attached along-with.}}
> The JAR file here is present on local driver Path. Also tried using HDFS path
> in the following manner
> {{hdfs://:/data/XXX-0.0.1-SNAPSHOT.jar}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (LIVY-636) Unable to create interactive session with additional JAR in spark.driver.extraClassPath
[
https://issues.apache.org/jira/browse/LIVY-636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140797#comment-17140797
]
locona commented on LIVY-636:
-
Hi [~glennthomas] cc [~ishitavirmani]
Is there any progress here?
It occurs in the following cases
{code:java}
// livy.conf
livy.spark.master=yarn
livy.spark.deploy-mode = client
{code}
Is there anything related to the following PR?
[https://github.com/cloudera/livy/pull/312/files]
> Unable to create interactive session with additional JAR in
> spark.driver.extraClassPath
> ---
>
> Key: LIVY-636
> URL: https://issues.apache.org/jira/browse/LIVY-636
> Project: Livy
> Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Ishita Virmani
>Priority: Major
> Attachments: applicationmaster.log, container.log, stacktrace.txt,
> test.png
>
>
> Command Run: c{{url -H "Content-Type: application/json" -X POST -d
> '\{"kind":"pyspark","conf":{"spark.driver.extraClassPath":"/data/XXX-0.0.1-SNAPSHOT.jar"}}'
> -i http:///session}}
> {{The above command fails to create a Spark Session on YARN with Null pointer
> exception. Stack trace for the same has been attached along-with.}}
> The JAR file here is present on local driver Path. Also tried using HDFS path
> in the following manner
> {{hdfs://:/data/XXX-0.0.1-SNAPSHOT.jar}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (LIVY-636) Unable to create interactive session with additional JAR in spark.driver.extraClassPath
[
https://issues.apache.org/jira/browse/LIVY-636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066455#comment-17066455
]
manaya sasaki commented on LIVY-636:
Hi Glenn Thomas
Thank you for your reply.
When I tried the assembly with includeScala=false, I could not exclude the
class file of scala-java8-compat. The basic library is configured exactly like
foo.bar.baz, but scala-java8-compat is configured just like scala.compat.java8.
How did you deal with this pattern when it happened?
> Unable to create interactive session with additional JAR in
> spark.driver.extraClassPath
> ---
>
> Key: LIVY-636
> URL: https://issues.apache.org/jira/browse/LIVY-636
> Project: Livy
> Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Ishita Virmani
>Priority: Major
> Attachments: applicationmaster.log, container.log, stacktrace.txt,
> test.png
>
>
> Command Run: c{{url -H "Content-Type: application/json" -X POST -d
> '\{"kind":"pyspark","conf":{"spark.driver.extraClassPath":"/data/XXX-0.0.1-SNAPSHOT.jar"}}'
> -i http:///session}}
> {{The above command fails to create a Spark Session on YARN with Null pointer
> exception. Stack trace for the same has been attached along-with.}}
> The JAR file here is present on local driver Path. Also tried using HDFS path
> in the following manner
> {{hdfs://:/data/XXX-0.0.1-SNAPSHOT.jar}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (LIVY-636) Unable to create interactive session with additional JAR in spark.driver.extraClassPath
[
https://issues.apache.org/jira/browse/LIVY-636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066309#comment-17066309
]
Glenn Thomas commented on LIVY-636:
---
We faced this issue as well, and can confirm that removing scala dependencies
from our jar solved the issue.
One interesting thing is that this jar works:
{code:java}
package foo.bar.baz
class MyClass {
def mytest(x:String) {
println("I am " + x)
}
}
{code}
While this jar does not work:
{code:java}
package scala.foo.bar.baz
class MyClass {
def mytest(x:String) {
println("I am " + x)
}
}
{code}
So perhaps the issue is something to do with "scala" being in package names
within the jar, rather than an issue with scala itself?
> Unable to create interactive session with additional JAR in
> spark.driver.extraClassPath
> ---
>
> Key: LIVY-636
> URL: https://issues.apache.org/jira/browse/LIVY-636
> Project: Livy
> Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Ishita Virmani
>Priority: Major
> Attachments: applicationmaster.log, container.log, stacktrace.txt,
> test.png
>
>
> Command Run: c{{url -H "Content-Type: application/json" -X POST -d
> '\{"kind":"pyspark","conf":{"spark.driver.extraClassPath":"/data/XXX-0.0.1-SNAPSHOT.jar"}}'
> -i http:///session}}
> {{The above command fails to create a Spark Session on YARN with Null pointer
> exception. Stack trace for the same has been attached along-with.}}
> The JAR file here is present on local driver Path. Also tried using HDFS path
> in the following manner
> {{hdfs://:/data/XXX-0.0.1-SNAPSHOT.jar}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (LIVY-636) Unable to create interactive session with additional JAR in spark.driver.extraClassPath
[
https://issues.apache.org/jira/browse/LIVY-636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065424#comment-17065424
]
manaya sasaki commented on LIVY-636:
Hi [~Cerberuser]
What if I want to use a package like scala-java8-compat_2.11-0.9.0.jar?
This jar extends scala.
> Unable to create interactive session with additional JAR in
> spark.driver.extraClassPath
> ---
>
> Key: LIVY-636
> URL: https://issues.apache.org/jira/browse/LIVY-636
> Project: Livy
> Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Ishita Virmani
>Priority: Major
> Attachments: applicationmaster.log, container.log, stacktrace.txt,
> test.png
>
>
> Command Run: c{{url -H "Content-Type: application/json" -X POST -d
> '\{"kind":"pyspark","conf":{"spark.driver.extraClassPath":"/data/XXX-0.0.1-SNAPSHOT.jar"}}'
> -i http:///session}}
> {{The above command fails to create a Spark Session on YARN with Null pointer
> exception. Stack trace for the same has been attached along-with.}}
> The JAR file here is present on local driver Path. Also tried using HDFS path
> in the following manner
> {{hdfs://:/data/XXX-0.0.1-SNAPSHOT.jar}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (LIVY-636) Unable to create interactive session with additional JAR in spark.driver.extraClassPath
[
https://issues.apache.org/jira/browse/LIVY-636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929104#comment-16929104
]
Konstantin commented on LIVY-636:
-
If you're using sbt-assembly for making this JAR, there is an option:
{code:java}
assemblyOption in assembly := (assemblyOption in
assembly).value.copy(includeScala = false)
{code}
Place this in build.sbt, and all should be fine.
> Unable to create interactive session with additional JAR in
> spark.driver.extraClassPath
> ---
>
> Key: LIVY-636
> URL: https://issues.apache.org/jira/browse/LIVY-636
> Project: Livy
> Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Ishita Virmani
>Priority: Major
> Attachments: applicationmaster.log, container.log, stacktrace.txt,
> test.png
>
>
> Command Run: c{{url -H "Content-Type: application/json" -X POST -d
> '\{"kind":"pyspark","conf":{"spark.driver.extraClassPath":"/data/XXX-0.0.1-SNAPSHOT.jar"}}'
> -i http:///session}}
> {{The above command fails to create a Spark Session on YARN with Null pointer
> exception. Stack trace for the same has been attached along-with.}}
> The JAR file here is present on local driver Path. Also tried using HDFS path
> in the following manner
> {{hdfs://:/data/XXX-0.0.1-SNAPSHOT.jar}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (LIVY-636) Unable to create interactive session with additional JAR in spark.driver.extraClassPath
[
https://issues.apache.org/jira/browse/LIVY-636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929079#comment-16929079
]
Ishita Virmani commented on LIVY-636:
-
Hi [~Cerberuser],
Your solution seem to have worked. Thanks
But right now I manually removed Scala folder from my JAR. But is there any
other way to remove Scala, since my library is coded in scala, and would thus
include Scala in the JAR by default.
> Unable to create interactive session with additional JAR in
> spark.driver.extraClassPath
> ---
>
> Key: LIVY-636
> URL: https://issues.apache.org/jira/browse/LIVY-636
> Project: Livy
> Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Ishita Virmani
>Priority: Major
> Attachments: applicationmaster.log, container.log, stacktrace.txt,
> test.png
>
>
> Command Run: c{{url -H "Content-Type: application/json" -X POST -d
> '\{"kind":"pyspark","conf":{"spark.driver.extraClassPath":"/data/XXX-0.0.1-SNAPSHOT.jar"}}'
> -i http:///session}}
> {{The above command fails to create a Spark Session on YARN with Null pointer
> exception. Stack trace for the same has been attached along-with.}}
> The JAR file here is present on local driver Path. Also tried using HDFS path
> in the following manner
> {{hdfs://:/data/XXX-0.0.1-SNAPSHOT.jar}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (LIVY-636) Unable to create interactive session with additional JAR in spark.driver.extraClassPath
[
https://issues.apache.org/jira/browse/LIVY-636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923973#comment-16923973
]
Konstantin commented on LIVY-636:
-
Update: the crash in my comment was due to the fact that I've included Scala in
the JAR being added. Once I rebuilt it without Scala libraries, all worked
smoothly.
To understand this, I've checked through the code in the stacktrace. It appears
that if the exception is thrown inside the [{{"run"}} function of
RSCDriver|https://github.com/apache/incubator-livy/blob/master/rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java#L321],
including the initialization process (that was the case here, since
"scala.reflect" will misbehave when there are two Scala libraries on the
classpath). If the error is thrown during initialization, the "jc" variable
remains "null". Then, after the exception is thrown, the "finally" block runs,
calling for shutdown. If there were any queued job, driver will try to cancel
it. But, since there is no context yet for the jobs, job.cancel (called
[here|https://github.com/apache/incubator-livy/blob/master/rsc/src/main/java/org/apache/livy/rsc/driver/RSCDriver.java#L127])
throws the NPE, which interrupts the program, hiding the original exception.
> Unable to create interactive session with additional JAR in
> spark.driver.extraClassPath
> ---
>
> Key: LIVY-636
> URL: https://issues.apache.org/jira/browse/LIVY-636
> Project: Livy
> Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Ishita Virmani
>Priority: Major
> Attachments: applicationmaster.log, container.log, stacktrace.txt,
> test.png
>
>
> Command Run: c{{url -H "Content-Type: application/json" -X POST -d
> '\{"kind":"pyspark","conf":{"spark.driver.extraClassPath":"/data/XXX-0.0.1-SNAPSHOT.jar"}}'
> -i http:///session}}
> {{The above command fails to create a Spark Session on YARN with Null pointer
> exception. Stack trace for the same has been attached along-with.}}
> The JAR file here is present on local driver Path. Also tried using HDFS path
> in the following manner
> {{hdfs://:/data/XXX-0.0.1-SNAPSHOT.jar}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (LIVY-636) Unable to create interactive session with additional JAR in spark.driver.extraClassPath
[
https://issues.apache.org/jira/browse/LIVY-636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922323#comment-16922323
]
Konstantin commented on LIVY-636:
-
Exactly same exception when trying to add jar from HDFS through the "jars"
field; possibly for the same reason. Tried to add the main class to the jar (it
was meant to be library-only, so there was no main class presented initially) -
exception goes away, but imports from this jar are still impossible.
> Unable to create interactive session with additional JAR in
> spark.driver.extraClassPath
> ---
>
> Key: LIVY-636
> URL: https://issues.apache.org/jira/browse/LIVY-636
> Project: Livy
> Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Ishita Virmani
>Priority: Major
> Attachments: applicationmaster.log, container.log, stacktrace.txt,
> test.png
>
>
> Command Run: c{{url -H "Content-Type: application/json" -X POST -d
> '\{"kind":"pyspark","conf":{"spark.driver.extraClassPath":"/data/XXX-0.0.1-SNAPSHOT.jar"}}'
> -i http:///session}}
> {{The above command fails to create a Spark Session on YARN with Null pointer
> exception. Stack trace for the same has been attached along-with.}}
> The JAR file here is present on local driver Path. Also tried using HDFS path
> in the following manner
> {{hdfs://:/data/XXX-0.0.1-SNAPSHOT.jar}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (LIVY-636) Unable to create interactive session with additional JAR in spark.driver.extraClassPath
[
https://issues.apache.org/jira/browse/LIVY-636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918305#comment-16918305
]
runzhiwang commented on LIVY-636:
-
Hi [~ishitavirmani]
Do you have any updates ?
> Unable to create interactive session with additional JAR in
> spark.driver.extraClassPath
> ---
>
> Key: LIVY-636
> URL: https://issues.apache.org/jira/browse/LIVY-636
> Project: Livy
> Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Ishita Virmani
>Priority: Major
> Attachments: applicationmaster.log, container.log, stacktrace.txt,
> test.png
>
>
> Command Run: c{{url -H "Content-Type: application/json" -X POST -d
> '\{"kind":"pyspark","conf":{"spark.driver.extraClassPath":"/data/XXX-0.0.1-SNAPSHOT.jar"}}'
> -i http:///session}}
> {{The above command fails to create a Spark Session on YARN with Null pointer
> exception. Stack trace for the same has been attached along-with.}}
> The JAR file here is present on local driver Path. Also tried using HDFS path
> in the following manner
> {{hdfs://:/data/XXX-0.0.1-SNAPSHOT.jar}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (LIVY-636) Unable to create interactive session with additional JAR in spark.driver.extraClassPath
[
https://issues.apache.org/jira/browse/LIVY-636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913827#comment-16913827
]
jiewang commented on LIVY-636:
--
Hi [~ishitavirmani]
The applicationmaster.log and container.log has been attached in the
Attachments.
> Unable to create interactive session with additional JAR in
> spark.driver.extraClassPath
> ---
>
> Key: LIVY-636
> URL: https://issues.apache.org/jira/browse/LIVY-636
> Project: Livy
> Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Ishita Virmani
>Priority: Major
> Attachments: applicationmaster.log, container.log, stacktrace.txt,
> test.png
>
>
> Command Run: c{{url -H "Content-Type: application/json" -X POST -d
> '\{"kind":"pyspark","conf":{"spark.driver.extraClassPath":"/data/XXX-0.0.1-SNAPSHOT.jar"}}'
> -i http:///session}}
> {{The above command fails to create a Spark Session on YARN with Null pointer
> exception. Stack trace for the same has been attached along-with.}}
> The JAR file here is present on local driver Path. Also tried using HDFS path
> in the following manner
> {{hdfs://:/data/XXX-0.0.1-SNAPSHOT.jar}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (LIVY-636) Unable to create interactive session with additional JAR in spark.driver.extraClassPath
[
https://issues.apache.org/jira/browse/LIVY-636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913049#comment-16913049
]
Ishita Virmani commented on LIVY-636:
-
Hi [~runzhiwang]
Then the versions do not seem to be the issue. Can you please share the logs of
the application created on YARN. From the screenshot above, looks like SC was
shutdown.
> Unable to create interactive session with additional JAR in
> spark.driver.extraClassPath
> ---
>
> Key: LIVY-636
> URL: https://issues.apache.org/jira/browse/LIVY-636
> Project: Livy
> Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Ishita Virmani
>Priority: Major
> Attachments: stacktrace.txt, test.png
>
>
> Command Run: c{{url -H "Content-Type: application/json" -X POST -d
> '\{"kind":"pyspark","conf":{"spark.driver.extraClassPath":"/data/XXX-0.0.1-SNAPSHOT.jar"}}'
> -i http:///session}}
> {{The above command fails to create a Spark Session on YARN with Null pointer
> exception. Stack trace for the same has been attached along-with.}}
> The JAR file here is present on local driver Path. Also tried using HDFS path
> in the following manner
> {{hdfs://:/data/XXX-0.0.1-SNAPSHOT.jar}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (LIVY-636) Unable to create interactive session with additional JAR in spark.driver.extraClassPath
[
https://issues.apache.org/jira/browse/LIVY-636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911090#comment-16911090
]
jiewang commented on LIVY-636:
--
Hi [~ishitavirmani]
Scala Version: 2.11.12
Spark Version: 2.4.1
Livy: apache-livy-0.6.0
> Unable to create interactive session with additional JAR in
> spark.driver.extraClassPath
> ---
>
> Key: LIVY-636
> URL: https://issues.apache.org/jira/browse/LIVY-636
> Project: Livy
> Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Ishita Virmani
>Priority: Major
> Attachments: stacktrace.txt, test.png
>
>
> Command Run: c{{url -H "Content-Type: application/json" -X POST -d
> '\{"kind":"pyspark","conf":{"spark.driver.extraClassPath":"/data/XXX-0.0.1-SNAPSHOT.jar"}}'
> -i http:///session}}
> {{The above command fails to create a Spark Session on YARN with Null pointer
> exception. Stack trace for the same has been attached along-with.}}
> The JAR file here is present on local driver Path. Also tried using HDFS path
> in the following manner
> {{hdfs://:/data/XXX-0.0.1-SNAPSHOT.jar}}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (LIVY-636) Unable to create interactive session with additional JAR in spark.driver.extraClassPath
[
https://issues.apache.org/jira/browse/LIVY-636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908064#comment-16908064
]
jiewang commented on LIVY-636:
--
hi, [~ishitavirmani]. I tried your command, and it seems everything is ok.
!test.png!
> Unable to create interactive session with additional JAR in
> spark.driver.extraClassPath
> ---
>
> Key: LIVY-636
> URL: https://issues.apache.org/jira/browse/LIVY-636
> Project: Livy
> Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Ishita Virmani
>Priority: Major
> Attachments: stacktrace.txt, test.png
>
>
> Command Run: c{{url -H "Content-Type: application/json" -X POST -d
> '\{"kind":"pyspark","conf":{"spark.driver.extraClassPath":"/data/XXX-0.0.1-SNAPSHOT.jar"}}'
> -i http:///session}}
> {{The above command fails to create a Spark Session on YARN with Null pointer
> exception. Stack trace for the same has been attached along-with.}}
> The JAR file here is present on local driver Path. Also tried using HDFS path
> in the following manner
> {{hdfs://:/data/XXX-0.0.1-SNAPSHOT.jar}}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
