[
https://issues.apache.org/jira/browse/LIVY-602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gyorgy Gal updated LIVY-602:
----------------------------
Fix Version/s: 0.10.0
(was: 0.9.0)
This issue has been moved to the 0.10.0 release as part of a bulk update. If
you feel this is moved out inappropriately, feel free to provide justification
and reset the Fix Version to 0.9.0.
> An error during 'magic' (table / json) encoding of a result
> -----------------------------------------------------------
>
> Key: LIVY-602
> URL: https://issues.apache.org/jira/browse/LIVY-602
> Project: Livy
> Issue Type: Bug
> Components: Interpreter
> Affects Versions: 0.6.0
> Reporter: Maksim Ochenashko
> Priority: Major
> Fix For: 0.10.0
>
>
> The error occurs during the execution of the statement:
> {code:java}
> val x = List((1, "a"), (3, "b")).toDF("number", "key")
> %table x //or %json x{code}
> *Environment:*
> Docker
> Livy: 0.6-incubating
> Spark: 2.4.0
> Hadoop: 3.0
> Dockerfile:
> {code:java}
> FROM gettyimages/spark:2.4.0-hadoop-3.0
> RUN apt-get update && \
> apt-get install -y wget && \
> rm -rf /var/lib/apt/lists/*
> RUN mkdir -p /apps && \
> cd /apps && \
> wget
> http://apache.cp.if.ua/incubator/livy/0.6.0-incubating/apache-livy-0.6.0-incubating-bin.zip
> && \
> unzip apache-livy-0.6.0-incubating-bin.zip -d /apps && \
> rm -f apache-livy-0.6.0-incubating-bin.zip && \
> mkdir /apps/apache-livy-0.6.0-incubating-bin/logs
> # Expose port
> EXPOSE 8998
> CMD ["/apps/apache-livy-0.6.0-incubating-bin/bin/livy-server"]{code}
> *Reproducing steps (bash):*
> {code:java}
> docker build . -t livy-spark
> docker run -d -p 8998:8998 livy-spark
> curl localhost:8998/sessions -X POST --data '{"kind":"spark"}' -H
> "Content-Type: application/json"
> curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type:
> application/json' -d '{"code":"val x = List((1, \"a\"), (3,
> \"b\")).toDF(\"number\", \"key\") \r\n%table x"}'
> curl localhost:8998/sessions/0/statements/0
> // response
> "id":0,"code":"val x = List((1, \"a\"), (3, \"b\")).toDF(\"number\", \"key\")
> \n%table
> x","state":"available","output":{"status":"error","execution_count":0,"ename":"Internal
> Error:
> java.lang.NoClassDefFoundError","evalue":"scala/tools/scalap/scalax/rules/scalasig/Type","traceback":[]},"progress":1.0}%
> curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type:
> application/json' -d '{"code":"val x = List((1, \"a\"), (3,
> \"b\")).toDF(\"number\", \"key\") \r\n%json x"}'
> curl localhost:8998/sessions/0/statements/1
> // response
> {"id":1,"code":"val x = List((1, \"a\"), (3, \"b\")).toDF(\"number\",
> \"key\") \n%json
> x","state":"available","output":{"status":"error","execution_count":1,"ename":"ValueError","evalue":"Failed
> to convert value into a JSON value","traceback":[]},"progress":1.0}%{code}
> *The error:*
> Livy cannot load the *scala/tools/scalap/scalax/rules/scalasig/Type* class.
> *The line causing the error:*
> [https://github.com/apache/incubator-livy/blob/v0.6.0-incubating/repl/src/main/scala/org/apache/livy/repl/AbstractSparkInterpreter.scala#L186]
> *Stacktrace:*
> [https://gist.github.com/iRevive/977a67940b05ab2faaa3f3be5b426038]
> *Possible root cause:*
> Looks like there is a problem with the classpath.
> Currently, Livy has *scalap-2.11.0.jar* in the classpath. I tried to replace
> it with *scalap-2.11.12.jar* but it still doesn't work.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
