Maksim Ochenashko created LIVY-602:
--------------------------------------
Summary: An error during 'magic' (table / json) encoding of a
result
Key: LIVY-602
URL: https://issues.apache.org/jira/browse/LIVY-602
Project: Livy
Issue Type: Bug
Components: Interpreter
Affects Versions: 0.6.0
Reporter: Maksim Ochenashko
The error occurs during the execution of the statement:
{code:java}
val x = List((1, "a"), (3, "b")).toDF("number", "key")
%table x //or %json x{code}
*Environment:*
Docker
Livy: 0.6-incubating
Spark: 2.4.0
Hadoop: 3.0
Dockerfile:
{code:java}
FROM gettyimages/spark:2.4.0-hadoop-3.0
RUN apt-get update && \
apt-get install -y wget && \
rm -rf /var/lib/apt/lists/*
RUN mkdir -p /apps && \
cd /apps && \
wget
http://apache.cp.if.ua/incubator/livy/0.6.0-incubating/apache-livy-0.6.0-incubating-bin.zip
&& \
unzip apache-livy-0.6.0-incubating-bin.zip -d /apps && \
rm -f apache-livy-0.6.0-incubating-bin.zip && \
mkdir /apps/apache-livy-0.6.0-incubating-bin/logs
# Expose port
EXPOSE 8998
CMD ["/apps/apache-livy-0.6.0-incubating-bin/bin/livy-server"]{code}
*Reproducing steps (bash):*
{code:java}
docker build . -t livy-spark
docker run -d -p 8998:8998 livy-spark
curl localhost:8998/sessions -X POST --data '{"kind":"spark"}' -H
"Content-Type: application/json"
curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type:
application/json' -d '{"code":"val x = List((1, \"a\"), (3,
\"b\")).toDF(\"number\", \"key\") \r\n%table x"}'
curl localhost:8998/sessions/0/statements/0
// response
"id":0,"code":"val x = List((1, \"a\"), (3, \"b\")).toDF(\"number\", \"key\")
\n%table
x","state":"available","output":{"status":"error","execution_count":0,"ename":"Internal
Error:
java.lang.NoClassDefFoundError","evalue":"scala/tools/scalap/scalax/rules/scalasig/Type","traceback":[]},"progress":1.0}%
curl localhost:8998/sessions/0/statements -X POST -H 'Content-Type:
application/json' -d '{"code":"val x = List((1, \"a\"), (3,
\"b\")).toDF(\"number\", \"key\") \r\n%json x"}'
curl localhost:8998/sessions/0/statements/1
// response
{"id":1,"code":"val x = List((1, \"a\"), (3, \"b\")).toDF(\"number\", \"key\")
\n%json
x","state":"available","output":{"status":"error","execution_count":1,"ename":"ValueError","evalue":"Failed
to convert value into a JSON value","traceback":[]},"progress":1.0}%{code}
*The error:*
Livy cannot load the *scala/tools/scalap/scalax/rules/scalasig/Type* class.
*The line causing the error:*
[https://github.com/apache/incubator-livy/blob/v0.6.0-incubating/repl/src/main/scala/org/apache/livy/repl/AbstractSparkInterpreter.scala#L186]
*Stacktrace:*
[https://gist.github.com/iRevive/977a67940b05ab2faaa3f3be5b426038]
*Possible root cause:*
Looks like there is a problem with the classpath.
Currently, Livy has *scalap-2.11.0.jar* in the classpath. I tried to replace it
with *scalap-2.11.12.jar* but it still doesn't work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
