Gerald G Probst created ZEPPELIN-5120:
-----------------------------------------
Summary: Unable to open Spark Interpreter after upgrading from
0.8.1 to 0.9.0-preview2
Key: ZEPPELIN-5120
URL: https://issues.apache.org/jira/browse/ZEPPELIN-5120
Project: Zeppelin
Issue Type: Bug
Components: Interpreters, spark
Affects Versions: 0.9.0
Environment: zeppelin-env.sh:
{code:java}
export MASTER="yarn"
export SPARK_HOME="/usr/lib/spark"
export SPARK_SUBMIT_OPTIONS="--driver-memory 12G --executor-memory 8G"
export AWS_ACCESS_KEY_ID=<key>
export AWS_SECRET_ACCESS_KEY=<secret>
export HADOOP_HOME="/usr/lib/hadoop"
export USE_HADOOP=true
export HADOOP_CONF_DIR="/etc/hadoop/conf"{code}
spark: 2.4.0
scala: 2.11.12
java: 1.8.01_232
OS: Debian 4.9.144-3.1
Reporter: Gerald G Probst
We were successfully running Zeppelin 0.8.1 on a Google Cloud Dataproc cluster
and we needed to upgrade as we were encounter in a bug that was resolved in
newer versions. After upgrading to 0.9.0-preview2 we can no longer connect to
the Spark Interpreter.
Any code returns the following :
{noformat}
org.apache.zeppelin.interpreter.InterpreterException:
org.apache.zeppelin.interpreter.InterpreterException: Fail to open
SparkInterpreter at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:760)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:668)
at org.apache.zeppelin.scheduler.Job.run(Job.java:172) at
org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:130)
at
org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:39)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) Caused by:
org.apache.zeppelin.interpreter.InterpreterException: Fail to open
SparkInterpreter at
org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:122) at
org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
... 8 more Caused by: java.lang.reflect.InvocationTargetException at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.zeppelin.spark.BaseSparkScalaInterpreter.spark2CreateContext(BaseSparkScalaInterpreter.scala:292)
at
org.apache.zeppelin.spark.BaseSparkScalaInterpreter.createSparkContext(BaseSparkScalaInterpreter.scala:230)
at
org.apache.zeppelin.spark.SparkScala211Interpreter.open(SparkScala211Interpreter.scala:98)
at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:106)
... 9 more Caused by: java.lang.VerifyError: Bad return type Exception Details:
Location:
org/apache/hadoop/hdfs/DFSClient.getQuotaUsage(Ljava/lang/String;)Lorg/apache/hadoop/fs/QuotaUsage;
@157: areturn Reason: Type 'org/apache/hadoop/fs/ContentSummary' (current
frame, stack[0]) is not assignable to 'org/apache/hadoop/fs/QuotaUsage' (from
method signature) Current Frame: bci: @157 flags: { } locals: {
'org/apache/hadoop/hdfs/DFSClient', 'java/lang/String',
'org/apache/hadoop/ipc/RemoteException', 'java/io/IOException' } stack: {
'org/apache/hadoop/fs/ContentSummary' } Bytecode: 0x0000000: 2ab6 00df 2a13
01ff 2bb6 00b4 4d01 4e2a 0x0000010: b400 422b b902 0002 003a 042c c600 1d2d
0x0000020: c600 152c b600 b6a7 0012 3a05 2d19 05b6 0x0000030: 00b8 a700 072c
b600 b619 04b0 3a04 1904 0x0000040: 4e19 04bf 3a06 2cc6 001d 2dc6 0015 2cb6
0x0000050: 00b6 a700 123a 072d 1907 b600 b8a7 0007 0x0000060: 2cb6 00b6 1906
bf4d 2c07 bd00 d159 0312 0x0000070: d353 5904 12dd 5359 0512 de53 5906 1302
0x0000080: 0153 b600 d44e 2dc1 0201 9900 14b2 0023 0x0000090: 1302 02b9 002b
0200 2a2b b602 03b0 2dbf 0x00000a0: Exception Handler Table: bci [35, 39] =>
handler: 42 bci [15, 27] => handler: 60 bci [15, 27] => handler: 68 bci [78,
82] => handler: 85 bci [60, 70] => handler: 68 bci [4, 57] => handler: 103 bci
[60, 103] => handler: 103 Stackmap Table:
full_frame(@42,{Object[#736],Object[#759],Object[#814],Object[#784],Object[#1217]},{Object[#784]})
same_frame(@53) same_frame(@57)
full_frame(@60,{Object[#736],Object[#759],Object[#814],Object[#784]},{Object[#784]})
same_locals_1_stack_item_frame(@68,Object[#784])
full_frame(@85,{Object[#736],Object[#759],Object[#814],Object[#784],Top,Top,Object[#784]},{Object[#784]})
same_frame(@96) same_frame(@100)
full_frame(@103,{Object[#736],Object[#759]},{Object[#839]})
append_frame(@158,Object[#839],Object[#799]) at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:164)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669) at
org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94) at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703) at
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685) at
org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373) at
org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1866) at
org.apache.spark.scheduler.EventLoggingListener.<init>(EventLoggingListener.scala:71)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:528) at
org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2530) at
org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
at
org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
at scala.Option.getOrElse(Option.scala:121) at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
... 17 more{noformat}
We can connect to spark via the spark-shell and run commands successfully.
This is blocking our current development and have been unable to find a way to
resolve this issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
