Dmitry Kravchuk created ZEPPELIN-5289: -----------------------------------------
Summary: Pyspark interpreter fails according to spark .jar accessing Key: ZEPPELIN-5289 URL: https://issues.apache.org/jira/browse/ZEPPELIN-5289 Project: Zeppelin Issue Type: Improvement Components: Interpreters, pySpark, zeppelin-interpreter Affects Versions: 0.9.0 Environment: * zeppelin 0.9.0 * hadoop 3.2 * spark 3.1.1 Reporter: Dmitry Kravchuk Fix For: 0.9.1 Attachments: interpreter 1.png, interpreter 2.png, zeppelin 0.9.0 spark 3.1 interpreter error.png Cannot run pyspark interpreter with spark 3.1.1 using zeppeling 0.9.0 !zeppelin 0.9.0 spark 3.1 interpreter error.png! Output: {code:java} org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: Fail to open SparkInterpreter at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:836) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:744) at org.apache.zeppelin.scheduler.Job.run(Job.java:172) at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:132) at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: Fail to open SparkInterpreter at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76) at org.apache.zeppelin.interpreter.Interpreter.getInterpreterInTheSameSessionByClassName(Interpreter.java:355) at org.apache.zeppelin.interpreter.Interpreter.getInterpreterInTheSameSessionByClassName(Interpreter.java:366) at org.apache.zeppelin.spark.PySparkInterpreter.open(PySparkInterpreter.java:90) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70) ... 8 more Caused by: org.apache.zeppelin.interpreter.InterpreterException: Fail to open SparkInterpreter at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:122) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70) ... 12 more Caused by: scala.reflect.internal.FatalError: Error accessing /opt/spark31/spark-3.1.1-bin-hadoop3.2/jars/accessors-smart-1.2.jar at scala.tools.nsc.classpath.AggregateClassPath.$anonfun$list$3(AggregateClassPath.scala:99) at scala.collection.Iterator.foreach(Iterator.scala:941) at scala.collection.Iterator.foreach$(Iterator.scala:941) at scala.collection.AbstractIterator.foreach(Iterator.scala:1429) at scala.collection.IterableLike.foreach(IterableLike.scala:74) at scala.collection.IterableLike.foreach$(IterableLike.scala:73) at scala.collection.AbstractIterable.foreach(Iterable.scala:56) at scala.tools.nsc.classpath.AggregateClassPath.list(AggregateClassPath.scala:87) at scala.tools.nsc.util.ClassPath.list(ClassPath.scala:36) at scala.tools.nsc.util.ClassPath.list$(ClassPath.scala:36) at scala.tools.nsc.classpath.AggregateClassPath.list(AggregateClassPath.scala:30) at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader.doComplete(SymbolLoaders.scala:284) at scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.complete(SymbolLoaders.scala:230) at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1542) at scala.reflect.internal.Mirrors$RootsBase.init(Mirrors.scala:257) at scala.tools.nsc.Global.rootMirror$lzycompute(Global.scala:74) at scala.tools.nsc.Global.rootMirror(Global.scala:72) at scala.tools.nsc.Global.rootMirror(Global.scala:44) at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass$lzycompute(Definitions.scala:295) at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass(Definitions.scala:295) at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1480) at scala.tools.nsc.Global$Run.<init>(Global.scala:1199) at scala.tools.nsc.interpreter.IMain._initialize(IMain.scala:132) at scala.tools.nsc.interpreter.IMain.initializeSynchronous(IMain.scala:154) at org.apache.zeppelin.spark.SparkScala212Interpreter.open(SparkScala212Interpreter.scala:86) at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:106) ... 13 more Caused by: java.io.IOException: Error accessing /opt/spark31/spark-3.1.1-bin-hadoop3.2/jars/accessors-smart-1.2.jar at scala.reflect.io.FileZipArchive.scala$reflect$io$FileZipArchive$$openZipFile(ZipArchive.scala:141) at scala.reflect.io.FileZipArchive.root$lzycompute(ZipArchive.scala:179) at scala.reflect.io.FileZipArchive.root(ZipArchive.scala:176) at scala.reflect.io.FileZipArchive.allDirs$lzycompute(ZipArchive.scala:212) at scala.reflect.io.FileZipArchive.allDirs(ZipArchive.scala:212) at scala.tools.nsc.classpath.ZipArchiveFileLookup.findDirEntry(ZipArchiveFileLookup.scala:76) at scala.tools.nsc.classpath.ZipArchiveFileLookup.list(ZipArchiveFileLookup.scala:63) at scala.tools.nsc.classpath.ZipArchiveFileLookup.list$(ZipArchiveFileLookup.scala:62) at scala.tools.nsc.classpath.ZipAndJarClassPathFactory$ZipArchiveClassPath.list(ZipAndJarFileLookupFactory.scala:56) at scala.tools.nsc.classpath.AggregateClassPath.$anonfun$list$3(AggregateClassPath.scala:91) ... 38 more Caused by: java.util.zip.ZipException: error in opening zip file at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.<init>(ZipFile.java:219) at java.util.zip.ZipFile.<init>(ZipFile.java:149) at java.util.zip.ZipFile.<init>(ZipFile.java:163) at scala.reflect.io.FileZipArchive.scala$reflect$io$FileZipArchive$$openZipFile(ZipArchive.scala:138) ... 47 more org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: Fail to open SparkInterpreter at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:836) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:744) at org.apache.zeppelin.scheduler.Job.run(Job.java:172) at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:132) at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: Fail to open SparkInterpreter at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76) at org.apache.zeppelin.interpreter.Interpreter.getInterpreterInTheSameSessionByClassName(Interpreter.java:355) at org.apache.zeppelin.interpreter.Interpreter.getInterpreterInTheSameSessionByClassName(Interpreter.java:366) at org.apache.zeppelin.spark.PySparkInterpreter.open(PySparkInterpreter.java:90) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70) ... 8 more Caused by: org.apache.zeppelin.interpreter.InterpreterException: Fail to open SparkInterpreter at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:122) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70) ... 12 more Caused by: scala.reflect.internal.FatalError: Error accessing /opt/spark31/spark-3.1.1-bin-hadoop3.2/jars/accessors-smart-1.2.jar at scala.tools.nsc.classpath.AggregateClassPath.$anonfun$list$3(AggregateClassPath.scala:99) at scala.collection.Iterator.foreach(Iterator.scala:941) at scala.collection.Iterator.foreach$(Iterator.scala:941) at scala.collection.AbstractIterator.foreach(Iterator.scala:1429) at scala.collection.IterableLike.foreach(IterableLike.scala:74) at scala.collection.IterableLike.foreach$(IterableLike.scala:73) at scala.collection.AbstractIterable.foreach(Iterable.scala:56) at scala.tools.nsc.classpath.AggregateClassPath.list(AggregateClassPath.scala:87) at scala.tools.nsc.util.ClassPath.list(ClassPath.scala:36) at scala.tools.nsc.util.ClassPath.list$(ClassPath.scala:36) at scala.tools.nsc.classpath.AggregateClassPath.list(AggregateClassPath.scala:30) at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader.doComplete(SymbolLoaders.scala:284) at scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.complete(SymbolLoaders.scala:230) at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1542) at scala.reflect.internal.Mirrors$RootsBase.init(Mirrors.scala:257) at scala.tools.nsc.Global.rootMirror$lzycompute(Global.scala:74) at scala.tools.nsc.Global.rootMirror(Global.scala:72) at scala.tools.nsc.Global.rootMirror(Global.scala:44) at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass$lzycompute(Definitions.scala:295) at scala.reflect.internal.Definitions$DefinitionsClass.ObjectClass(Definitions.scala:295) at scala.reflect.internal.Definitions$DefinitionsClass.init(Definitions.scala:1480) at scala.tools.nsc.Global$Run.<init>(Global.scala:1199) at scala.tools.nsc.interpreter.IMain._initialize(IMain.scala:132) at scala.tools.nsc.interpreter.IMain.initializeSynchronous(IMain.scala:154) at org.apache.zeppelin.spark.SparkScala212Interpreter.open(SparkScala212Interpreter.scala:86) at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:106) ... 13 more Caused by: java.io.IOException: Error accessing /opt/spark31/spark-3.1.1-bin-hadoop3.2/jars/accessors-smart-1.2.jar at scala.reflect.io.FileZipArchive.scala$reflect$io$FileZipArchive$$openZipFile(ZipArchive.scala:141) at scala.reflect.io.FileZipArchive.root$lzycompute(ZipArchive.scala:179) at scala.reflect.io.FileZipArchive.root(ZipArchive.scala:176) at scala.reflect.io.FileZipArchive.allDirs$lzycompute(ZipArchive.scala:212) at scala.reflect.io.FileZipArchive.allDirs(ZipArchive.scala:212) at scala.tools.nsc.classpath.ZipArchiveFileLookup.findDirEntry(ZipArchiveFileLookup.scala:76) at scala.tools.nsc.classpath.ZipArchiveFileLookup.list(ZipArchiveFileLookup.scala:63) at scala.tools.nsc.classpath.ZipArchiveFileLookup.list$(ZipArchiveFileLookup.scala:62) at scala.tools.nsc.classpath.ZipAndJarClassPathFactory$ZipArchiveClassPath.list(ZipAndJarFileLookupFactory.scala:56) at scala.tools.nsc.classpath.AggregateClassPath.$anonfun$list$3(AggregateClassPath.scala:91) ... 38 more Caused by: java.util.zip.ZipException: error in opening zip file at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.<init>(ZipFile.java:219) at java.util.zip.ZipFile.<init>(ZipFile.java:149) at java.util.zip.ZipFile.<init>(ZipFile.java:163) at scala.reflect.io.FileZipArchive.scala$reflect$io$FileZipArchive$$openZipFile(ZipArchive.scala:138) ... 47 more {code} Interpreter settings: !interpreter 1.png! !interpreter 2.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)