zl created ZEPPELIN-4988:
----------------------------
Summary: exception when running zeppelin on the machine where k8s
is installed
Key: ZEPPELIN-4988
URL: https://issues.apache.org/jira/browse/ZEPPELIN-4988
Project: Zeppelin
Issue Type: Bug
Components: interpreter-launcher
Reporter: zl
Attachments: 252170F0.PNG
zeppelin is deployed on a machine which is also a k8s node（zeppelin is running
outside of k8s）, and when I run paragraph, the following exception happens:
---
INFO [2020-08-07 10:53:10,776] (\{SchedulerFactory2}
PluginManager.java[loadInterpreterLauncher]:141) - Loading Interpreter Launcher
Plugin: K8sStandardInterpreterLauncher
WARN [2020-08-07 10:53:10,777] (\{SchedulerFactory2}
PluginManager.java[loadInterpreterLauncher]:152) - **Fail to instantiate
InterpreterLauncher from classpath directly:K8sStandardInterpreterLauncher**
java.lang.ClassNotFoundException:
org.apache.zeppelin.interpreter.launcher.K8sStandardInterpreterLauncher
at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:315)
at
org.apache.zeppelin.plugin.PluginManager.loadInterpreterLauncher(PluginManager.java:146)
at
org.apache.zeppelin.interpreter.InterpreterSetting.createLauncher(InterpreterSetting.java:308)
at
org.apache.zeppelin.interpreter.InterpreterSetting.createInterpreterProcess(InterpreterSetting.java:842)
at
org.apache.zeppelin.interpreter.ManagedInterpreterGroup.getOrCreateInterpreterProcess(ManagedInterpreterGroup.java:63)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:110)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:163)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:131)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:298)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:431)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:74)
at org.apache.zeppelin.scheduler.Job.run(Job.java:172)
at
org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:130)
at
org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:159)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
WARN [2020-08-07 10:53:10,782] (\{SchedulerFactory2}
PluginManager.java[loadInterpreterLauncher]:164) - **Fail to instantiate
Launcher K8sStandardInterpreterLauncher from plugin pluginDir:
/mnt/disk1/zl/app/zeppelin-0.9.0-preview1-bin-all/plugins**
java.lang.reflect.InvocationTargetException
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at
org.apache.zeppelin.plugin.PluginManager.loadInterpreterLauncher(PluginManager.java:161)
at
org.apache.zeppelin.interpreter.InterpreterSetting.createLauncher(InterpreterSetting.java:308)
at
org.apache.zeppelin.interpreter.InterpreterSetting.createInterpreterProcess(InterpreterSetting.java:842)
at
org.apache.zeppelin.interpreter.ManagedInterpreterGroup.getOrCreateInterpreterProcess(ManagedInterpreterGroup.java:63)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:110)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:163)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:131)
at
org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:298)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:431)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:74)
at org.apache.zeppelin.scheduler.Job.run(Job.java:172)
at
org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:130)
at
org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:159)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
**Caused by: java.nio.file.NoSuchFileException:
/var/run/secrets/kubernetes.io/serviceaccount/namespace**
at
java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at
java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at
java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:215)
at java.base/java.nio.file.Files.newByteChannel(Files.java:370)
at java.base/java.nio.file.Files.newByteChannel(Files.java:421)
at java.base/java.nio.file.Files.readAllBytes(Files.java:3205)
at
org.apache.zeppelin.interpreter.launcher.K8sStandardInterpreterLauncher.readFile(K8sStandardInterpreterLauncher.java:174)
at
org.apache.zeppelin.interpreter.launcher.K8sStandardInterpreterLauncher.getNamespace(K8sStandardInterpreterLauncher.java:86)
at
org.apache.zeppelin.interpreter.launcher.K8sStandardInterpreterLauncher.<init>(K8sStandardInterpreterLauncher.java:51)
... 23 more
---
after some research, I found that there is a problem when checking whether
zeppelin is running on kubernetes. As shown in the following code snippet,
!252170F0.PNG!
if */var/run/secrets/kubernetes.io* exists, we confirm that zeppelin is running
on k8s, however, */var/run/secrets/kubernetes.io* not only exists in k8s pod
but also in k8s node. so if zeppelin is deployed on a machine which is also a
k8s node（zeppelin is running outside of k8s）, we will still confirm that
zeppelin is running on k8s, and try to load **K8sStandardInterpreterLauncher**,
which is wrong.
instead of ***/var/run/secrets/kubernetes.io***, we could use
***/var/run/secrets/kubernetes.io/serviceaccount/namespace*** to check whether
zeppelin is running on kubernetes, because
***/var/run/secrets/kubernetes.io/serviceaccount/namespace*** only exists in
kubernetes pod.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
