[zeppelin] branch master updated: [ZEPPELIN-5644] Reduce redundancy during the interpreter execution process (#4294)
This is an automated email from the ASF dual-hosted git repository.
zjffdu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
The following commit(s) were added to refs/heads/master by this push:
new 7aad7d9 [ZEPPELIN-5644] Reduce redundancy during the interpreter
execution process (#4294)
7aad7d9 is described below
commit 7aad7d92d255cf19a56272b2d3ac40f47373a7ec
Author: Leomax_Sun <282130...@qq.com>
AuthorDate: Fri Mar 18 07:58:25 2022 +0800
[ZEPPELIN-5644] Reduce redundancy during the interpreter execution process
(#4294)
---
.../apache/zeppelin/spark/SparkInterpreter.java| 57 +-
.../zeppelin/spark/SparkInterpreterTest.java | 5 +-
.../zeppelin/spark/SparkScala211Interpreter.scala | 1 +
.../zeppelin/spark/SparkScala212Interpreter.scala | 1 +
.../zeppelin/spark/BaseSparkScalaInterpreter.scala | 3 +-
.../remote/RemoteInterpreterServer.java| 19 +++-
.../remote/RemoteInterpreterServerTest.java| 32 +++-
7 files changed, 91 insertions(+), 27 deletions(-)
diff --git
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
index fef0998..b13d67f 100644
---
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
+++
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
@@ -66,6 +66,7 @@ public class SparkInterpreter extends AbstractInterpreter {
}
private static AtomicInteger SESSION_NUM = new AtomicInteger(0);
+ private static Class innerInterpreterClazz;
private AbstractSparkScalaInterpreter innerInterpreter;
private Map innerInterpreterClassMap = new HashMap<>();
private SparkContext sc;
@@ -151,38 +152,45 @@ public class SparkInterpreter extends AbstractInterpreter
{
*/
private AbstractSparkScalaInterpreter loadSparkScalaInterpreter(SparkConf
conf) throws Exception {
scalaVersion = extractScalaVersion(conf);
-ClassLoader scalaInterpreterClassLoader =
Thread.currentThread().getContextClassLoader();
-
-String zeppelinHome = System.getenv("ZEPPELIN_HOME");
-if (zeppelinHome != null) {
- // ZEPPELIN_HOME is null in yarn-cluster mode, load it directly via
current ClassLoader.
- // otherwise, load from the specific folder
ZEPPELIN_HOME/interpreter/spark/scala-
-
- File scalaJarFolder = new File(zeppelinHome +
"/interpreter/spark/scala-" + scalaVersion);
- List urls = new ArrayList<>();
- for (File file : scalaJarFolder.listFiles()) {
-LOGGER.debug("Add file " + file.getAbsolutePath() + " to classpath of
spark scala interpreter: "
-+ scalaJarFolder);
-urls.add(file.toURI().toURL());
+// Make sure the innerInterpreter Class is loaded only once into JVM
+// Use double lock to ensure thread safety
+if (innerInterpreterClazz == null) {
+ synchronized (SparkInterpreter.class) {
+if (innerInterpreterClazz == null) {
+ LOGGER.debug("innerInterpreterClazz is null, thread:{}",
Thread.currentThread().getName());
+ ClassLoader scalaInterpreterClassLoader =
Thread.currentThread().getContextClassLoader();
+ String zeppelinHome = System.getenv("ZEPPELIN_HOME");
+ if (zeppelinHome != null) {
+// ZEPPELIN_HOME is null in yarn-cluster mode, load it directly
via current ClassLoader.
+// otherwise, load from the specific folder
ZEPPELIN_HOME/interpreter/spark/scala-
+File scalaJarFolder = new File(zeppelinHome +
"/interpreter/spark/scala-" + scalaVersion);
+List urls = new ArrayList<>();
+for (File file : scalaJarFolder.listFiles()) {
+ LOGGER.debug("Add file " + file.getAbsolutePath() + " to
classpath of spark scala interpreter: "
+ + scalaJarFolder);
+ urls.add(file.toURI().toURL());
+}
+scalaInterpreterClassLoader = new URLClassLoader(urls.toArray(new
URL[0]),
+Thread.currentThread().getContextClassLoader());
+ }
+ String innerIntpClassName =
innerInterpreterClassMap.get(scalaVersion);
+ innerInterpreterClazz =
scalaInterpreterClassLoader.loadClass(innerIntpClassName);
+}
}
- scalaInterpreterClassLoader = new URLClassLoader(urls.toArray(new
URL[0]),
- Thread.currentThread().getContextClassLoader());
}
-
-String innerIntpClassName = innerInterpreterClassMap.get(scalaVersion);
-Class clazz = scalaInterpreterClassLoader.loadClass(innerIntpClassName);
return (AbstractSparkScalaInterpreter)
-clazz.getConstructor(SparkConf.class, List.class,
Properties.class, InterpreterGroup.class, URLClassLoader.class, File.class)
-.newInstance(conf, getDependencyFiles(), getProperties(),
getInterpreterGroup(),
[zeppelin] branch master updated (812bc0f -> 9cecd23)
This is an automated email from the ASF dual-hosted git repository. zjffdu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/zeppelin.git. from 812bc0f [ZEPPELIN-5620] Staging directory is not deleted in Flink's yarn-application (#4306) add 9cecd23 [ZEPPELIN-5678] Remove the wordcount init job in Flink yarn-application mode (#4313) No new revisions were added by this update. Summary of changes: .../flink/ApplicationModeExecutionEnvironment.java| 7 +++ .../flink/ApplicationModeStreamEnvironment.java | 13 + .../apache/zeppelin/flink/FlinkScalaInterpreter.scala | 19 --- 3 files changed, 20 insertions(+), 19 deletions(-)
[zeppelin] branch master updated (be13f03 -> 812bc0f)
This is an automated email from the ASF dual-hosted git repository. zjffdu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/zeppelin.git. from be13f03 upgrade commons-compress and commons-io due to security concerns (#4315) add 812bc0f [ZEPPELIN-5620] Staging directory is not deleted in Flink's yarn-application (#4306) No new revisions were added by this update. Summary of changes: .../org/apache/zeppelin/flink/HadoopUtils.java | 5 ++--- .../zeppelin/flink/FlinkScalaInterpreter.scala | 23 +- .../zeppelin/flink/internal/FlinkShell.scala | 12 +-- 3 files changed, 26 insertions(+), 14 deletions(-)
[zeppelin] branch master updated: upgrade commons-compress and commons-io due to security concerns (#4315)
This is an automated email from the ASF dual-hosted git repository. zjffdu pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/zeppelin.git The following commit(s) were added to refs/heads/master by this push: new be13f03 upgrade commons-compress and commons-io due to security concerns (#4315) be13f03 is described below commit be13f03f537a314657246d8aff719aed6691acd0 Author: PJ Fanning AuthorDate: Thu Mar 17 09:53:42 2022 +0100 upgrade commons-compress and commons-io due to security concerns (#4315) --- pom.xml | 4 ++-- zeppelin-distribution/src/bin_license/LICENSE | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/pom.xml b/pom.xml index a8dee84..33541f4 100644 --- a/pom.xml +++ b/pom.xml @@ -130,13 +130,13 @@ 4.4.1 4.5.13 4.0.2 -1.20 +1.21 3.10 1.8 2.7 1.3 1.14 -2.6 +2.7 3.2.2 1.4 1.7.0 diff --git a/zeppelin-distribution/src/bin_license/LICENSE b/zeppelin-distribution/src/bin_license/LICENSE index c956727..583b8c0 100644 --- a/zeppelin-distribution/src/bin_license/LICENSE +++ b/zeppelin-distribution/src/bin_license/LICENSE @@ -6,8 +6,8 @@ The following components are provided under Apache License. (Apache 2.0) JavaEWAH v0.7.9 (https://github.com/lemire/javaewah) - https://github.com/lemire/javaewah/blob/master/LICENSE-2.0.txt (Apache 2.0) Apache Commons Logging (commons-logging:commons-logging:1.1.1 - http://commons.apache.org/proper/commons-logging/) (Apache 2.0) Apache Commons Codec (commons-codec:commons-codec:1.5 - http://commons.apache.org/proper/commons-codec/) -(Apache 2.0) Apache Commons Collections (commons-collections:commons-collections:3.2.1 - http://commons.apache.org/proper/commons-configuration/) -(Apache 2.0) Apache Commons Compress (org.apache.commons:commons-compress:1.9 - http://commons.apache.org/proper/commons-compress/) +(Apache 2.0) Apache Commons Collections (commons-collections:commons-collections:3.2.2 - http://commons.apache.org/proper/commons-configuration/) +(Apache 2.0) Apache Commons Compress (org.apache.commons:commons-compress:1.21 - http://commons.apache.org/proper/commons-compress/) (Apache 2.0) Apache Commons Configuration (org.apache.commons:commons-configuration2:2.7 - http://commons.apache.org/configuration/) (Apache 2.0) Apache Commons CLI (commons-cli:commons-cli:1.2 - http://commons.apache.org/cli/) (Apache 2.0) Apache Commons Exec (commons-exec:commons-exec:1.3 - http://commons.apache.org/exec/) @@ -20,7 +20,7 @@ The following components are provided under Apache License. (Apache 2.0) Apache Commons Net (commons-net:commons-net:2.2 - http://commons.apache.org/proper/commons-net/) (Apache 2.0) Apache Commons Pool2 (commons-exec:commons-pool2:2.3 - https://commons.apache.org/proper/commons-pool/) (Apache 2.0) Apache Commons FileUpload (commons-fileupload:commons-fileupload:1.3.1 - http://commons.apache.org/fileupload/) -(Apache 2.0) Apache Commons IO (commons-io:commons-io:2.4 - http://commons.apache.org/io/) +(Apache 2.0) Apache Commons IO (commons-io:commons-io:2.7 - http://commons.apache.org/io/) (Apache 2.0) Apache Commons VFS2 (org.apache.commons:commons-vfs2:2.0 - https://commons.apache.org/proper/commons-vfs/) (Apache 2.0) Apache Jackrabbit webdav (org.apache.jackrabbit:jackrabbit-webdav:jar:1.5.2 - https://jackrabbit.apache.org/jcr/components/jackrabbit-webdav-library.html) (Apache 2.0) Apache Jackrabbit JCR commons - http://jackrabbit.apache.org/jcr/components/jackrabbit-jcr-commons.html)
