Reamer commented on code in PR #4668:
URL: https://github.com/apache/zeppelin/pull/4668#discussion_r1350366811
##########
python/src/main/java/org/apache/zeppelin/python/PythonUtils.java:
##########
@@ -37,35 +36,22 @@ public class PythonUtils {
public static GatewayServer createGatewayServer(Object entryPoint,
String serverAddress,
int port,
- String secretKey,
- boolean useAuth) throws
IOException {
- LOGGER.info("Launching GatewayServer at " + serverAddress + ":" + port +
- ", useAuth: " + useAuth);
- if (useAuth) {
- try {
- Class clz = Class.forName("py4j.GatewayServer$GatewayServerBuilder",
true,
- Thread.currentThread().getContextClassLoader());
- Object builder =
clz.getConstructor(Object.class).newInstance(entryPoint);
- builder.getClass().getMethod("authToken",
String.class).invoke(builder, secretKey);
- builder.getClass().getMethod("javaPort", int.class).invoke(builder,
port);
- builder.getClass().getMethod("javaAddress",
InetAddress.class).invoke(builder,
- InetAddress.getByName(serverAddress));
- builder.getClass()
- .getMethod("callbackClient", int.class, InetAddress.class,
String.class)
- .invoke(builder, port, InetAddress.getByName(serverAddress),
secretKey);
- return (GatewayServer)
builder.getClass().getMethod("build").invoke(builder);
- } catch (Exception e) {
- throw new IOException(e);
- }
- } else {
- return new GatewayServer(entryPoint,
- port,
- GatewayServer.DEFAULT_PYTHON_PORT,
- InetAddress.getByName(serverAddress),
- InetAddress.getByName(serverAddress),
- GatewayServer.DEFAULT_CONNECT_TIMEOUT,
- GatewayServer.DEFAULT_READ_TIMEOUT,
- (List) null);
+ String secretKey) throws
IOException {
+ LOGGER.info("Launching GatewayServer at {}:{}", serverAddress, port);
+ try {
+ Class clz = Class.forName("py4j.GatewayServer$GatewayServerBuilder",
true,
Review Comment:
Can you document here why reflection is used here.
##########
python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java:
##########
@@ -116,7 +115,6 @@ public void open() throws InterpreterException {
}
try {
- this.usePy4jAuth =
Boolean.parseBoolean(getProperty("zeppelin.py4j.useAuth", "true"));
Review Comment:
:heavy_check_mark:
##########
zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/launcher/SparkInterpreterLauncher.java:
##########
@@ -294,37 +292,17 @@ private String detectSparkScalaVersion(String sparkHome,
Map<String, String> env
}
private String detectSparkScalaVersionByReplClass(String sparkHome) throws
Exception {
- File sparkLibFolder = new File(sparkHome + "/lib");
- if (sparkLibFolder.exists()) {
- // spark 1.6 if spark/lib exists
- File[] sparkAssemblyJars = new File(sparkHome + "/lib").listFiles(new
FilenameFilter() {
- @Override
- public boolean accept(File dir, String name) {
- return name.contains("spark-assembly");
- }
- });
- if (sparkAssemblyJars.length == 0) {
- throw new Exception("No spark assembly file found in SPARK_HOME: " +
sparkHome);
- }
- if (sparkAssemblyJars.length > 1) {
- throw new Exception("Multiple spark assembly file found in SPARK_HOME:
" + sparkHome);
- }
- try (URLClassLoader urlClassLoader = new URLClassLoader(new
URL[]{sparkAssemblyJars[0].toURI().toURL()});){
- urlClassLoader.loadClass("org.apache.spark.repl.SparkCommandLine");
- return "2.10";
- } catch (ClassNotFoundException e) {
- return "2.11";
- }
+ File sparkJarsFolder = new File(sparkHome + "/jars");
+ boolean sparkRepl212Exists =
+ Stream.of(sparkJarsFolder.listFiles()).anyMatch(file ->
file.getName().contains("spark-repl_2.12"));
+ boolean sparkRepl213Exists =
+ Stream.of(sparkJarsFolder.listFiles()).anyMatch(file ->
file.getName().contains("spark-repl_2.13"));
+ if (sparkRepl212Exists) {
Review Comment:
I like this approach. Do we want to throw an exception as soon as multiple
versions are found?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
