[zeppelin] branch branch-0.9 updated: [ZEPPELIN-5175]. Should not set driver-class-path when it is yarn cluster mode

[zjffdu]

This is an automated email from the ASF dual-hosted git repository.

zjffdu pushed a commit to branch branch-0.9
in repository https://gitbox.apache.org/repos/asf/zeppelin.git


The following commit(s) were added to refs/heads/branch-0.9 by this push:
     new 58f068a  [ZEPPELIN-5175]. Should not set driver-class-path when it is 
yarn cluster mode
58f068a is described below

commit 58f068aed6cc5d7db83a01171e9df04834b54e27
Author: Jeff Zhang <zjf...@apache.org>
AuthorDate: Thu Dec 24 16:42:01 2020 +0800

    [ZEPPELIN-5175]. Should not set driver-class-path when it is yarn cluster 
mode
    
    ### What is this PR for?
    
    For yarn cluster mode, driver runs in node manager whether there's no 
zeppelin jars.
    
    ### What type of PR is it?
    [Improvement ]
    
    ### Todos
    * [ ] - Task
    
    ### What is the Jira issue?
    * https://issues.apache.org/jira/browse/ZEPPELIN-5175
    
    ### How should this be tested?
    * CI pass
    
    ### Screenshots (if appropriate)
    
    ### Questions:
    * Does the licenses files need update? No
    * Is there breaking changes for older versions? No
    * Does this needs documentation? No
    
    Author: Jeff Zhang <zjf...@apache.org>
    
    Closes #4004 from zjffdu/ZEPPELIN-5175 and squashes the following commits:
    
    761c45f81 [Jeff Zhang] [ZEPPELIN-5175]. Should not set driver-class-path 
when it is yarn cluster mode
    
    (cherry picked from commit 8e3bf304fb31f6b46252e89a8f268784eb4a8359)
    Signed-off-by: Jeff Zhang <zjf...@apache.org>
---
 bin/interpreter.sh | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/bin/interpreter.sh b/bin/interpreter.sh
index 2eac09d..1280ac0 100755
--- a/bin/interpreter.sh
+++ b/bin/interpreter.sh
@@ -284,11 +284,17 @@ fi
 if [[ -n "${SPARK_SUBMIT}" ]]; then
   IFS=' ' read -r -a SPARK_SUBMIT_OPTIONS_ARRAY <<< "${SPARK_SUBMIT_OPTIONS}"
   IFS=' ' read -r -a ZEPPELIN_SPARK_CONF_ARRAY <<< "${ZEPPELIN_SPARK_CONF}"
-  INTERPRETER_RUN_COMMAND+=("${SPARK_SUBMIT}" "--class" "${ZEPPELIN_SERVER}" 
"--driver-class-path" 
"${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${ZEPPELIN_INTP_CLASSPATH}" 
"--driver-java-options" "${JAVA_INTP_OPTS}" "${SPARK_SUBMIT_OPTIONS_ARRAY[@]}" 
"${ZEPPELIN_SPARK_CONF_ARRAY[@]}" "${SPARK_APP_JAR}" "${CALLBACK_HOST}" 
"${PORT}" "${INTP_GROUP_ID}" "${INTP_PORT}")
+  if [[ "${ZEPPELIN_SPARK_YARN_CLUSTER}" == "true"  ]]; then
+    INTERPRETER_RUN_COMMAND+=("${SPARK_SUBMIT}" "--class" "${ZEPPELIN_SERVER}" 
"--driver-java-options" "${JAVA_INTP_OPTS}" "${SPARK_SUBMIT_OPTIONS_ARRAY[@]}" 
"${ZEPPELIN_SPARK_CONF_ARRAY[@]}" "${SPARK_APP_JAR}" "${CALLBACK_HOST}" 
"${PORT}" "${INTP_GROUP_ID}" "${INTP_PORT}")
+  else
+    INTERPRETER_RUN_COMMAND+=("${SPARK_SUBMIT}" "--class" "${ZEPPELIN_SERVER}" 
"--driver-class-path" 
"${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${ZEPPELIN_INTP_CLASSPATH}" 
"--driver-java-options" "${JAVA_INTP_OPTS}" "${SPARK_SUBMIT_OPTIONS_ARRAY[@]}" 
"${ZEPPELIN_SPARK_CONF_ARRAY[@]}" "${SPARK_APP_JAR}" "${CALLBACK_HOST}" 
"${PORT}" "${INTP_GROUP_ID}" "${INTP_PORT}")
+  fi
 else
   IFS=' ' read -r -a JAVA_INTP_OPTS_ARRAY <<< "${JAVA_INTP_OPTS}"
   IFS=' ' read -r -a ZEPPELIN_INTP_MEM_ARRAY <<< "${ZEPPELIN_INTP_MEM}"
   INTERPRETER_RUN_COMMAND+=("${ZEPPELIN_RUNNER}" "${JAVA_INTP_OPTS_ARRAY[@]}" 
"${ZEPPELIN_INTP_MEM_ARRAY[@]}" "-cp" 
"${ZEPPELIN_INTP_CLASSPATH_OVERRIDES}:${ZEPPELIN_INTP_CLASSPATH}" 
"${ZEPPELIN_SERVER}" "${CALLBACK_HOST}" "${PORT}" "${INTP_GROUP_ID}" 
"${INTP_PORT}")
 fi
 
+# Don't remove this echo, it is for diagnose, this line of output will be 
redirected to java log4j output.
+echo "Interpreter launch command: ${INTERPRETER_RUN_COMMAND[@]}"
 exec "${INTERPRETER_RUN_COMMAND[@]}"