pan3793 commented on code in PR #2579: URL: https://github.com/apache/incubator-kyuubi/pull/2579#discussion_r867981889
########## docs/deployment/engine_on_yarn.md: ########## @@ -169,12 +169,33 @@ export HADOOP_CLASSPATH=`hadoop classpath` echo "stop" | ./bin/yarn-session.sh -id application_XXXXX_XXX ``` -If the `TopSpeedWindowing` passes, configure it in `$KYUUBI_HOME/conf/kyuubi-env.sh` or `$FLINK_HOME/bin/config.sh`, e.g. +If the `TopSpeedWindowing` passes, configure it in `$KYUUBI_HOME/conf/kyuubi-env.sh` ```bash $ echo "export HADOOP_CONF_DIR=/path/to/hadoop/conf" >> $KYUUBI_HOME/conf/kyuubi-env.sh ``` +#### Required Environment Variable + +The `FLINK_HADOOP_CLASSPATH` is required, too. + +If using Hadoop 3.x, configured it in `$KYUUBI_HOME/conf/kyuubi-env.sh` as follows: + +```bash +$ echo "export FLINK_HADOOP_CLASSPATH=/path/to/hadoop-client-runtime-*.jar:/path/to/hadoop-client-api-*.jar" >> $KYUUBI_HOME/conf/kyuubi-env.sh +``` +`hadoop-client-api-*.jar` and `hadoop-client-runtime-*.jar` are only applicable to users whose + +hadoop version is 3.x, if users use hadoop2.x, FLINK_HADOOP_CLASSPATH is recommended to + +be set to `hadoop classpath` + +If using Hadoop 2.x, configured it in `$KYUUBI_HOME/conf/kyuubi-env.sh` as follows: + +```bash +$ echo "export FLINK_HADOOP_CLASSPATH=`hadoop classpath`" >> $KYUUBI_HOME/conf/kyuubi-env.sh +``` Review Comment: For users who are using Hadoop 3.x, Hadoop shaded client is recommaned instead of Hadoop vanilla jars. For users who are using Hadoop 2.x, `FLINK_HADOOP_CLASSPATH` should be set to `hadoop classpath` to use Hadoop vanilla jars. For users which does not use Hadoop services, e.g. HDFS, YARN at all, Hadoop client jars is also required, and recommand to use Hadoop shaded client as Hadoop 3.x's users. See [HADOOP-11656](https://issues.apache.org/jira/browse/HADOOP-11656) for details of Hadoop shaded client. To use Hadoop shaded client, please configure `$KYUUBI_HOME/conf/kyuubi-env.sh` as follows: ```bash $ echo "export FLINK_HADOOP_CLASSPATH=/path/to/hadoop-client-runtime-3.3.2.jar:/path/to/hadoop-client-api-3.3.2.jar" >> $KYUUBI_HOME/conf/kyuubi-env.sh ``` To use Hadoop vanilla jars, please configure `$KYUUBI_HOME/conf/kyuubi-env.sh` as follows: ```bash $ echo "export FLINK_HADOOP_CLASSPATH=`hadoop classpath`" >> $KYUUBI_HOME/conf/kyuubi-env.sh ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
