Github user uce commented on a diff in the pull request:
https://github.com/apache/flink/pull/2708#discussion_r86542605
--- Diff: flink-dist/src/main/flink-bin/yarn-bin/yarn-session.sh ---
@@ -35,14 +33,15 @@ JVM_ARGS="$JVM_ARGS -Xmx512m"
# Flink CLI client
constructCLIClientClassPath() {
- for jarfile in $FLINK_LIB_DIR/*.jar ; do
- if [[ $CC_CLASSPATH = "" ]]; then
- CC_CLASSPATH=$jarfile;
- else
- CC_CLASSPATH=$CC_CLASSPATH:$jarfile
- fi
- done
- echo $CC_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS
+ while read -d '' -r jarfile ; do
+ if [[ $FLINK_CLASSPATH = "" ]]; then
+ CC_CLASSPATH="$jarfile";
+ else
+ CC_CLASSPATH="$CC_CLASSPATH":"$jarfile"
+ fi
+ done < <(find "$FLINK_LIB_DIR" -name '*.jar' -print0)
+
+ echo $CC_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS
--- End diff --
I think this is not enough for the YARN side of things. We need to update
the logic in `AbstractYarnClusterDescriptor` (and maybe `FlinkYarnSessionCli`
as well).
In `AbstractYarnClusterDescriptor` there is `addLibFolderToShipFiles` which
adds the lib directory to the list of ship files. These are uploaded and the
classpath of the containers has it as `*` (see line 620 of the descriptor).
We need to update this to upload the sub directories as well and add these
to the class path. Maybe it's best to just iterate over the lib folder
recursively and add each JAR file. @mxm What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
