bgeng777 commented on a change in pull request #18531:
URL: https://github.com/apache/flink/pull/18531#discussion_r801320099
##########
File path:
flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java
##########
@@ -1685,6 +1699,35 @@ void addLibFoldersToShipFiles(Collection<File>
effectiveShipFiles) {
}
}
+ @VisibleForTesting
+ void addUsrLibFolderToShipFiles(
+ Collection<File> effectiveShipFiles, Collection<File>
systemShipFiles) {
+ // Add usrlib folder to the ship files if it exists
+ // Classes in the folder will be loaded by UserClassLoader if
CLASSPATH_INCLUDE_USER_JAR is
+ // DISABLED.
+ final Optional<File> usrLibDir = getLocalUsrLibDirectory();
+
+ if (usrLibDir.isPresent()) {
+ File usrLibDirFile = usrLibDir.get();
+ if (usrLibDirFile.isDirectory()) {
+ checkArgument(
Review comment:
Hi Yang, thanks for the feedback. In fact, my first version of codes is
pretty like your code snippet.
But after thinking more carefully, I use my current implementation
(including writing my own getLocalUsrLibDirectory()) for the following reason:
1. AFAIK, YarnClusterDescriptor is executed in the flink client side while
ClusterEntryPointUtil is executed on the remote cluster side. If I use the
`ClusterEntrypointUtils.tryFindUserLibDirectory()` in client side, due to its
impl, the user must specify the FLINK_LIB_DIR correctly. But IIUC, we do not
have such requirement before when using YARN.
2. In my `getLocalUsrLibDirectory()`, I use the FLINK_HOME as a pivot to
find the `usrlib`. This is not so perfect but comparing with FLINK_LIB_DIR,
FLINK_HOME may be more widely used.
In summary, I bet the key question is if we should ask the action of finding
`usrlib` to adopt same logic(using FLINK_LIB_DIR or use current working dir) in
both client and cluster side. Do you have any suggestion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
