[GitHub] [flink] wangyang0918 commented on a diff in pull request #23164: [FLINK- 32775]Add parent dir of files to classpath using yarn.provided.lib.dirs

[via GitHub]

wangyang0918 commented on code in PR #23164:
URL: https://github.com/apache/flink/pull/23164#discussion_r1318475489


##########
flink-yarn/src/main/java/org/apache/flink/yarn/YarnApplicationFileUploader.java:
##########
@@ -360,11 +362,27 @@ List<String> registerProvidedLocalResources() {
                     envShipResourceList.add(descriptor);
 
                     if (!isFlinkDistJar(filePath.getName()) && 
!isPlugin(filePath)) {
-                        classPaths.add(fileName);
+                        URI parentDirectoryUri = new 
Path(fileName).getParent().toUri();
+                        String relativeParentDirectory =

Review Comment:
   ```
   
-Dyarn.provided.lib.dirs="hdfs://myhdfs/tmp/flink-1.18-SNAPSHOT/lib;hdfs://myhdfs/tmp/flink-1.18-SNAPSHOT/plugins;hdfs://myhdfs/tmp/flink-1.18-SNAPSHOT/bin"
   ```
   
   It seems that we already do the relativize in the 
`getAllFilesInProvidedLibDirs`. So the input `providedLibDirs` is already 
relative path(e.g. `bin/config.sh`, `bin/find-flink-home.sh`).
   
   In addition, adding a non-jar file into the classpath does not take too many 
benefits. I suggest to only add the parent directory to the classpath. The Yarn 
ship files have the same behavior.
   
   Then we could have a simpler code here.
   ```
   if (fileName.endsWith("jar")) {
       resourcesJar.add(fileName);
   } else {
       resourcesDir.add(new Path(fileName).getParent().toString());
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org