venkata91 commented on code in PR #23164:
URL: https://github.com/apache/flink/pull/23164#discussion_r1296273712
##########
flink-yarn/src/main/java/org/apache/flink/yarn/YarnApplicationFileUploader.java:
##########
@@ -360,11 +362,27 @@ List<String> registerProvidedLocalResources() {
envShipResourceList.add(descriptor);
if (!isFlinkDistJar(filePath.getName()) &&
!isPlugin(filePath)) {
- classPaths.add(fileName);
+ URI parentDirectoryUri = new
Path(fileName).getParent().toUri();
+ String relativeParentDirectory =
+ new Path(filePath.getName())
+ .toUri()
+ .relativize(parentDirectoryUri)
+ .toString();
+
+ if (!resourcesDir.contains(relativeParentDirectory)) {
+ resourcesDir.add(relativeParentDirectory);
+ }
+ resources.add(fileName);
Review Comment:
The existing config `yarn.provided.lib.dirs` supports adding all the
resources (jars or configs) under each of those dirs separated by comma in to
the classpath. Why adding the parent dir also to the classpath is an overkill?
Take `yarn.ship-files` as an example takes arbitrary list of dirs and adds
both the files and the dir under each of the individual dirs to the classpath.
Isn't this the case with `yarn.provided.lib.dirs` as well? Except that in the
`yarn.provided.lib.dirs` is more of a platform specific libs which won't change
between different flink apps.
Also introducing a new `resources` dir and only supporting that feels
limiting. Ideally, users or platforms would like to logically group resources
in to their own dirs. For eg: hadoop, hive libs etc in to their own dirs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
