eogramow commented on a change in pull request #12791:
URL: https://github.com/apache/flink/pull/12791#discussion_r452731277
##########
File path:
flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java
##########
@@ -267,6 +278,29 @@ public void addShipFiles(List<File> shipFiles) {
this.shipFiles.addAll(shipFiles);
}
+ /**
+ * Adds the given files to the list of archives to ship.
+ **
+ * @param shipArchives archives to ship
+ */
+ public void addShipArchives(List<File> shipArchives) {
+
checkArgument(isArchiveOnlyIncludedInShipArchiveFiles(shipArchives),
"Non-archive files are included.");
+ this.shipArchives.addAll(shipArchives);
+ }
+
+ private static boolean
isArchiveOnlyIncludedInShipArchiveFiles(List<File> shipFiles) {
+ return shipFiles.stream()
+ .filter(File::isFile)
+ .map(File::getName)
+ .map(String::toLowerCase)
+ .allMatch(name -> name.endsWith(".tar.gz") ||
+ name.endsWith(".tar") ||
+ name.endsWith(".tgz") ||
+ name.endsWith(".dst") ||
+ name.endsWith(".jar") ||
Review comment:
I've done what you mentioned except adding a test to the
YarnFileStageTest. I need a little more time in order to do that. I'll push a
commit about javadoc, style and etc first.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
