yanyan300300 commented on a change in pull request #8215: [FLINK-8801][yarn/s3]
Fix jars downloading issues due to inconsistent timestamp in S3 Filesystem
URL: https://github.com/apache/flink/pull/8215#discussion_r279876022
##########
File path: flink-yarn/src/main/java/org/apache/flink/yarn/Utils.java
##########
@@ -160,8 +170,37 @@ public static void setupYarnClassPath(Configuration conf,
Map<String, String> ap
fs.copyFromLocalFile(false, true, localSrcPath, dst);
+ // Note: If we directly used registerLocalResource(FileSystem,
Path) here, we would access the remote
+ // file once again which has problems with eventually
consistent read-after-write file
+ // systems. Instead, we decide to wait until the remote
file be available.
+
+ FileStatus[] fss = null;
+ int iter = 1;
+ while (iter <= REMOTE_RESOURCES_FETCH_NUM_RETRY) {
Review comment:
And could you clarify by "did in
YarnFileStageTest#testCopyFromLocalRecursive()"? Are you suggesting testing the
retry on `FileNotFoundException`? I think currently Flink is using a real S3
filesystem instead of a mock, so I am not quite sure how to do it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
