wangyang0918 commented on issue #8303: [FLINK-12343] [flink-yarn] add file replication config for yarn configuration URL: https://github.com/apache/flink/pull/8303#issuecomment-527065400 Hi @tillrohrmann, FYI 1. `FileSystem#SetReplica()` happens asynchronously. The blocks will be populated or removed in the background. You could check the [ClientProtocol.java#L276](https://github.com/apache/hadoop/blob/1129288cf5045e17b0e761a0d75f40bf2fe6de03/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java#L276) for details. Also the hdfs client implements an internal waitForReplication to support block call. 2. A datanode could serve multiple files downloading from nodemanager. Generally, the bandwidth becomes a bottleneck. So increase the replication could reduce the localization duration.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
