kl0u edited a comment on issue #7313: [FLINK-11116][fs-connector] Removed randomness from HDFS and Local fs writers. URL: https://github.com/apache/flink/pull/7313#issuecomment-448570597 Actually the only scenario where the randomness in the tmp files can help is in the case where we have a "split-brain" scenario. In this scenario, some TaskManagers (TMs) are considered down but they are not. So the job restarts, but the old TMs keep on processing data. In this case, the old TMs will not be able to commit any data, as they have **no** JobManager (JM) so no checkpoints, but they will be writing in temporary files. Now the new TMs, will also be writing in tmp files (and then committing them). If there is no randomness, then old and new TMs may be writing in the same temp files (if the FS allows it). That said, I am not so sure if this scenario can happen, _i.e._ if TMs can keep running without a JM and I am not sure if handling it at the sink is enough. @StephanEwen ?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
