[jira] (FLINK-5668) Reduce dependency on HDFS at job startup time

[Stephan Ewen (JIRA)]

Title: Message Title

Stephan Ewen commented on FLINK-5668   Re: Reduce dependency on HDFS at job startup time The YARN setup relies heavily on HDFS - there are more places where this is used. YARN's own fault tolerance and high availability relies on HDFS, so I think it is fair that Flink-on-YARN also does that. Before doing any change there, I would like to understand what you want to fix by reducing one dependency on HDFS when there are so many others. This change may compromise high-availability, so we need to be very conservative and have a good case to change such a thing. I would also suggest for such critical changes to discuss the approach on the issue or mailing list before - it helps prevent cases where pull requests were rejected due to incompatible design, an the contributor was upset for having written a lot of code that was not merged. Concerning the web server: The JobManager web server is also not a good place to distribute files - it is really intended for displaying information and metrics. It is not guaranteed to even run and has no high-availability resilience Add Comment   This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)