Hi everyone, We would like to start a discussion thread on "FLIP-53: Fine Grained Resource Management"[1], where we propose how to improve Flink resource management and scheduling.
This FLIP mainly discusses the following issues. - How to support tasks with fine grained resource requirements. - How to unify resource management for jobs with / without fine grained resource requirements. - How to unify resource management for streaming / batch jobs. Key changes proposed in the FLIP are as follows. - Unify memory management for operators with / without fine grained resource requirements by applying a fraction based quota mechanism. - Unify resource scheduling for streaming and batch jobs by setting slot sharing groups for pipelined regions during compiling stage. - Dynamically allocate slots from task executors' available resources. Please find more details in the FLIP wiki document [1]. Looking forward to your feedbacks. Thank you~ Xintong Song [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-53%3A+Fine+Grained+Resource+Management
