QiLuo-BD commented on issue #7186: [FLINK-10941] Keep slots which contain unconsumed result partitions URL: https://github.com/apache/flink/pull/7186#issuecomment-453921155 Thanks @Zhijiang. I agree with you that a shuffle manager will be a better solution for this issue in Flink 1.8. If more users request this before they can upgrade to Flink 1.8, we could still merge this PR to Flink 1.5~1.7 branch. > On Jan 14, 2019, at 3:33 PM, zhijiang <[email protected]> wrote: > > Hi @QiLuo-BD <https://github.com/QiLuo-BD> , very sorry for the late response! > > During these days, the proposal of shuffle manager is under discussion and already reaches an basic agreement. So the formal solution for this issue is introducing a ShuffleMaster component in JobMaster for global partition management, then the life cycle of task/partition/TM could be decoupled completely. And the fist version wishes to be covered in FLINK 1.8. > > I think your current solution might also work from functional aspect in current implementation. But it seems a little hacky/tricky, the reasons are as follows: > > Task and its partitions are still coupled. The task is finished, but task#canBeReleased is based on partition#isReleased. > TM and internal partitions are still coupled. For external shuffle service in future, the TM might be released if the partition is written onto external store. > So my suggestion is if the current modifications are small enough to work around temporarily, then it might make sense for merging. But this PR seems a bit large, involved in slot manager, task slot and network stack modifications. It is better to wait for the formal FLINK 1.8 or adjust the parameters to work around currently. :) > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub <https://github.com/apache/flink/pull/7186#issuecomment-453919737>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ArUS82sO5WIUtbT_JL3kk3Q7A428wKoMks5vDDKsgaJpZM4Y3THW>. >
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
