Re: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs
Hi, all. I have started voting[1] for this FLIP. If you have any questions, you can reply in this discussion thread. [1]: https://lists.apache.org/thread/gdymm7pr2slzy9gqkfo97vn73496w0cj Best regards, Junrui Xintong Song 于2023年1月4日周三 18:16写道: > Thanks for the proposal. > > Another potential benefit I see in this FLIP is that it may reduce the > complexity and maintenance overhead of the scheduler. During developing > hybrid shuffle, we had to re-implement some similar logic to make both > default and adaptive batch schedulers support the new shuffle type. For > batch workloads, the functionality provided by adaptive batch scheduler > should already be a superset of those provided by the default scheduler. If > that can be confirmed from user feedback after making the adaptive batch > scheduler default, we may consider making default scheduler streaming > dedicated. That should significantly simplify the default scheduler, > benefiting both the maintenance of the default scheduler and the > development of new features in the batch area. > > So +1 from my side. > > Best, > > Xintong > > > > On Wed, Jan 4, 2023 at 1:14 PM JunRui Lee wrote: > > > Hi, Yuxia > > > > Thanks for the reply. > > > > > 1. About the configuration renaming. The old configurations ... > > > > Thanks for your suggestion. And this explanation has been updated > > to the FLIP to describe this change more clearly. The old configurations > > will be deprecated after the configuration renaming, and it still will be > > considered to guarantee compatibility. > > > > > 2. I'm cursion in which case users will disable auto parallelism... > > > > Actually, the adaptive batch scheduler is not only used for automatic > > parallelism derivation, but also for speculative execution. If users > > want to use speculative execution and do not enable automatic > > parallelism, they can configure the configuration > > "execution.batch.adaptive.auto-parallelism.enabled" instead of > > configuring the configuration "jobmanager.scheduler". > > > > In addition, “adaptive” represents a function, not just the automatic > > derivation of parallelism, and more functions may be added in the > > future. So we want users to focus on enabling or disabling the > > parallelism derivation function itself rather than replacing the > > scheduler type. > > > > Best, > > JunRui > > > > Lijie Wang 于2023年1月3日周二 11:03写道: > > > > > Hi Junrui, > > > > > > Thanks for driving this FLIP, + 1 for this proposal. I believe it will > > > greatly improve the experiences of batch users. > > > > > > Best, > > > Lijie > > > > > > Zhu Zhu 于2022年12月30日周五 12:40写道: > > > > > > > Hi Junrui, > > > > > > > > Thanks for creating this FLIP! > > > > > > > > AdaptiveBatchScheduler is more powerful than DefaultScheduler in > batch > > > > scheduling, also with some must-have features like speculative > > execution. > > > > It will be great that users can easily use it, without required to > > > knowing > > > > the underlying scheduler and configuring some advanced items. > > > > > > > > So generally +1 for this proposal. > > > > > > > > Regarding the configuration key renaming, like yuxia mentioned, we > > should > > > > deprecate the old ones and add new ones with new names, to guarantee > > > > compatibility. > > > > > > > > Thanks, > > > > Zhu > > > > > > > > yuxia 于2022年12月30日周五 11:10写道: > > > > > > > > > > Hi, JunRui Lee. > > > > > > > > > > Thanks for driving this FLIP. It must a good improvement for batch > > > > users' experiences. > > > > > I have few questions about this FLIP: > > > > > 1: About the configuration renaming. The old configurations will be > > > > deprecated or removed directly? if user upgrade their Flink version, > > > these > > > > old configuration will still be considered or just ignored? If > ignore, > > > the > > > > users may need to modify their configurations after they upgrade > their > > > > Flink. > > > > > > > > > > 2: I'm cursion in which case users will disable auto parallelism > > > > derivation if they have enabled adaptive batch scheduler. IIUC, auto > > > > parallelism derivation is what adaptive batch sched
Re: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs
Thanks for the proposal. Another potential benefit I see in this FLIP is that it may reduce the complexity and maintenance overhead of the scheduler. During developing hybrid shuffle, we had to re-implement some similar logic to make both default and adaptive batch schedulers support the new shuffle type. For batch workloads, the functionality provided by adaptive batch scheduler should already be a superset of those provided by the default scheduler. If that can be confirmed from user feedback after making the adaptive batch scheduler default, we may consider making default scheduler streaming dedicated. That should significantly simplify the default scheduler, benefiting both the maintenance of the default scheduler and the development of new features in the batch area. So +1 from my side. Best, Xintong On Wed, Jan 4, 2023 at 1:14 PM JunRui Lee wrote: > Hi, Yuxia > > Thanks for the reply. > > > 1. About the configuration renaming. The old configurations ... > > Thanks for your suggestion. And this explanation has been updated > to the FLIP to describe this change more clearly. The old configurations > will be deprecated after the configuration renaming, and it still will be > considered to guarantee compatibility. > > > 2. I'm cursion in which case users will disable auto parallelism... > > Actually, the adaptive batch scheduler is not only used for automatic > parallelism derivation, but also for speculative execution. If users > want to use speculative execution and do not enable automatic > parallelism, they can configure the configuration > "execution.batch.adaptive.auto-parallelism.enabled" instead of > configuring the configuration "jobmanager.scheduler". > > In addition, “adaptive” represents a function, not just the automatic > derivation of parallelism, and more functions may be added in the > future. So we want users to focus on enabling or disabling the > parallelism derivation function itself rather than replacing the > scheduler type. > > Best, > JunRui > > Lijie Wang 于2023年1月3日周二 11:03写道: > > > Hi Junrui, > > > > Thanks for driving this FLIP, + 1 for this proposal. I believe it will > > greatly improve the experiences of batch users. > > > > Best, > > Lijie > > > > Zhu Zhu 于2022年12月30日周五 12:40写道: > > > > > Hi Junrui, > > > > > > Thanks for creating this FLIP! > > > > > > AdaptiveBatchScheduler is more powerful than DefaultScheduler in batch > > > scheduling, also with some must-have features like speculative > execution. > > > It will be great that users can easily use it, without required to > > knowing > > > the underlying scheduler and configuring some advanced items. > > > > > > So generally +1 for this proposal. > > > > > > Regarding the configuration key renaming, like yuxia mentioned, we > should > > > deprecate the old ones and add new ones with new names, to guarantee > > > compatibility. > > > > > > Thanks, > > > Zhu > > > > > > yuxia 于2022年12月30日周五 11:10写道: > > > > > > > > Hi, JunRui Lee. > > > > > > > > Thanks for driving this FLIP. It must a good improvement for batch > > > users' experiences. > > > > I have few questions about this FLIP: > > > > 1: About the configuration renaming. The old configurations will be > > > deprecated or removed directly? if user upgrade their Flink version, > > these > > > old configuration will still be considered or just ignored? If ignore, > > the > > > users may need to modify their configurations after they upgrade their > > > Flink. > > > > > > > > 2: I'm cursion in which case users will disable auto parallelism > > > derivation if they have enabled adaptive batch scheduler. IIUC, auto > > > parallelism derivation is what adaptive batch scheduler aim to do. If > use > > > want to diable auto parallelism derivation, can they just disable > > adaptive > > > batch scheduler.? > > > > > > > > Best regards, > > > > Yuxia > > > > > > > > - 原始邮件 - > > > > 发件人: "JunRui Lee" > > > > 收件人: "dev" > > > > 发送时间: 星期四, 2022年 12 月 29日 下午 7:45:36 > > > > 主题: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default > > > scheduler for batch jobs > > > > > > > > Hi, devs, > > > > > > > > I'd like to start a discussion about FLIP-283: Use adaptive batch > > > > scheduler as default
Re: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs
Hi, Yuxia Thanks for the reply. > 1. About the configuration renaming. The old configurations ... Thanks for your suggestion. And this explanation has been updated to the FLIP to describe this change more clearly. The old configurations will be deprecated after the configuration renaming, and it still will be considered to guarantee compatibility. > 2. I'm cursion in which case users will disable auto parallelism... Actually, the adaptive batch scheduler is not only used for automatic parallelism derivation, but also for speculative execution. If users want to use speculative execution and do not enable automatic parallelism, they can configure the configuration "execution.batch.adaptive.auto-parallelism.enabled" instead of configuring the configuration "jobmanager.scheduler". In addition, “adaptive” represents a function, not just the automatic derivation of parallelism, and more functions may be added in the future. So we want users to focus on enabling or disabling the parallelism derivation function itself rather than replacing the scheduler type. Best, JunRui Lijie Wang 于2023年1月3日周二 11:03写道: > Hi Junrui, > > Thanks for driving this FLIP, + 1 for this proposal. I believe it will > greatly improve the experiences of batch users. > > Best, > Lijie > > Zhu Zhu 于2022年12月30日周五 12:40写道: > > > Hi Junrui, > > > > Thanks for creating this FLIP! > > > > AdaptiveBatchScheduler is more powerful than DefaultScheduler in batch > > scheduling, also with some must-have features like speculative execution. > > It will be great that users can easily use it, without required to > knowing > > the underlying scheduler and configuring some advanced items. > > > > So generally +1 for this proposal. > > > > Regarding the configuration key renaming, like yuxia mentioned, we should > > deprecate the old ones and add new ones with new names, to guarantee > > compatibility. > > > > Thanks, > > Zhu > > > > yuxia 于2022年12月30日周五 11:10写道: > > > > > > Hi, JunRui Lee. > > > > > > Thanks for driving this FLIP. It must a good improvement for batch > > users' experiences. > > > I have few questions about this FLIP: > > > 1: About the configuration renaming. The old configurations will be > > deprecated or removed directly? if user upgrade their Flink version, > these > > old configuration will still be considered or just ignored? If ignore, > the > > users may need to modify their configurations after they upgrade their > > Flink. > > > > > > 2: I'm cursion in which case users will disable auto parallelism > > derivation if they have enabled adaptive batch scheduler. IIUC, auto > > parallelism derivation is what adaptive batch scheduler aim to do. If use > > want to diable auto parallelism derivation, can they just disable > adaptive > > batch scheduler.? > > > > > > Best regards, > > > Yuxia > > > > > > - 原始邮件 - > > > 发件人: "JunRui Lee" > > > 收件人: "dev" > > > 发送时间: 星期四, 2022年 12 月 29日 下午 7:45:36 > > > 主题: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default > > scheduler for batch jobs > > > > > > Hi, devs, > > > > > > I'd like to start a discussion about FLIP-283: Use adaptive batch > > > scheduler as default scheduler for batch jobs[1]. > > > > > > In FLIP-187, we introduced an adaptive batch scheduler. The adaptive > > > batch scheduler has stronger batch scheduling capabilities, including > > > automatically deciding parallelisms of job vertices for batch > > > jobs (FLIP-187)[2], data balanced distribution (FLINK-29663)[3], > > > and speculative execution (FLIP-168)[4]. To further use the adaptive > > > batch scheduler to improve flink's batch capability, in this FLIP > > > we aim to make the adaptive batch scheduler as the default batch > > > scheduler. > > > > > > Currently, users have to set some configuration of the adaptive > > > batch scheduler, which is not very convenient. To use the adaptive > > > batch scheduler as the default batch scheduler, we need to improve > > > the user's out-of-the-box experience. Therefore, we also need to > > > optimize the current adaptive batch scheduler configuration. > > > > > > Looking forward to your feedback. > > > > > > [1]: > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-283%3A+Use+adaptive+batch+scheduler+as+default+scheduler+for+batch+jobs > > > [2]: > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-187%3A+Adaptive+Batch+Scheduler > > > [3]:https://issues.apache.org/jira/browse/FLINK-29663 > > > [4]: > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+Execution+for+Batch+Job > > > > > > Best regards, > > > JunRui Lee > > >
Re: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs
Hi Junrui, Thanks for driving this FLIP, + 1 for this proposal. I believe it will greatly improve the experiences of batch users. Best, Lijie Zhu Zhu 于2022年12月30日周五 12:40写道: > Hi Junrui, > > Thanks for creating this FLIP! > > AdaptiveBatchScheduler is more powerful than DefaultScheduler in batch > scheduling, also with some must-have features like speculative execution. > It will be great that users can easily use it, without required to knowing > the underlying scheduler and configuring some advanced items. > > So generally +1 for this proposal. > > Regarding the configuration key renaming, like yuxia mentioned, we should > deprecate the old ones and add new ones with new names, to guarantee > compatibility. > > Thanks, > Zhu > > yuxia 于2022年12月30日周五 11:10写道: > > > > Hi, JunRui Lee. > > > > Thanks for driving this FLIP. It must a good improvement for batch > users' experiences. > > I have few questions about this FLIP: > > 1: About the configuration renaming. The old configurations will be > deprecated or removed directly? if user upgrade their Flink version, these > old configuration will still be considered or just ignored? If ignore, the > users may need to modify their configurations after they upgrade their > Flink. > > > > 2: I'm cursion in which case users will disable auto parallelism > derivation if they have enabled adaptive batch scheduler. IIUC, auto > parallelism derivation is what adaptive batch scheduler aim to do. If use > want to diable auto parallelism derivation, can they just disable adaptive > batch scheduler.? > > > > Best regards, > > Yuxia > > > > ----- 原始邮件 ----- > > 发件人: "JunRui Lee" > > 收件人: "dev" > > 发送时间: 星期四, 2022年 12 月 29日 下午 7:45:36 > > 主题: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default > scheduler for batch jobs > > > > Hi, devs, > > > > I'd like to start a discussion about FLIP-283: Use adaptive batch > > scheduler as default scheduler for batch jobs[1]. > > > > In FLIP-187, we introduced an adaptive batch scheduler. The adaptive > > batch scheduler has stronger batch scheduling capabilities, including > > automatically deciding parallelisms of job vertices for batch > > jobs (FLIP-187)[2], data balanced distribution (FLINK-29663)[3], > > and speculative execution (FLIP-168)[4]. To further use the adaptive > > batch scheduler to improve flink's batch capability, in this FLIP > > we aim to make the adaptive batch scheduler as the default batch > > scheduler. > > > > Currently, users have to set some configuration of the adaptive > > batch scheduler, which is not very convenient. To use the adaptive > > batch scheduler as the default batch scheduler, we need to improve > > the user's out-of-the-box experience. Therefore, we also need to > > optimize the current adaptive batch scheduler configuration. > > > > Looking forward to your feedback. > > > > [1]: > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-283%3A+Use+adaptive+batch+scheduler+as+default+scheduler+for+batch+jobs > > [2]: > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-187%3A+Adaptive+Batch+Scheduler > > [3]:https://issues.apache.org/jira/browse/FLINK-29663 > > [4]: > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+Execution+for+Batch+Job > > > > Best regards, > > JunRui Lee >
Re: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs
Hi Junrui, Thanks for creating this FLIP! AdaptiveBatchScheduler is more powerful than DefaultScheduler in batch scheduling, also with some must-have features like speculative execution. It will be great that users can easily use it, without required to knowing the underlying scheduler and configuring some advanced items. So generally +1 for this proposal. Regarding the configuration key renaming, like yuxia mentioned, we should deprecate the old ones and add new ones with new names, to guarantee compatibility. Thanks, Zhu yuxia 于2022年12月30日周五 11:10写道: > > Hi, JunRui Lee. > > Thanks for driving this FLIP. It must a good improvement for batch users' > experiences. > I have few questions about this FLIP: > 1: About the configuration renaming. The old configurations will be > deprecated or removed directly? if user upgrade their Flink version, these > old configuration will still be considered or just ignored? If ignore, the > users may need to modify their configurations after they upgrade their Flink. > > 2: I'm cursion in which case users will disable auto parallelism derivation > if they have enabled adaptive batch scheduler. IIUC, auto parallelism > derivation is what adaptive batch scheduler aim to do. If use want to diable > auto parallelism derivation, can they just disable adaptive batch scheduler.? > > Best regards, > Yuxia > > - 原始邮件 - > 发件人: "JunRui Lee" > 收件人: "dev" > 发送时间: 星期四, 2022年 12 月 29日 下午 7:45:36 > 主题: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default scheduler for > batch jobs > > Hi, devs, > > I'd like to start a discussion about FLIP-283: Use adaptive batch > scheduler as default scheduler for batch jobs[1]. > > In FLIP-187, we introduced an adaptive batch scheduler. The adaptive > batch scheduler has stronger batch scheduling capabilities, including > automatically deciding parallelisms of job vertices for batch > jobs (FLIP-187)[2], data balanced distribution (FLINK-29663)[3], > and speculative execution (FLIP-168)[4]. To further use the adaptive > batch scheduler to improve flink's batch capability, in this FLIP > we aim to make the adaptive batch scheduler as the default batch > scheduler. > > Currently, users have to set some configuration of the adaptive > batch scheduler, which is not very convenient. To use the adaptive > batch scheduler as the default batch scheduler, we need to improve > the user's out-of-the-box experience. Therefore, we also need to > optimize the current adaptive batch scheduler configuration. > > Looking forward to your feedback. > > [1]: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-283%3A+Use+adaptive+batch+scheduler+as+default+scheduler+for+batch+jobs > [2]: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-187%3A+Adaptive+Batch+Scheduler > [3]:https://issues.apache.org/jira/browse/FLINK-29663 > [4]: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+Execution+for+Batch+Job > > Best regards, > JunRui Lee
Re: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs
Hi, JunRui Lee. Thanks for driving this FLIP. It must a good improvement for batch users' experiences. I have few questions about this FLIP: 1: About the configuration renaming. The old configurations will be deprecated or removed directly? if user upgrade their Flink version, these old configuration will still be considered or just ignored? If ignore, the users may need to modify their configurations after they upgrade their Flink. 2: I'm cursion in which case users will disable auto parallelism derivation if they have enabled adaptive batch scheduler. IIUC, auto parallelism derivation is what adaptive batch scheduler aim to do. If use want to diable auto parallelism derivation, can they just disable adaptive batch scheduler.? Best regards, Yuxia - 原始邮件 - 发件人: "JunRui Lee" 收件人: "dev" 发送时间: 星期四, 2022年 12 月 29日 下午 7:45:36 主题: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs Hi, devs, I'd like to start a discussion about FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs[1]. In FLIP-187, we introduced an adaptive batch scheduler. The adaptive batch scheduler has stronger batch scheduling capabilities, including automatically deciding parallelisms of job vertices for batch jobs (FLIP-187)[2], data balanced distribution (FLINK-29663)[3], and speculative execution (FLIP-168)[4]. To further use the adaptive batch scheduler to improve flink's batch capability, in this FLIP we aim to make the adaptive batch scheduler as the default batch scheduler. Currently, users have to set some configuration of the adaptive batch scheduler, which is not very convenient. To use the adaptive batch scheduler as the default batch scheduler, we need to improve the user's out-of-the-box experience. Therefore, we also need to optimize the current adaptive batch scheduler configuration. Looking forward to your feedback. [1]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-283%3A+Use+adaptive+batch+scheduler+as+default+scheduler+for+batch+jobs [2]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-187%3A+Adaptive+Batch+Scheduler [3]:https://issues.apache.org/jira/browse/FLINK-29663 [4]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+Execution+for+Batch+Job Best regards, JunRui Lee
[DISCUSS] FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs
Hi, devs, I'd like to start a discussion about FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs[1]. In FLIP-187, we introduced an adaptive batch scheduler. The adaptive batch scheduler has stronger batch scheduling capabilities, including automatically deciding parallelisms of job vertices for batch jobs (FLIP-187)[2], data balanced distribution (FLINK-29663)[3], and speculative execution (FLIP-168)[4]. To further use the adaptive batch scheduler to improve flink's batch capability, in this FLIP we aim to make the adaptive batch scheduler as the default batch scheduler. Currently, users have to set some configuration of the adaptive batch scheduler, which is not very convenient. To use the adaptive batch scheduler as the default batch scheduler, we need to improve the user's out-of-the-box experience. Therefore, we also need to optimize the current adaptive batch scheduler configuration. Looking forward to your feedback. [1]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-283%3A+Use+adaptive+batch+scheduler+as+default+scheduler+for+batch+jobs [2]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-187%3A+Adaptive+Batch+Scheduler [3]:https://issues.apache.org/jira/browse/FLINK-29663 [4]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+Execution+for+Batch+Job Best regards, JunRui Lee