thinkharderdev commented on pull request #1935: URL: https://github.com/apache/arrow-datafusion/pull/1935#issuecomment-1063780420
> Hi @thinkharderdev, you are right. For #1908, #1909, #1910 and #1934, these 4 issues are mainly focusing on the refactoring and changing the scheduler server state changing to be event-based. > > While for #1936, I'll introduce stage and the state machine for it. With the state machine of the job stage, it will be much easier and more efficient to do error handling, speculative execution. In general, event-based processing is very suitable for handling the state machine. > > However, the PR for #1936 is really complicated. I'm still working on it. Cool. Looks good to me in general but would like to see the other PRs merged before approving. For work like this in the future (large refactorings broken into multiple PRs which depend on each other) it might be a good idea to do something similar to the `arrow2` branch. That is, we can create a feature branch for the overall refactoring and raise the incremental work as PRs to the feature branch. It is a little more annoying to manage but it makes the individual PRs much easier to review. This basically approximates a "stacked diffs" (https://kurtisnusbaum.medium.com/stacked-diffs-keeping-phabricator-diffs-small-d9964f4dcfa6) model. @alamb @alamb @yjshen @realno @matthewmturner and @liukun4515 Does ^^ make sense? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
