Hi Yupeng, I managed to find some time to respond. See my answers inlined below.
Matthias On Fri, Jan 3, 2025 at 11:48 AM Yuepeng Pan <panyuep...@apache.org> wrote: > [...] > Sorry for not expressing this part clearly earlier. > IIUC, based on the Adaptive Scheduler state diagram [1], > when a stop-with-savepoint operation fails but the job is restartable, > it transitions to the WaitingForResources state, which implies a > potential rescaling process may trigger. > Yes, you're right. That's a scenario that should be covered by the FLIP but doesn't include having the Executing state as an initial AdaptiveScheduler state. That's also a corner case which is not covered by the current implementation. ...at least we're not considering any job resource requirement updates during StopWithSavepoint. I'm wondering whether that could be even considered a bug that manifests itself if the stopWithSavepoint mechanism fails and the job is executed. > From the current logic, a rescale event may be triggered under the > following circumstances, called 'rescale triggers' here: > - updateJobResourceRequirement > - Restart due to recoverable failure > - newResourceAvailable > > From Fig [1], the entire chain of rescale-related scheduler > states typically involves several loops: > - WaitingForResources -> CreatingExecutionGraph -> Executing -> > StopWithSavepoint (error & restartable) -> Restarting -> WaitingForResources > - CreatingExecutionGraph [-> WaitingForResources -> > CreatingExecutionGraph](optinal loop ) -> Executing(rescale triggers) -> > Restarting -> CreatingExecutionGraph > > Following your shared ideas: > Based on what I pointed out in my previous > post, I would think that a rescale operation has its > starting point in > the AdaptiveScheduler's Executing state (i.e. when the job is running). I > attempted to interpret the boundary definition of rescale events. > > The historical states of the Adaptive Scheduler during amsuccessful > rescale event would likely match one of the following patterns ? > - (Starting) Executing(rescale triggers) -> Restarting -> > CreatingExecutionGraph [ -> WaitingForResources -> CreatingExecutionGraph] > -> (Ending) Executing > - (Starting) Executing -> StopWithSavepoint (error & restartable) -> > Restarting -> WaitingForResources -> CreatingExecutionGraph [ -> > WaitingForResources -> CreatingExecutionGraph] -> (Ending) Executing > > If my understanding is incorrect, please feel free to correct me. > No, I see your point now. I was too focused on the Executing state. But you're right: WaitingForRescale can also be part of error handling that happened while Executing the job or trying to stop the job (i.e. StopWithSavepoint). These cases need to be considered when collecting the rescale events. Sorry for not grasping your intention earlier and thanks for your clarification. It might be useful to make this clearer in the FLIP as well (i.e. adding more context aside from the bullet points). 2 - About the status of a rescale event. > > - Rescale status: I would say that this section needs to be reworked. > Can I understand that Rescale Event does indeed need some status > information, > such as FAILED, IGNORED, SUCCESS, and PENDING, etc, > to indicate the final status of a completed event or the current status of > an ongoing event? > In other words, the current status fields and their associations are > unreasonable, > so its need to be redesigned rather than discarding this descriptive > mechanism. > Sounds reasonable. > [1] > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#id-[WIP]FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-CurrentadaptiveschedulerstatetransitionFig > . > > Thank you. > > Best, > Yuepeng >
