Hi Yuepeng, thanks for reminding me of this FLIP. I went over it and have a few items which we might need to address before we can actually finalize the vote:
1. You mention a few options for when it comes to storing the data which is good. The FLIP doesn't point out, though, what option you're going to go for as part of this FLIP (as far as I can see). It would be good to only outline the option to go for in the FLIP and list the other options as rejected alternatives (with the pro's and con's). I think it make sense to go for option 3 (i.e. following what's done for the ExecutionGraphInfoStore for now). The other options can be considered as a follow-up. 2. About the terminal states of a rescaling (i.e. IGNORED, FAILED, COMPLETED): Can we we clarify in the FLIP under what conditions the rescaling transitions into each of the three terminal states? 3. The section "The information to record in a rescale event" could be restructured in four sections (to remove redundancy): a) The IDs (Rescale ID, resourceRequirementsEpochID, subRescaleIdOfResourceRequirementsEpochID): What about making these names easier to read: GlobalRescaleID, RescaleUUID, RescaleAttemptId) b) Per-vertex data which includes: JobVertexID, JobVertexName, SlotSharingGroupId, the different parallelisms (pre-rescale, sufficient, desired, post-rescale) c) The SlotSharingGroup information: SlotSharingGroupId, name, ResourceProfile d) Other information: Timestamps of state transitions, etc. as laid out in the FLIP already 4. The FLIP doesn't explain how the data is passed through the AdaptiveScheduler states. We should be handling some kind of RescaleSnapshot that is passed through the different states and updated and its final state is stored somewhere within AdaptiveScheduler in the end, I guess. Can we clarify that in the FLIP? 5. You mention the config parameters for the cache in the public interface section. But there's no mentioning of any caching and how that is used within the FLIP. 6. The REST endpoint is probably better suited in FLIP-487. FLIP-495 should be about the actual implementation details and how the data is stored internally whereas FLIP-487 is about exposing the information to the outside through the REST API and the Flink UI. That would be a way to decrease the scope of FLIP-495. WDYT? Best, Matthias On Mon, Mar 24, 2025 at 11:37 AM Yuepeng Pan <panyuep...@apache.org> wrote: > Hi, Community, > > There haven’t been any further responses to this email over the past few > days. > I'd like to initiate a vote on the current proposal[1] in the next few > days. > Please rest assured that I’m proceeding cautiously and not rushing the > process. > If there are any concerns about this FLIP-495[1], > I will gladly pause and make the adjustments. > > Best regards, > Yuepeng Pan > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history > > > On 2024/12/17 15:18:45 Yuepeng Pan wrote: > > Hi community, > > > > > > > > > > We discussed several aspects of FLIP-487[1] 'Show history of rescales in > Web UI for AdaptiveScheduler' > > and received a lot of valuable feedback. Based on the suggestions from > the email thread[2], > > we plan to split the original proposal for FLIP-487[1]. > > > > > > > > > > The current email thread and the FLIP-495[3] wiki will be used to > discuss 'Support AdaptiveScheduler in recording and querying the rescale > history', > > while FLIP-487[1] will primarily focus on displaying-related design > content > > > > > > > > > > Looking forward to any feedback and opinions on FLIP-495[3]. > > > > > > > > > > [1] > https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-487%3A+Show+history+of+rescales+in+Web+UI+for+AdaptiveScheduler > > > > [2] https://lists.apache.org/thread/f4md4btkf006mxcxf66bng1kfz0rsn8c > > > > [3] > https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history > > > > > > > > > > Thank you very much. > > > > > > > > > > Best, > > > > Regards. > > > > Yuepeng Pan >
