Hi Pradeepta, Generally, the proposal looks good to me. The only issue I see is that it is fairly generic. On the one hand, it allows plugging in additional evaluation logic into the autoscaler, which allows for many new ways to extend the current autoscaling logic. On the other hand, normal users won't benefit from this. I wish we could build some predictive logic directly into the autoscaler. So far, we refrained from doing so because we wanted to get the basics right first (something that most Flink autoscaling solutions never got around to).
+1 for the proposal as it is meaningful. Hopefully, we can also make some progress towards a built-in predictive logic as part of a new FLIP :) -Max On Thu, Mar 20, 2025 at 8:28 AM Gyula Fóra <gyula.f...@gmail.com> wrote: > > Hi! > > I think this is a very nice proposal, +1 from me! > > The only practical challenge is to actually be able to identify what job > vertex is what in the job graph to make good predictions / feed in external > information but this is definitely out of scope here. > The autoscaler in general needs to have a good way to associate user code / > operators with vertex ids for these types of use-cases that we need to > investigate as a followup. > > Cheers > Gyula > > On Wed, Mar 12, 2025 at 6:24 PM Pradeepta Choudhury > <pchoudhur...@apple.com.invalid> wrote: > > > Hi All, > > > > I would like to start a discussion of FLIP-514: Custom Evaluator plugin > > for Flink Autoscaler [1]. > > > > Feel free to share your thoughts and suggestions to make this feature > > better. > > > > > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-514%3A+Custom+Evaluator+plugin+for+Flink+Autoscaler > > > > > > > > Thanks > > Pradeepta
