Hi Community, Our org has been heavily using the Flink autoscaling algorithm. It greatly reduced our operation overhead and improved cost efficiency as users always over provision resources when onboard. Recently, we have had some requirements to customize the auto scaling algorithm for different scenarios, for example, during the holiday season large but predictable traffic spike, increase checkpoint interval together with scale up for streaming ingestion use cases.
We search through the discussion about the topic in the mail list including the existing FLIP-514 <https://cwiki.apache.org/confluence/display/FLINK/FLIP-514%3A+Custom+Evaluator+plugin+for+Flink+Autoscaler>. Looks like the discussion is not finalized yet. To accelerate the process, we adopt and combine the existing opinions from the community and create a proposal in FLIP-543 <https://cwiki.apache.org/confluence/display/FLINK/FLIP-543%3A+Support+Customized+Autoscale+Algorithm>. The basic idea is to make some core components of autoscaler pluggable, for example, MetricsCollector, Metrics Evaluator, and ScalingRealizer, at the same keep the core logic skeleton (which is already well justified in large amount of users) of autoscaler untouched. Looking forward to any feedback and opinions on FLIP-543. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-543%3A+Support+Customized+Autoscale+Algorithm [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-514%3A+Custom+Evaluator+plugin+for+Flink+Autoscaler [3] Other related discussion thread https://lists.apache.org/thread/749l74z1h5jylkxrw3rtjmxcj2t9p7ws https://lists.apache.org/thread/mcd7jcn4kz6oqtyqq5hfycjf9mqh6c53 Best Regards Peter Huang
