Hi Everyone, I would like to start a discussion on FLIP-XXX: Dynamic Rate Limiting for FLIP-27 Source [1].
Flink's FLIP-27 Source API already supports rate limiting (RateLimiter / RateLimiterStrategy / RateLimitedSourceReader in flink-core), but the limit is static: it's fixed once at source construction time and has no visibility into how the job is actually running. Two pain points observed in this static rate limit feature. 1. It is hard to set up a static limit for different scenarios, for example, periodical traffic peak, regional failover situation, and etc. 2. For pipelines that throughput is bounded by in process RPC or downstream service. Users prefer to slow down the process rather than impact these services or job failure due to checkpoint timeout. The proposal aims to add features below to improve the resiliency of Flink application with adjustable ratelimit with the feedback loop from checkpoint time: 1. Add a dynamic rate limit to the Source interface that can adjust itself automatically, using the historical checkpoint completion time tracked by the SourceCoordinator as the feedback signal (checkpoints getting slower → back off; checkpoints healthy → ramp back up). 2. Expose the tuning knobs as standard Flink ConfigOptions instead of a hand-rolled parameters object, so the feature is configurable the same way as any other Flink job setting. 3. Base the actual rate adjustment on the source operator's current observed throughput, not on the previously-broadcast target rate, so a throttle/ramp decision takes effect immediately instead of compounding off a stale number. Looking forward to your feedback! Best Regards Peter Huang [1] https://docs.google.com/document/d/1oOUBY6cbLnbvA3Xs9Hm8irvxStPHy5jIEjFec_46rNg/edit?tab=t.0#heading=h.exegx0l01gnh
