Hi Steve, If your intended implementation involves changes to the Source API, it should certainly be a separate FLIP. Looking forward to your proposals.
Best, Alexandere Fedulov On Thu, Jul 14, 2022 at 7:18 AM Steve Yurong Su <r...@apache.org> wrote: > Hi Alexander, > > The reference [3] mentioned in my previous email is your discussion > thread on FLIP-238. We have noticed your idea of providing a new > rate-limiter, which is great and matches our starting point. I was > just about to chat with you on slack about this feature :) > > I tend to propose a separate FLIP for the following reasons: > 1. Source rate limiting is an independent feature and involves changes > to the Source API. > 2. Source rate limiting is a much more important feature. Projects > such as Flink CDC can directly benefit from this feature, and a > separate FLIP will help its implementation and application. > 3. The function of the source rate-limiter may go further, it can > provide not only limits on throughput but also limits on the number of > requests and so on (perhaps your implementation is enough for now), > which can better serve different types of sources. > > What do you think? Is it necessary to raise a FLIP separately? If > necessary, you can also write the first version of the proposal for > source rate-limiting, and then I will add some of my ideas :) > > Thanks, > Steve > > Alexander Fedulov <alexan...@ververica.com> 于2022年7月14日周四 00:47写道： > > > > Hi Steve, > > > > in the FLIP-238 [1] I propose to also introduce a rate limiter with a > > simple API at the level of SourceReader: > > RateLimiter.java > > < > https://github.com/afedulov/flink/blob/FLINK-27919-generator-source/flink-core/src/main/java/org/apache/flink/api/common/io/ratelimiting/RateLimiter.java > > > > GuavaRateLimiter.java > > < > https://github.com/afedulov/flink/blob/FLINK-27919-generator-source/flink-core/src/main/java/org/apache/flink/api/common/io/ratelimiting/GuavaRateLimiter.java > > > > RateLimitedSourceReader.java > > < > https://github.com/afedulov/flink/blob/FLINK-27919-generator-source/flink-core/src/main/java/org/apache/flink/api/connector/source/lib/util/RateLimitedSourceReader.java > > > > DataGeneratorSourceV3.java#L140 > > < > https://github.com/afedulov/flink/blob/FLINK-27919-generator-source/flink-core/src/main/java/org/apache/flink/api/connector/source/lib/DataGeneratorSourceV3.java#L140 > > > > > > Did you plan to go further than this? > > > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-238%3A+Introduce+FLIP-27-based+Data+Generator+Source > > > > Best, > > Alexander Fedulov > > > > On Wed, Jul 13, 2022 at 6:38 PM Steve Yurong Su <r...@apache.org> wrote: > > > > > Hi Jark, > > > > > > Thanks a lot :) > > > > > > Steve > > > > > > Jark Wu <imj...@gmail.com> 于2022年7月13日周三 22:48写道： > > > > > > > > Hi Steve, > > > > > > > > Welcome to the Flink community. I have granted the edit permission to > > > you. > > > > Looking forward to your FLIP. > > > > > > > > Best, > > > > Jark > > > > > > > > On Wed, 13 Jul 2022 at 19:56, Steve Yurong Su <r...@apache.org> > wrote: > > > > > > > > > Hi folks! > > > > > > > > > > My name is Steve Su, and I am currently a third-year master’s > student > > > > > at Tsinghua University. > > > > > > > > > > I am now participating in a Flink-related student project in OSPP > > > > > (Open Source Promotion Plan[1]). The project aims to design and > > > > > implement the generic rate limiter for the FLIP-27 source and apply > > > > > the rate-limiting feature to the Flink CDC project. @Leonard Xu is > my > > > > > mentor in this project. > > > > > > > > > > We already had many discussions on source rate-limiting on the > mailing > > > > > list, such as [2], [3], and so on. Leonard and I had an offline > > > > > discussion, and we thought that it would be great to have a generic > > > > > rate limiter, so I’d like to raise a new FLIP to illustrate the > design > > > > > further. > > > > > > > > > > Please grant my account FLIP edit permissions. My apache confluence > > > > > account id is `steveyurongsu`. :D > > > > > > > > > > Thanks, > > > > > Steve Su > > > > > --- > > > > > [1] https://summer-ospp.ac.cn/homepagy > > > > > [2] > https://lists.apache.org/thread/ff6mcos8g4otnhjrp030lrcrf3omgfmf > > > > > [3] > https://lists.apache.org/thread/7gjxto1rmkpff4kl54j8nlg5db2rqhkt > > > > > > > > >
