Hi everyone, thank you Jing for redirecting the discussion back to the topic at hand. I agree with all of your points.
+1 to deprecate SourceFunction Is there really no replacement for the StreamExecutionEnvironment#readXXX. There is already a FLIP-27 based FileSource, right? What's missing to recommend using that as opposed to the the readXXX methods? Cheers, Konstantin Am Do., 9. Juni 2022 um 20:11 Uhr schrieb Alexander Fedulov < alexan...@ververica.com>: > Hi all, > > It seems that there is some understandable cautiousness with regard to > deprecating methods and subclasses that do not have alternatives just yet. > > We should probably first agree if it is in general OK for Flink to use > @Deprecated > annotation for parts of the code that do not have alternatives. In that > case, > we could add a comment along the lines of: > "This implementation is based on a deprecated SourceFunction API that > will gradually be phased out from Flink. No direct substitute exists at the > moment. > If you want to have a more future-proof solution, consider helping the > project by > contributing an implementation based on the new Source API." > > This should clearly communicate the message that usage of these > methods/classes > is discouraged and at the same time promote contributions for addressing > the gap. > What do you think? > > Best, > Alexander Fedulov > > > On Thu, Jun 9, 2022 at 6:27 PM Ingo Bürk <airbla...@apache.org> wrote: > > > Hi, > > > > these APIs don't expose the underlying source directly, so I don't think > > we need to worry about deprecating them as well. There's also nothing > > inherently wrong with using a deprecated API internally, though even > > just for the experience of using our own new APIs I would personally say > > that they should be migrated to the new Source API. It's hard to reason > > that users must migrate to a new API if we don't do it internally as > well. > > > > > > Best > > Ingo > > > > On 09.06.22 15:41, Lijie Wang wrote: > > > Hi Martijn, > > > > > > I don't mean it's a blocker. Just a information. And I'm also +1 for > > this. > > > > > > Put it another way: should we migrate the `#readFile(...)` to new API > or > > > provide a similar method "readxxx“ based on the new Source API? > > > > > > And if we don't migrate it, does it mean that the `#readFile(...)` > should > > > also be marked as deprecated? > > > > > > Best, > > > Lijie > > > > > > Martijn Visser <martijnvis...@apache.org> 于2022年6月9日周四 21:03写道： > > > > > >> Hi Lijie, > > >> > > >> I don't see any problem with deprecating those methods at this moment, > > as > > >> long as we don't remove them until the replacements are available. > > Besides > > >> that, are we sure there are no replacements already, especially with > the > > >> new FileSource? > > >> > > >> Best regards, > > >> > > >> Martijn > > >> > > >> Op do 9 jun. 2022 om 14:23 schreef Lijie Wang < > wangdachui9...@gmail.com > > >: > > >> > > >>> Hi all, > > >>> > > >>> FYI, currently, some commonly used methods in > > StreamExecutionEnvironment > > >>> are still based on the old SourceFunction (and there is no > > alternative): > > >>> `StreamExecutionEnvironment#readFile(...)` > > >>> `StreamExecutionEnvironment#readTextFile(...)` > > >>> > > >>> I think these should be migrated to the new source API before > deprecate > > >> the > > >>> SourceFunction. > > >>> > > >>> Best, > > >>> Lijie > > >>> > > >>> Martijn Visser <martijnvis...@apache.org> 于2022年6月9日周四 16:05写道： > > >>> > > >>>> Hi all, > > >>>> > > >>>> I think implicitly we've already considered the SourceFunction and > > >>>> SinkFunction as deprecated. They are even marked as so on the Flink > > >>> roadmap > > >>>> [1]. That also shows that connectors that are using these interfaces > > >> are > > >>>> either approaching end-of-life. The fact that we're actively > migrating > > >>>> connectors from Source/SinkFunction to FLIP-27/FLIP-143 (plus add-on > > >>> FLIPs) > > >>>> shows that we've already determined that target. > > >>>> > > >>>> With regards to the motivation of FLIP-27, I think reading up on the > > >>>> original discussion thread is also worthwhile [2] to see more > context. > > >>>> FLIP-27 was also very important as it brought a unified connector > > which > > >>> can > > >>>> support both streaming and batch (with batch being considered a > > special > > >>>> case of streaming in Flink's vision). > > >>>> > > >>>> So +1 to deprecate SourceFunction. I would also argue that we should > > >>>> already mark the SinkFunction as deprecated to avoid having this > > >>> discussion > > >>>> again in a couple of months. > > >>>> > > >>>> Best regards, > > >>>> > > >>>> Martijn > > >>>> > > >>>> [1] https://flink.apache.org/roadmap.html > > >>>> [2] > https://lists.apache.org/thread/334co89dbhc8qpr9nvmz8t1gp4sz2c8y > > >>>> > > >>>> Op do 9 jun. 2022 om 09:48 schreef Jing Ge <j...@ververica.com>: > > >>>> > > >>>>> Hi, > > >>>>> > > >>>>> I am very happy to see opinions from different perspectives. That > > >> will > > >>>> help > > >>>>> us understand the problem better. Thanks all for the informative > > >>>>> discussion. > > >>>>> > > >>>>> Let's see the big picture and check following facts together: > > >>>>> > > >>>>> 1. FLIP-27 was intended to solve some technical issues that are > very > > >>>>> difficult to solve with SourceFunction[1]. When we say > > >> "SourceFunction > > >>> is > > >>>>> easy", well, it depends. If we take a look at the implementation of > > >> the > > >>>>> Kafka connector, we will know how complicated it is to build a > > >> serious > > >>>>> connector for production with the old SourceFunction. To every > > >> problem > > >>>>> there is a solution and to every solution there is a problem. The > > >> fact > > >>> is > > >>>>> that there is no perfect but a feasible solution. If we try to > solve > > >>>>> complicated problems, we have to expose some complexity. Comparing > to > > >>>>> connectors for POC, demo, training(no offense), I would also solve > > >>> issues > > >>>>> for connectors like Kafka connector that are widely used in > > >> production > > >>>> with > > >>>>> higher priority. I think that should be one reason why FLIP-27 has > > >> been > > >>>>> designed and why the new source API went public. > > >>>>> > > >>>>> 2. FLIP-27 and the implementation was introduced roughly at the end > > >> of > > >>>> 2019 > > >>>>> and went public on 19.04.2021, which means Flink has provided two > > >>>> different > > >>>>> public/graduated source solutions for more than one year. On the > day > > >>> that > > >>>>> the new source API went public, there should be a consensus in the > > >>>>> community that we should start the migration. Old SourceFunction > > >>>> interface, > > >>>>> in the ideal case, should have been deprecated on that day, > otherwise > > >>> we > > >>>>> should not graduate the new source API to avoid confusing > (connector) > > >>>>> developers[2]. > > >>>>> > > >>>>> 3. It is true that the new source API is hard to understand and > even > > >>> hard > > >>>>> to implement for simple cases. Thanks for the feedback. That is > > >>> something > > >>>>> we need to improve. The current design&implementation could be > > >>> considered > > >>>>> as the low level API. The next step is to create the high level API > > >> to > > >>>>> reduce some unnecessary complexity for those simple cases. But, > IMHO, > > >>>> this > > >>>>> should not be the prerequisite to postpone the deprecation of the > old > > >>>>> SourceFunction APIs. > > >>>>> > > >>>>> 4. As long as the old SourceFunction is not marked as deprecated, > > >>>>> developers will continue asking which one should be used. Let's > make > > >> a > > >>>>> concrete example. If a new connector is developed now and the > > >> developer > > >>>>> asks for a suggestion of the choice between the old and new source > > >> API > > >>> on > > >>>>> the ML, which one should we suggest? I think it should be the new > > >>> Source > > >>>>> API. If a fresh new connector has been developed with the old > > >>>>> SourceFunction API before asking for the consensus in the community > > >> and > > >>>> the > > >>>>> developer wants to merge it to the master. Should we allow it? If > the > > >>>>> answer of all these questions is pointing to the new Source API, > the > > >>> old > > >>>>> SourceFunction is de facto already deprecated, just has not been > > >> marked > > >>>> as > > >>>>> @deprecated, which confuses developers even more. > > >>>>> > > >>>>> Best regards, > > >>>>> Jing > > >>>>> > > >>>>> [1] > > >>>>> > > >>>>> > > >>>> > > >>> > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface > > >>>>> [2] > https://lists.apache.org/thread/7okp4y46n3o3rx5mn0t3qobrof8zxwqs > > >>>>> > > >>>>> On Wed, Jun 8, 2022 at 2:21 AM Alexander Fedulov < > > >>>> alexan...@ververica.com> > > >>>>> wrote: > > >>>>> > > >>>>>> Hey Austin, > > >>>>>> > > >>>>>> Since we are getting deeper into the implementation details of the > > >>>>>> DataGeneratorSource > > >>>>>> and it is not the main topic of this thread, I propose to move our > > >>>>>> discussion to where it belongs: [DISCUSS] FLIP-238 [1]. Could you > > >>>> please > > >>>>>> briefly formulate your requirements to make it easier for the > > >> others > > >>> to > > >>>>>> follow? I am happy to continue this conversation there. > > >>>>>> > > >>>>>> [1] > > >> https://lists.apache.org/thread/7gjxto1rmkpff4kl54j8nlg5db2rqhkt > > >>>>>> > > >>>>>> Best, > > >>>>>> Alexander Fedulov > > >>>>>> > > >>>>>> On Tue, Jun 7, 2022 at 6:14 PM Austin Cawley-Edwards < > > >>>>>> austin.caw...@gmail.com> wrote: > > >>>>>> > > >>>>>>>> @Austin, in the FLIP I mentioned above [1], the user is > > >> expected > > >>> to > > >>>>>>> pass a MapFunction<Long, > > >>>>>>> OUT> > > >>>>>>> to the generator. I wonder if you could have your external client > > >>> and > > >>>>>>> polling logic wrapped in a custom > > >>>>>>> MapFunction implementation class? Would that answer your needs or > > >>> do > > >>>>> you > > >>>>>>> have some > > >>>>>>> more sophisticated scenario in mind? > > >>>>>>> > > >>>>>>> At first glance, the FLIP looks good but for this case in regards > > >>> to > > >>>>> the > > >>>>>>> map function, but leaves out 1) ability to control polling > > >>> intervals > > >>>>> and > > >>>>>> 2) > > >>>>>>> ability to produce an unknown number of records, both per-poll > > >> and > > >>>>>> overall > > >>>>>>> boundedness. Do you think something like this could be built from > > >>> the > > >>>>>> same > > >>>>>>> pieces? > > >>>>>>> I'm also wondering what handles threading, is that on the user or > > >>> is > > >>>>> that > > >>>>>>> part of the DataGeneratorSource? > > >>>>>>> > > >>>>>>> Best, > > >>>>>>> Austin > > >>>>>>> > > >>>>>>> On Tue, Jun 7, 2022 at 9:34 AM Alexander Fedulov < > > >>>>>> alexan...@ververica.com> > > >>>>>>> wrote: > > >>>>>>> > > >>>>>>>> Hi everyone, > > >>>>>>>> > > >>>>>>>> Thanks for all the input and a lively discussion. It seems that > > >>>> there > > >>>>>> is > > >>>>>>> a > > >>>>>>>> consensus that due to > > >>>>>>>> the inherent complexity of FLIP-27 sources we should provide > > >> more > > >>>>>>>> user-facing utilities to bridge > > >>>>>>>> the gap between the existing SourceFunction-based functionality > > >>> and > > >>>>> the > > >>>>>>> new > > >>>>>>>> APIs. > > >>>>>>>> > > >>>>>>>> To start addressing this I picked the issue that David raised > > >> and > > >>>>> many > > >>>>>>>> upvoted. Here is a proposal > > >>>>>>>> for the new DataGeneratorSource: FLIP-238 [1]. Please take a > > >>>> look, I > > >>>>>> am > > >>>>>>>> going to open a separate > > >>>>>>>> discussion thread on it shortly. > > >>>>>>>> > > >>>>>>>> Jing also raised some great points regarding the interfaces and > > >>>>>>> subclasses. > > >>>>>>>> It seems to me that > > >>>>>>>> what might actually help is some sort of a "soft deprecation" > > >>>> concept > > >>>>>> and > > >>>>>>>> annotation. It could be > > >>>>>>>> used in places where we do not have an alternative > > >> implementation > > >>>>> yet, > > >>>>>>> but > > >>>>>>>> we clearly want > > >>>>>>>> to indicate that continuing to build on top of these interfaces > > >>> is > > >>>>>>>> discouraged. The area of > > >>>>>>>> impact of deprecating all SourceFunction subclasses is rather > > >>> big, > > >>>>> and > > >>>>>> we > > >>>>>>>> can expect it to > > >>>>>>>> take a while. The hope would be that if in the meantime someone > > >>>> finds > > >>>>>>>> themselves using one of > > >>>>>>>> such old APIs, the "soft deprecation" annotation will be a > > >> clear > > >>>>>>> indication > > >>>>>>>> and encouragement to > > >>>>>>>> work on introducing an alternative FLIP-27-based implementation > > >>>>>> instead. > > >>>>>>>> > > >>>>>>>> @Austin, in the FLIP I mentioned above [1], the user is > > >> expected > > >>> to > > >>>>>>>> pass a MapFunction<Long, > > >>>>>>>> OUT> > > >>>>>>>> to the generator. I wonder if you could have your external > > >> client > > >>>> and > > >>>>>>>> polling logic wrapped in a custom > > >>>>>>>> MapFunction implementation class? Would that answer your needs > > >> or > > >>>> do > > >>>>>> you > > >>>>>>>> have some > > >>>>>>>> more sophisticated scenario in mind? > > >>>>>>>> > > >>>>>>>> [1] https://cwiki.apache.org/confluence/x/9Av1D > > >>>>>>>> Best, > > >>>>>>>> Alexander Fedulov > > >>>>>>>> > > >>>>>>>> On Mon, Jun 6, 2022 at 7:08 PM Austin Cawley-Edwards < > > >>>>>>>> austin.caw...@gmail.com> wrote: > > >>>>>>>> > > >>>>>>>>> Thanks for the nice discussion all. > > >>>>>>>>> > > >>>>>>>>> I was recently trying to implement a very simple polling > > >> source > > >>>> and > > >>>>>>>>> would've loved a higher-level base to work from. I'm > > >> wondering > > >>> if > > >>>>> in > > >>>>>>>>> addition to the data generator use cases, it would be good to > > >>>>>> support a > > >>>>>>>>> simple non-parallel polling abstraction to make it easier to, > > >>> for > > >>>>>>>> instance, > > >>>>>>>>> start prototyping with data in existing APIs without adding a > > >>>> Kafka > > >>>>>> or > > >>>>>>>> such > > >>>>>>>>> in the middle. > > >>>>>>>>> > > >>>>>>>>> Best, > > >>>>>>>>> Austin > > >>>>>>>>> > > >>>>>>>>> On Mon, Jun 6, 2022 at 10:02 AM tison <wander4...@gmail.com> > > >>>>> wrote: > > >>>>>>>>> > > >>>>>>>>>> Well. It's a bit off-topic. For deprecating SourceFunction > > >> as > > >>>>>> FLIP-27 > > >>>>>>>>>> series works go ahead, +1 from my side. It's a significant > > >>> work > > >>>>>>> towards > > >>>>>>>>> the > > >>>>>>>>>> unification of batch and streaming effort :) > > >>>>>>>>>> > > >>>>>>>>>> Best, > > >>>>>>>>>> tison. > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> tison <wander4...@gmail.com> 于2022年6月6日周一 21:54写道： > > >>>>>>>>>> > > >>>>>>>>>>> The starting point of the version bump and removal > > >> question > > >>>> is > > >>>>>> that > > >>>>>>>>>>> downstream projects may experience a tough time to adapt > > >>> new > > >>>>>>>> interfaces > > >>>>>>>>>>> while Flink keeps in 1.x versions so that users may > > >> expect > > >>> it > > >>>>> as > > >>>>>> an > > >>>>>>>>> easy > > >>>>>>>>>>> task. From my experience, it's really challenge to > > >> maintain > > >>>>>>>>>>> compatibility between multiple versions of Flink while > > >>>>>> significant > > >>>>>>>>>> changes > > >>>>>>>>>>> made but sharing 1.x version series - users may not be > > >>> aware > > >>>>> that > > >>>>>>>> it's > > >>>>>>>>>>> almost a major version bump. > > >>>>>>>>>>> > > >>>>>>>>>>> Best, > > >>>>>>>>>>> tison. > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> tison <wander4...@gmail.com> 于2022年6月6日周一 21:51写道： > > >>>>>>>>>>> > > >>>>>>>>>>>> One question from my side: > > >>>>>>>>>>>> > > >>>>>>>>>>>> As SourceFunction a @Public interface, we cannot remove > > >> it > > >>>>>> before > > >>>>>>>>> doing > > >>>>>>>>>>>> a major version bump (Flink 2.0). > > >>>>>>>>>>>> > > >>>>>>>>>>>> Of course it's not a blocker to make such deprecation > > >> and > > >>>> let > > >>>>>> the > > >>>>>>>> new > > >>>>>>>>>>>> interface step in. My question is whether we have a plan > > >>> to > > >>>>>>> finally > > >>>>>>>>>> remove > > >>>>>>>>>>>> the deprecated interfaces, or postpone it until a clear > > >>> plan > > >>>>> of > > >>>>>>>> Flink > > >>>>>>>>>> 2.0? > > >>>>>>>>>>>> > > >>>>>>>>>>>> Best, > > >>>>>>>>>>>> tison. > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> David Anderson <dander...@apache.org> 于2022年6月6日周一 > > >>> 21:35写道： > > >>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> David, can you elaborate why you need watermark > > >>>> generation > > >>>>> in > > >>>>>>> the > > >>>>>>>>>>>>> source > > >>>>>>>>>>>>>> for your data generators? > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> The training exercises should strive to provide > > >> examples > > >>> of > > >>>>>> best > > >>>>>>>>>>>>> practices. > > >>>>>>>>>>>>> If the exercises and their solutions use > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> env.fromSource(source, > > >> WatermarkStrategy.noWatermarks(), > > >>>>>>>>>>>>> "name-of-source") > > >>>>>>>>>>>>> .map(...) > > >>>>>>>>>>>>> .assignTimestampsAndWatermarks(...) > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> this will help establish this anti-pattern as the > > >> normal > > >>>> way > > >>>>> of > > >>>>>>>> doing > > >>>>>>>>>>>>> things. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Most new Flink users are using a KafkaSource with a > > >>>>>> noWatermarks > > >>>>>>>>>> strategy > > >>>>>>>>>>>>> and a SimpleStringSchema, followed by a map that does > > >> the > > >>>>> real > > >>>>>>>>>>>>> deserialization, followed by the real watermarking -- > > >>>> because > > >>>>>>> they > > >>>>>>>>>> aren't > > >>>>>>>>>>>>> seeing examples that teach how these interfaces are > > >> meant > > >>>> to > > >>>>> be > > >>>>>>>> used. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> When we redo the sources used in training exercises, I > > >>> want > > >>>>> to > > >>>>>>>> avoid > > >>>>>>>>>>>>> these > > >>>>>>>>>>>>> pitfalls. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> David > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> On Mon, Jun 6, 2022 at 9:12 AM Konstantin Knauf < > > >>>>>>> kna...@apache.org > > >>>>>>>>> > > >>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> Hi everyone, > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> very interesting thread. The proposal for deprecation > > >>>> seems > > >>>>>> to > > >>>>>>>> have > > >>>>>>>>>>>>> sparked > > >>>>>>>>>>>>>> a very important discussion. Do we what users > > >> struggle > > >>>> with > > >>>>>>>>>>>>> specifically? > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Speaking for myself, when I upgrade flink-faker to > > >> the > > >>>> new > > >>>>>>> Source > > >>>>>>>>> API > > >>>>>>>>>>>>> an > > >>>>>>>>>>>>>> unbounded version of the NumberSequenceSource would > > >>> have > > >>>>> been > > >>>>>>>> all I > > >>>>>>>>>>>>> needed, > > >>>>>>>>>>>>>> but that's just the data generator use case. I think, > > >>>> that > > >>>>>> one > > >>>>>>>>> could > > >>>>>>>>>> be > > >>>>>>>>>>>>>> solved quite easily. David, can you elaborate why you > > >>>> need > > >>>>>>>>> watermark > > >>>>>>>>>>>>>> generation in the source for your data generators? > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Cheers, > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Konstantin > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Am So., 5. Juni 2022 um 17:48 Uhr schrieb Piotr > > >>> Nowojski > > >>>> < > > >>>>>>>>>>>>>> pnowoj...@apache.org>: > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Also +1 to what David has written. But it doesn't > > >>> mean > > >>>> we > > >>>>>>>> should > > >>>>>>>>> be > > >>>>>>>>>>>>>> waiting > > >>>>>>>>>>>>>>> indefinitely to deprecate SourceFunction. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Best, > > >>>>>>>>>>>>>>> Piotrek > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> niedz., 5 cze 2022 o 16:46 Jark Wu < > > >> imj...@gmail.com > > >>>> > > >>>>>>>>> napisał(a): > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> +1 to David's point. > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> Usually, when we deprecate some interfaces, we > > >>> should > > >>>>>> point > > >>>>>>>>> users > > >>>>>>>>>>>>> to > > >>>>>>>>>>>>>> use > > >>>>>>>>>>>>>>>> the recommended alternatives. > > >>>>>>>>>>>>>>>> However, implementing the new Source interface > > >> for > > >>>> some > > >>>>>>>> simple > > >>>>>>>>>>>>>> scenarios > > >>>>>>>>>>>>>>> is > > >>>>>>>>>>>>>>>> too challenging and complex. > > >>>>>>>>>>>>>>>> We also found it isn't easy to push the internal > > >>>>>> connector > > >>>>>>> to > > >>>>>>>>>>>>> upgrade > > >>>>>>>>>>>>>> to > > >>>>>>>>>>>>>>>> the new Source because > > >>>>>>>>>>>>>>>> "FLIP-27 are hard to understand, while > > >>> SourceFunction > > >>>>> is > > >>>>>>>> easy". > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> +1 to make implementing a simple Source easier > > >>> before > > >>>>>>>>> deprecating > > >>>>>>>>>>>>>>>> SourceFunction. > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> Best, > > >>>>>>>>>>>>>>>> Jark > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> On Sun, 5 Jun 2022 at 07:29, Jingsong Lee < > > >>>>>>>>>> lzljs3620...@apache.org > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> +1 to David and Ingo. > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Before deprecate and remove SourceFunction, we > > >>>> should > > >>>>>>> have > > >>>>>>>>> some > > >>>>>>>>>>>>>> easier > > >>>>>>>>>>>>>>>> APIs > > >>>>>>>>>>>>>>>>> to wrap new Source, the cost to write a new > > >>> Source > > >>>> is > > >>>>>> too > > >>>>>>>>> high > > >>>>>>>>>>>>> now. > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Ingo Bürk <airbla...@apache.org>于2022年6月5日 > > >>>>> 周日05:32写道： > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> I +1 everything David said. The new Source > > >> API > > >>>>> raised > > >>>>>>> the > > >>>>>>>>>>>>>> complexity > > >>>>>>>>>>>>>>>>>> significantly. It's great to have such a > > >> rich, > > >>>>>> powerful > > >>>>>>>> API > > >>>>>>>>>>>>> that > > >>>>>>>>>>>>>> can > > >>>>>>>>>>>>>>> do > > >>>>>>>>>>>>>>>>>> everything, but in the process we lost the > > >>>> ability > > >>>>> to > > >>>>>>>>> onboard > > >>>>>>>>>>>>>> people > > >>>>>>>>>>>>>>> to > > >>>>>>>>>>>>>>>>>> the APIs. > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> Best > > >>>>>>>>>>>>>>>>>> Ingo > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> On 04.06.22 21:21, David Anderson wrote: > > >>>>>>>>>>>>>>>>>>> I'm in favor of this, but I think we need > > >> to > > >>>> make > > >>>>>> it > > >>>>>>>>> easier > > >>>>>>>>>>>>> to > > >>>>>>>>>>>>>>>>> implement > > >>>>>>>>>>>>>>>>>>> data generators and test sources. As things > > >>>> stand > > >>>>>> in > > >>>>>>>>> 1.15, > > >>>>>>>>>>>>> unless > > >>>>>>>>>>>>>>> you > > >>>>>>>>>>>>>>>>> can > > >>>>>>>>>>>>>>>>>>> be satisfied with using a > > >>> NumberSequenceSource > > >>>>>>> followed > > >>>>>>>>> by > > >>>>>>>>>> a > > >>>>>>>>>>>>> map, > > >>>>>>>>>>>>>>>>> things > > >>>>>>>>>>>>>>>>>>> get quite complicated. I looked into > > >>> reworking > > >>>>> the > > >>>>>>> data > > >>>>>>>>>>>>>> generators > > >>>>>>>>>>>>>>>> used > > >>>>>>>>>>>>>>>>>> in > > >>>>>>>>>>>>>>>>>>> the training exercises, and got discouraged > > >>> by > > >>>>> the > > >>>>>>>> amount > > >>>>>>>>>> of > > >>>>>>>>>>>>> work > > >>>>>>>>>>>>>>>>>> involved. > > >>>>>>>>>>>>>>>>>>> (The sources used in the training want to > > >> be > > >>>>>>> unbounded, > > >>>>>>>>> and > > >>>>>>>>>>>>> need > > >>>>>>>>>>>>>>>>>>> watermarking in the sources, which means > > >> that > > >>>>> using > > >>>>>>>>>>>>>>>>> NumberSequenceSource > > >>>>>>>>>>>>>>>>>>> isn't an option.) > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> I think the proposed deprecation will be > > >>> better > > >>>>>>>> received > > >>>>>>>>> if > > >>>>>>>>>>>>> it > > >>>>>>>>>>>>>> can > > >>>>>>>>>>>>>>> be > > >>>>>>>>>>>>>>>>>>> accompanied by something that makes > > >>>> implementing > > >>>>> a > > >>>>>>>> simple > > >>>>>>>>>>>>> Source > > >>>>>>>>>>>>>>>> easier > > >>>>>>>>>>>>>>>>>>> than it is now. People are continuing to > > >>>>> implement > > >>>>>>> new > > >>>>>>>>>>>>>>>> SourceFunctions > > >>>>>>>>>>>>>>>>>>> because the interfaces defined by FLIP-27 > > >> are > > >>>>> hard > > >>>>>> to > > >>>>>>>>>>>>> understand, > > >>>>>>>>>>>>>>>> while > > >>>>>>>>>>>>>>>>>>> SourceFunction is easy. Alex, I believe you > > >>>> were > > >>>>>>>> looking > > >>>>>>>>>> into > > >>>>>>>>>>>>>>>>>> implementing > > >>>>>>>>>>>>>>>>>>> an easier-to-use building block that could > > >> be > > >>>>> used > > >>>>>> in > > >>>>>>>>>>>>> situations > > >>>>>>>>>>>>>>> like > > >>>>>>>>>>>>>>>>>> this. > > >>>>>>>>>>>>>>>>>>> Can we get something like that in place > > >>> first? > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> David > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> On Fri, Jun 3, 2022 at 4:52 PM Jing Ge < > > >>>>>>>>> j...@ververica.com > > >>>>>>>>>>> > > >>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> Hi, > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> Thanks Alex for driving this! > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> +1 To give the Flink developers, > > >> especially > > >>>>>>> Connector > > >>>>>>>>>>>>> developers > > >>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>> clear > > >>>>>>>>>>>>>>>>>>>> signal that the new Source API is > > >>> recommended > > >>>>>>>> according > > >>>>>>>>> to > > >>>>>>>>>>>>>>> FLIP-27, > > >>>>>>>>>>>>>>>> we > > >>>>>>>>>>>>>>>>>>>> should mark them as deprecated. > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> There are some open questions to discuss: > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> 1. Do we need to mark all > > >>>>> subinterfaces/subclasses > > >>>>>>> as > > >>>>>>>>>>>>>> deprecated? > > >>>>>>>>>>>>>>>> e.g. > > >>>>>>>>>>>>>>>>>>>> FromElementsFunction, etc. there are many. > > >>>> What > > >>>>>> are > > >>>>>>>> the > > >>>>>>>>>>>>>>>> replacements? > > >>>>>>>>>>>>>>>>>>>> 2. Do we need to mark all subclasses that > > >>> have > > >>>>>>>>> replacement > > >>>>>>>>>>>>> as > > >>>>>>>>>>>>>>>>>> deprecated? > > >>>>>>>>>>>>>>>>>>>> e.g. ExternallyInducedSource whose > > >>> replacement > > >>>>>>> class, > > >>>>>>>>> if I > > >>>>>>>>>>>>> am > > >>>>>>>>>>>>>> not > > >>>>>>>>>>>>>>>>>> mistaken, > > >>>>>>>>>>>>>>>>>>>> ExternallyInducedSourceReader is > > >>> @Experimental > > >>>>>>>>>>>>>>>>>>>> 3. Do we need to mark all related test > > >>> utility > > >>>>>>> classes > > >>>>>>>>> as > > >>>>>>>>>>>>>>>> deprecated? > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> I think it might make sense to create an > > >>>>> umbrella > > >>>>>>>> ticket > > >>>>>>>>>> to > > >>>>>>>>>>>>>> cover > > >>>>>>>>>>>>>>>> all > > >>>>>>>>>>>>>>>>> of > > >>>>>>>>>>>>>>>>>>>> these with the following process: > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> 1. Mark SourceFunction as deprecated asap. > > >>>>>>>>>>>>>>>>>>>> 2. Mark subinterfaces and subclasses as > > >>>>>> deprecated, > > >>>>>>> if > > >>>>>>>>>>>>> there are > > >>>>>>>>>>>>>>>>>> graduated > > >>>>>>>>>>>>>>>>>>>> replacements. Good example is that > > >>> KafkaSource > > >>>>>>>> replaced > > >>>>>>>>>>>>>>>> KafkaConsumer > > >>>>>>>>>>>>>>>>>> which > > >>>>>>>>>>>>>>>>>>>> has been marked as deprecated. > > >>>>>>>>>>>>>>>>>>>> 3. Do not mark subinterfaces and > > >> subclasses > > >>> as > > >>>>>>>>> deprecated, > > >>>>>>>>>>>>> if > > >>>>>>>>>>>>>>>>>> replacement > > >>>>>>>>>>>>>>>>>>>> classes are still experimental, check if > > >> it > > >>> is > > >>>>>> time > > >>>>>>> to > > >>>>>>>>>>>>> graduate > > >>>>>>>>>>>>>>>> them. > > >>>>>>>>>>>>>>>>>> After > > >>>>>>>>>>>>>>>>>>>> graduation, go to step 2. It might take a > > >>>> while > > >>>>>> for > > >>>>>>>>>>>>> graduation. > > >>>>>>>>>>>>>>>>>>>> 4. Do not mark subinterfaces and > > >> subclasses > > >>> as > > >>>>>>>>> deprecated, > > >>>>>>>>>>>>> if > > >>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>>>> replacement classes are experimental and > > >> are > > >>>> too > > >>>>>>> young > > >>>>>>>>> to > > >>>>>>>>>>>>>>> graduate. > > >>>>>>>>>>>>>>>> We > > >>>>>>>>>>>>>>>>>> have > > >>>>>>>>>>>>>>>>>>>> to wait. But in this case we could create > > >>> new > > >>>>>>> tickets > > >>>>>>>>>> under > > >>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>> umbrella > > >>>>>>>>>>>>>>>>>>>> ticket. > > >>>>>>>>>>>>>>>>>>>> 5. Do not mark subinterfaces and > > >> subclasses > > >>> as > > >>>>>>>>> deprecated, > > >>>>>>>>>>>>> if > > >>>>>>>>>>>>>>> there > > >>>>>>>>>>>>>>>> is > > >>>>>>>>>>>>>>>>>> no > > >>>>>>>>>>>>>>>>>>>> replacement at all. We have to create new > > >>>>> tickets > > >>>>>>> and > > >>>>>>>>> wait > > >>>>>>>>>>>>> until > > >>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>> new > > >>>>>>>>>>>>>>>>>>>> implementation has been done and > > >> graduated. > > >>> It > > >>>>>> will > > >>>>>>>>> take a > > >>>>>>>>>>>>>> longer > > >>>>>>>>>>>>>>>>> time, > > >>>>>>>>>>>>>>>>>>>> roughly 1,5 years. > > >>>>>>>>>>>>>>>>>>>> 6. For test classes, we could follow the > > >>> same > > >>>>>> rule. > > >>>>>>>> But > > >>>>>>>>> I > > >>>>>>>>>>>>> think > > >>>>>>>>>>>>>>> for > > >>>>>>>>>>>>>>>>> some > > >>>>>>>>>>>>>>>>>>>> cases, we could consider doing the > > >>> replacement > > >>>>>>>> directly > > >>>>>>>>>>>>> without > > >>>>>>>>>>>>>>>> going > > >>>>>>>>>>>>>>>>>>>> through the deprecation phase. > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> When we look back on all of these, we can > > >>>>> realize > > >>>>>> it > > >>>>>>>> is > > >>>>>>>>> a > > >>>>>>>>>>>>> big > > >>>>>>>>>>>>>> epic > > >>>>>>>>>>>>>>>>> (even > > >>>>>>>>>>>>>>>>>>>> bigger than an epic). It needs someone to > > >>>> drive > > >>>>> it > > >>>>>>> and > > >>>>>>>>>> keep > > >>>>>>>>>>>>>> focus > > >>>>>>>>>>>>>>> on > > >>>>>>>>>>>>>>>>> it > > >>>>>>>>>>>>>>>>>>>> continuously with support from the > > >> community > > >>>> and > > >>>>>>> push > > >>>>>>>>> the > > >>>>>>>>>>>>>>>> development > > >>>>>>>>>>>>>>>>>>>> towards the new Source API of FLIP-27. > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> If we could have consensus for this, Alex > > >>>> and I > > >>>>>>> could > > >>>>>>>>>>>>> create > > >>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>> umbrella > > >>>>>>>>>>>>>>>>>>>> ticket to kick it off. > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> Best regards, > > >>>>>>>>>>>>>>>>>>>> Jing > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> On Fri, Jun 3, 2022 at 3:54 PM Alexander > > >>>>> Fedulov < > > >>>>>>>>>>>>>>>>>> alexan...@ververica.com> > > >>>>>>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> Hi everyone, > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> I would like to start the discussion > > >> about > > >>>>>> marking > > >>>>>>>>>>>>>>>>> SourceFunction-based > > >>>>>>>>>>>>>>>>>>>>> interfaces as deprecated. With the > > >> FLIP-27 > > >>>> APIs > > >>>>>>>>> becoming > > >>>>>>>>>>>>> the > > >>>>>>>>>>>>>> new > > >>>>>>>>>>>>>>>>>>>> standard, > > >>>>>>>>>>>>>>>>>>>>> the old ones have to be eventually phased > > >>>> out. > > >>>>>>>> Although > > >>>>>>>>>>>>> this > > >>>>>>>>>>>>>>> state > > >>>>>>>>>>>>>>>> is > > >>>>>>>>>>>>>>>>>>>> well > > >>>>>>>>>>>>>>>>>>>>> known within the community and no new > > >>>>> connectors > > >>>>>>>> based > > >>>>>>>>> on > > >>>>>>>>>>>>> the > > >>>>>>>>>>>>>> old > > >>>>>>>>>>>>>>>>>>>>> interfaces can be accepted into the > > >>> project, > > >>>>> the > > >>>>>>>>>> footprint > > >>>>>>>>>>>>> of > > >>>>>>>>>>>>>>>>>>>>> SourceFunction in the user code still > > >> keeps > > >>>>>> growing > > >>>>>>>>>>>>> (primarily > > >>>>>>>>>>>>>>> for > > >>>>>>>>>>>>>>>>> data > > >>>>>>>>>>>>>>>>>>>>> generators and test utilities). I believe > > >>> it > > >>>> is > > >>>>>>> best > > >>>>>>>> to > > >>>>>>>>>>>>> mark > > >>>>>>>>>>>>>>>>>>>> SourceFunction > > >>>>>>>>>>>>>>>>>>>>> as deprecated as soon as possible. What > > >> do > > >>>> you > > >>>>>>> think? > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>>> Best, > > >>>>>>>>>>>>>>>>>>>>> Alexander Fedulov > > >>>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> -- > > >>>>>>>>>>>>>> https://twitter.com/snntrable > > >>>>>>>>>>>>>> https://github.com/knaufk > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > >> > > > > > > -- https://twitter.com/snntrable https://github.com/knaufk
