Hi, There are some typos in my last email, although hopefully I made myself clear with the context, sorry about that:
FLIP-179 -> FLIP-197 FlinkKafkaConsumer -> FlinkKafkaProducer Best regards, Jing On Thu, Jan 19, 2023 at 5:30 AM Jing Ge <j...@ververica.com> wrote: > Hi, > > Thanks Martijn for bringing this to our attention. This > discussion deserves a much wider range of attention. because graduation of > the Sink API will have an impact on all connectors. > > For me, briefly speaking, like I already pointed out in another thread > [1], it is too hasty and risky to do it in 1.17 and even in 1.18. Here are > some reasons I knew and there might be more: > > 1. commonly, it is fine to follow FLIP-179 [2], since most API has only > one implementation. Two release cycles for @PublicEvolving should be good > enough to test the API design and implementation and get feedback and make > the API stable. But, Connector API (including the Sink API) is a special > case. There will be more than 10 different implementations, Some of them > are very important for Flink users' daily work. Rules defined in FLIP-179 > should be adapted in this case. > > 2. Currently, afaik, there are only two Sink API implementations and none > of them is graduated. KafkaSink is @PublicEvloving and FileSink > is @Experimental. It is very risky to graduate the fundamental Sink API > before none of its implementation has been graduated. > > 3. We have an awkward situation with one of the most important connectors, > the Kafka connector. The KafakSink is @PublicEvolving but the > FlinkKafkaConsumer is already deprecated. Users are confused and some of > them are still sticking to FlinkKafkaConsumer, since it is normally not > recommended to use @PublicEvolving in production. I tried to graduate the > KafkaSink and remove the FlinkKafkaProducer to push users using KafkaSink > in this thread [3]. But there are some tasks to finish as a prerequisite. > Things got more complicated as expected and some new rules > need to be defined and discussed. I will write a new FLIP to address this > issue later. > > 4. Even if we could graduate the Kafka connector, it is still risky to > graduate the SinkV2 API, which will make the Kafka connector special > comparing to others, i.e. kafka connector as the first class vs. others as > the second class. This is not what we want to see. > > 5. Afaik, there are still some fresh construction wrt the SinkV2 @Yun Gao > <yungao...@aliyun.com> > > 6. Flink wants to be the unified batch and stream processing framework. It > might be rational, if a more batch oriented Sink implementation could be > done and graduated before we graduate the Sink API. The JDBC connector > might be a feasible choice. > > Therefore, a solid solution should be: > > 1. Develop three SinkV2 implementations. Considering batch, stream, > popularity, etc. I personally would choose Kafka, File, and JDBC. But the > list deserves further discussion. > 2. graduate all three SinkV2 implementations and remove old > implementations. If there are any rejections in the community for removing > the old implementation. It means the related SinkV2 implementation is not > ready to graduate. We will need more time. > 3. 1 - 2 release cycles to get feedback from users wrt those SinkV2 > implementations. > 3. graduate the SinkV2 API. > > Now we could see, if we could find developers focusing on them and deliver > them on time, we would still need at least 4 release cycles before we > could graduate the Sink API. > > If we want to pace up and could live with some risk, we could go with the > following process: > > 1. Remove FlinkKafkaProucer and graduate KafkaSink in Flink 1.18 in the > ideal case. > 2. Get feedback and stabilize the API while we continue working on Flink > and release 1.19 > 3. granduate the Sink API in Flink 1.20 > 4. be aware that we might have to develop SinkV3 if any unknown > requirements become known after new Sink implementation is developed. > > Please pay attention, this process is still hasty and risky. Because it > only satisfied Kafka and might have a big impact on other connectors, i.e. > we are treating the Kafka connector as the king. > > Please don't get me wrong. I am not trying to block this change. On the > contrary, I am keen to graduate SinkV2 API and remove old API and > implementations. But I think it is important to be aware of the risk and > make sure everyone is on the same page. There might be some other > information I don't know and I am happy to hear any opposite opinions that > can push the SinkV2 API to be graduated asap. > > Last but not least, this discussion starts during the holiday season in > China. @Qingsheng Ren <renqs...@gmail.com> @Jark Wu <imj...@gmail.com> I > would like to have your attention please. > > Best regards, > Jing > > > [1] https://lists.apache.org/thread/l05m6cf8fwkkbpnjtzbg9l2lo40oxzd1 > [2] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process > [3] https://lists.apache.org/thread/m3o48c2d8j9g5t9s89hqs6qvr924s71o > > On Thu, Jan 19, 2023 at 3:26 AM Lijie Wang <wangdachui9...@gmail.com> > wrote: > >> Hi Martijn, >> >> Thanks for driving this. I have a only concern about the Sink.InitContext. >> >> Does the Sink.InitContext will also be changed to @Public ? As described >> in >> FLIP-287, currently the Sink.InitContext still lacks some necessary >> information to migrate existing connectors to new sinks. If it is marked >> as >> public/stable, we can no longer modify it in the future(since most >> connectors are not migrated to SinkV2 currently, we may find we need more >> information via InitContext in the future migrations). >> >> Best, >> Lijie >> >> Yun Tang <myas...@live.com> 于2023年1月18日周三 21:13写道： >> >> > SinkV2 was introduced in Flink-1.15 and annotated as @PublicEvolving >> from >> > the 1st day [1]. From FLIP-197, we can promote it to @Public since it >> > already existed with two releases. >> > And I didn't find a FLIP to discuss the process to deprecate APIs, >> > considering the SinkFunction has actually been stale for some time, I >> think >> > we can deprecate it with the @Public SinkV2. >> > >> > Thus, +1 (binding) for this proposal. >> > >> > [1] https://issues.apache.org/jira/browse/FLINK-25555 >> > >> > Best >> > Yun Tang >> > >> > ________________________________ >> > From: Martijn Visser <martijnvis...@apache.org> >> > Sent: Wednesday, January 18, 2023 18:50 >> > To: dev <dev@flink.apache.org>; Jing Ge <j...@ververica.com>; Yun Tang >> < >> > myas...@live.com> >> > Subject: [DISCUSS] Promote SinkV2 to @Public and deprecate SinkFunction >> > >> > Hi all, >> > >> > While discussing FLIP-281 [1] the discussion also turned to the >> > SinkFunction and the SinkV2 API. For a broader discussion I'm opening >> up a >> > separate discussion thread. >> > >> > As Yun Tang has mentioned in that discussion thread, it would be a good >> > time to deprecate the SinkFunction to avoid the need to introduce new >> > functions towards (to be) deprecated APIs. Jing rightfully mentioned >> that >> > it would be confusing to deprecate the SinkFunction if its successor is >> not >> > yet marked as @Public (it's currently @PublicEvolving). >> > >> > My proposal would be to promote the SinkV2 API to @public in Flink 1.17 >> > and mark the SinkFunction as @deprecated in Flink 1.17 >> > >> > The original Sink interface was introduced in Flink 1.12 with FLIP-143 >> [2] >> > and extended with FLIP-177 in Flink 1.14 [3] and has been improved on >> > further as Sink V2 via FLIP-191 in Flink 1.15 [4]. >> > >> > Looking at the API stability graduation process [5], the fact that Sink >> V2 >> > was introduced in Flink 1.15 would mean that we could warrant a >> promotion >> > to @public already (given that there have been two releases with 1.15 >> and >> > 1.16 where it was introduced). Combined with the fact that SinkV2 has >> been >> > the result of iteration over the introduction of the original Sink API >> > since Flink 1.12, I would argue that the promotion is overdue. >> > >> > If we promote the Sink API to @public, I think we should also >> immediately >> > mark the SinkFunction as @deprecated. >> > >> > Looking forward to your thoughts. >> > >> > Best regards, >> > >> > Martijn >> > >> > >> > [1] https://lists.apache.org/thread/l05m6cf8fwkkbpnjtzbg9l2lo40oxzd1 >> > [2] >> > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API >> > [3] >> > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-177%3A+Extend+Sink+API >> > [4] >> > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction >> > [5] >> > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process >> > >> > >> >
