I see. Thanks for the explanation. I may have not looked into this deeply enough, and would trust the decision from you and the community members who participated in the discussion & vote.
Best, Xintong On Thu, Jun 29, 2023 at 10:28 PM Alexander Fedulov < alexander.fedu...@gmail.com> wrote: > > However, I'm not sure about 2. > > I am not aware of a bylaw that states the specific requirements in order to > mark something as @Deprecated. My understanding from the discussion and the > vote was that the community recognizes the necessity to make it explicit > that > the usage of the SourceFunction API is discouraged. This can actually > stimulate > authors of connectors that rely on this very specific and non-baseline > functionality to contribute extensions to the new Source API themselves in > order to > close the gap. ExternallyInducedSource, for instance, was driven by Pravega > to > begin with, since it was only needed for their purposes [1]. We are not > removing > anything - until 2.0 everything will continue to work and we can work on > resolving the limitations until then, I personally don't see a big issue > here. > > >Do you think it is feasible to resolve them by the feature freeze date of > 1.18? > No, these are rather complex additions that would probably require FLIP(s). > > [1] > > https://flink.apache.org/2022/01/20/pravega-flink-connector-101/#checkpoint-integration > > On Thu, 29 Jun 2023 at 14:25, Xintong Song <tonysong...@gmail.com> wrote: > > > Thanks for the explanation, Alex. > > > > Not blocking the deprecation on 1 & 3 makes sense to me. However, I'm not > > sure about 2. > > > > It sounds to me that, without FLINK-28051 & FLINK-28054, some of the > > connectors cannot migrate to the new Source API, or at least further > > investigation is needed to understand the situation. If this is the case, > > we probably should not deprecate the API until these issues are resolved. > > Do you think it is feasible to resolve them by the feature freeze date of > > 1.18? > > > > Best, > > > > Xintong > > > > > > > > On Thu, Jun 29, 2023 at 8:02 PM Alexander Fedulov < > > alexander.fedu...@gmail.com> wrote: > > > > > @Xintong > > > The original discussion [1] and vote [2] converged on the idea that it > is > > > better > > > to make it clear to the users that they should stop using > SourceFunction > > > since it > > > is going away. The longer we do not have this indication, the more user > > > implementations will be based on it and the more pain will be induced > > when > > > we > > > finally drop it. Users now have an alternative API that they should use > > and > > > which > > > is fully functional, from that perspective nothing blocks marking it > > > @Deprecated. > > > As for the remaining work items - there are primarily three kinds: > > > > > > 1. Where Flink internally uses SourceFunction, without exposing this > fact > > > to the > > > outside world: > > > - FLINK-28050 [3] > > > - FLINK-28229 [4] > > > - FLINK-28048 [5] > > > > > > 2. Very specific edge cases that might not be covered by the Source API > > as > > > is: > > > - FLINK-28054 [6] > > > - FLINK-28051 [7] > > > > > > 3. Usability improvements - something that was easily doable with > > > SourceFunction, > > > but requires deep knowledge of the new, significantly more complex, > > > Source API > > > to achieve: > > > - FLINK-28056 [8] > > > > > > In my mind, none of those are blockers for proceeding with adding the > > > @Deprecated > > > annotation: > > > (1) is a simple case of encapsulation, internals should not concern the > > API > > > users > > > (2) is really only relevant for "exotic" use cases. Does not mean we > > should > > > not > > > consider those, but since it is irrelevant for 99.9% of the users, I do > > not > > > think > > > we should get stuck here. > > > (3) is purely a nice to have. Formally speaking, all of the tools are > > > there, it is > > > just that due to the complexity of the new Source API some "simple" > > things > > > become > > > non-trivial and ideally we want to do better here. > > > > > > [1] https://lists.apache.org/thread/d6cwqw9b3105wcpdkwq7rr4s7x4ywqr9 > > > [2] https://lists.apache.org/thread/kv9rj3w2rmkb8jtss5bqffhw57or7v8v > > > [3] https://issues.apache.org/jira/browse/FLINK-28050 > > > [4] https://issues.apache.org/jira/browse/FLINK-28229 > > > [5] https://issues.apache.org/jira/browse/FLINK-28048 > > > [6] https://issues.apache.org/jira/browse/FLINK-28054 > > > [7] https://issues.apache.org/jira/browse/FLINK-28051 > > > [8] https://issues.apache.org/jira/browse/FLINK-28056 > > > > > > On Thu, 29 Jun 2023 at 13:13, Xintong Song <tonysong...@gmail.com> > > wrote: > > > > > > > Thanks for the inputs, Martijn, Jing and Alex. > > > > > > > > @Martijn, > > > > Regarding the Scala supports, I personally don't think "a fully > striked > > > > through experience in the IDE" is something we want to avoid, > > especially > > > > given that we are planning to remove the deprecated APIs soon, unlike > > > when > > > > FLINK-29740 was resolved we didn't really know when they would be > > > removed. > > > > Moreover, the even entry point for DataStream Scala > > > > (`StreamExecutionEnvironment`) is not annotated. > > > > > > > > > > > > @Jing and @Alex, > > > > > > > > IIUC, you mean SourceFunction can be annotated as `@Deprecated` in > > 1.18, > > > > and there's already a PR doing so. However, after the deprecation, > > there > > > > are still issues that need to be addressed before removing it in 2.0? > > > This > > > > sounds a bit weird. If the API cannot be dropped, which means without > > > this > > > > API some of functions cannot be supported, then how could it be > > > deprecated? > > > > How would we expect users to migrate away from it? > > > > > > > > > > > > @Jing, > > > > > > > > Sounds like it's impractical to deprecate SinkFunction in 1.18. Any > > > > expectation / plan on when / how it can be deprecated / removed? > > > > > > > > > > > > Best, > > > > > > > > Xintong > > > > > > > > > > > > > > > > On Thu, Jun 29, 2023 at 6:12 PM Alexander Fedulov < > > > > alexander.fedu...@gmail.com> wrote: > > > > > > > > > Hi Xintong, > > > > > > > > > > Thanks for bringing up this topic. I can provide some details > > regarding > > > > > the SourceFunction deprecation efforts. Marking SourceFunction as > > > > > deprecated was not possible until now since we have stringent > > compiler > > > > > checks in flink-examples against using any deprecated APIs. We > > actually > > > > > merged the migration of all examples to the new FLIP-27-based > > > > > DataGeneratorSource [1] just two days ago [2]. Now the PR marking > > > > > it @Deprecated is finally unblocked [3] (I would be grateful if you > > > could > > > > > merge it). > > > > > > > > > > With regards to the Flink 2.0 scope, I compiled a list of items > > > required > > > > to > > > > > be able to drop the SourceFunction API [4] a while ago and as you > can > > > > see, > > > > > there is still quite some work to be done. Some items [5] might > even > > > > > require additions to the new Source API. Overall, I am happy to > take > > > > > ownership of completing this work package. > > > > > > > > > > Best, > > > > > Alex > > > > > > > > > > > > > > > [1] https://cwiki.apache.org/confluence/x/9Av1D > > > > > [2] https://github.com/apache/flink/pull/21774 > > > > > [3] https://github.com/apache/flink/pull/20049 > > > > > [4] https://issues.apache.org/jira/browse/FLINK-28045 > > > > > [5] https://issues.apache.org/jira/browse/FLINK-28054 > > > > > > > > > > On Thu, 29 Jun 2023 at 10:45, Martijn Visser < > > martijnvis...@apache.org > > > > > > > > > wrote: > > > > > > > > > > > Hi Xintong, > > > > > > > > > > > > With regards to the deprecation of the Scala APIs, during the PR > > > > > > review it was requested to not mark all APIs as deprecated but > only > > > > > > the entry point [1], to avoid a fully striked through experience > in > > > > > > the IDE. I think the same idea was applicable on the DataSet > API. I > > > > > > think it depends on how formal we want to treat this: if really > > > > > > formal, then we should deprecate them in 1.18. I think in both > > cases, > > > > > > it's quite well known that they are deprecated. I'm +0 for either > > > way, > > > > > > as long as we're all agreeing that they can be removed in 2.0. > > > > > > > > > > > > With regards to Queryable State and Source/SinkFunction, +1 to > mark > > > > > > these as deprecated. > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Martijn > > > > > > > > > > > > [1] > > > > > > > > > > > > https://github.com/apache/flink/pull/21176#pullrequestreview-1159706808 > > > > > > > > > > > > On Thu, Jun 29, 2023 at 10:23 AM Xintong Song < > > tonysong...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > Hi devs, > > > > > > > > > > > > > > Looking at the release 2.0 proposals [1], I noticed that many > > APIs > > > > that > > > > > > are > > > > > > > proposed to be removed in 2.0 are not (fully) deprecated yet. > We > > > > might > > > > > > want > > > > > > > to properly mark them as `@Deprecated` in 1.18 if we agree they > > > > should > > > > > be > > > > > > > removed in 2.0. Moreover, according to FLIP-321 [2] (not voted > > yet > > > > but > > > > > > IMO > > > > > > > is close to consensus IMO), a migration period is required > after > > > APIs > > > > > are > > > > > > > deprecated and before they can be removed. > > > > > > > > > > > > > > I might not be familiar with the status of all the APIs below. > So > > > I'd > > > > > > like > > > > > > > to bring them up and see if there's any concern regarding > > > deprecating > > > > > > them > > > > > > > in 1.18. If there's concern for deprecating API, we can start a > > > > > separate > > > > > > > discussion thread for it. For those with no objections, I'd > > create > > > > JIRA > > > > > > > tickets and try to properly deprecate them in 1.18. > > > > > > > > > > > > > > 1. DataSet API > > > > > > > It's described as "legacy", "soft deprecated" in user > > documentation > > > > > [3]. > > > > > > > However, it's not annotated with `@Deprecated` in codes. > > According > > > to > > > > > > > FLIP-131 [4], DataSet API should be deprecated when DataStream > > API > > > > and > > > > > > > Table API / SQL meet certain requirements. AFAICS, all the > > > > requirements > > > > > > > mentioned in the FLIP are already fulfilled. We should annotate > > it > > > as > > > > > > > `@Deprecated` now. > > > > > > > > > > > > > > 2. SourceFunction / SinkFunction > > > > > > > They are described as deprecated in the roadmap[5], and I don't > > > find > > > > > > > anything regarding them in user documentation. But they are > also > > > not > > > > > > > annotated with `@Deprecated` in codes. TBH, I'm not aware of > any > > > > formal > > > > > > > decision to deprecate these. AFAICS, the replacement for > > > > SourceFunction > > > > > > > (Source) has already been promoted to `@Public`, while the > > > > replacement > > > > > > for > > > > > > > SinkFunction (SinkV2) is still `@PublicEvolving`. I found a > > > > > discussion[6] > > > > > > > regarding promoting SinkV2 to `@Public`, but it's unclear to me > > > what > > > > > the > > > > > > > conclusion is. > > > > > > > > > > > > > > 3. Queryable State > > > > > > > It's described as approaching end-of-life in the roadmap [5], > but > > > is > > > > > > > neither deprecated in codes nor in user documentation [7]. I > also > > > > > found a > > > > > > > discussion [8] about rescuing it from deprecation, and it seems > > to > > > me > > > > > > there > > > > > > > are more negative opinions than positive ones. > > > > > > > > > > > > > > 4. All Scala APIs > > > > > > > I think we agreed to drop Scala API support in FLIP-265 [9], > and > > > have > > > > > > tried > > > > > > > to deprecate them in FLINK-29740 [10]. Also, both user > > > documentation > > > > > and > > > > > > > roadmap[5] shows that scala API supports are deprecated. > However, > > > > > AFAICS, > > > > > > > none of the APIs in `flink-streaming-scala` are annotated with > > > > > > > `@Deprecated`, and only `ExecutionEnvironment` and `package` > are > > > > marked > > > > > > > `@Deprecated` in `flink-scala`. > > > > > > > > > > > > > > Looking forward to your feedback. > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > Xintong > > > > > > > > > > > > > > > > > > > > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/2.0+Release > > > > > > > > > > > > > > [2] > > > https://lists.apache.org/thread/vmhzv8fcw2b33pqxp43486owrxbkd5x9 > > > > > > > > > > > > > > [3] > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/dataset/overview/ > > > > > > > > > > > > > > [4] > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866741 > > > > > > > > > > > > > > [5] https://flink.apache.org/roadmap/ > > > > > > > > > > > > > > [6] > > > https://lists.apache.org/thread/q62nj89rrz0t5xtggy5n65on95f2rmmx > > > > > > > > > > > > > > [7] > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/queryable_state/ > > > > > > > > > > > > > > [8] > > > https://lists.apache.org/thread/9hmwcjb3q5c24pk3qshjvybfqk62v17m > > > > > > > > > > > > > > [9] > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support > > > > > > > > > > > > > > [10] https://issues.apache.org/jira/browse/FLINK-29740 > > > > > > > > > > > > > > > > > > > > >
