> Existing APIs in Flink which are marked with @Internal also don't
offer any compatibility guarantee. Yet we are OK to have classes in
Flink use them.
Well of course, it's the same codebase. If we change the API we can
change all usages as well. Hence, there's no problem using internal APIs.
> After all, externalized connectors are Flink project's code, not
Flink users' code, and thus can be considered internal code by default
That's not how API compatibility works. Being "part of the Public API"
means that we intend other software projects to make use of these APIs.
This also includes other Flink software projects, like statefun,
connectors, flink-ml and so on.
> By "fully independent", I assume you mean that we want to be able to
bump up the version of Flink in externalized connectors without having
to change those connectors.
What I mean is that a connector build for 1.x continues to work for 1.y.
> Strictly speaking, we can not be fully independent because even APIs
marked with @Public can still have backward incompatible changes across
the major release.
Of course, that's what makes a major release a major release. It's
expected that things might not work across 1.x and 2.y.
On 31/01/2023 01:06, Dong Lin wrote:
On Mon, Jan 30, 2023 at 8:51 PM Chesnay Schepler <ches...@apache.org> wrote:
On 30/01/2023 13:24, Dong Lin wrote:
it should be OK for
apache/flink-connector-aws to use non-public API, similar to how classes
inside Flink can use APIs marked with @Internal.
We can mark related classes with @Internal as appropriate, without
requiring any new FLIP.
It's not though, because these APIs don't offer any compatibility
guarantees.
Existing APIs in Flink which are marked with @Internal also don't offer any
compatibility guarantee. Yet we are OK to have classes in Flink use them.
This is because we (the Flink community developers) have full control of
those code in apache/flink and are free to update these code to work with
any backward incompatible changes made to those internal API. After all,
externalized connectors are Flink project's code, not Flink users' code,
and thus can be considered internal code by default
I think the same logic apply to those code in apache/flink-connector-aws.
Can you explain why it is not OK?
We want to arrive at a state where connectors are fully independent of
Flink releases. Using non-public APIs runs counter to that.
By "fully independent", I assume you mean that we want to be able to bump
up the version of Flink in externalized connectors without having to change
those connectors.
Strictly speaking, we can not be fully independent because even APIs marked
with @Public can still have backward incompatible changes across the major
release. Yes we can minimize the changes needed to externalized connectors
by using only public PAI. Though it is a nice property, I don't think this
is a strict requirement due to the reasons explained above.
Maybe we should discuss this in a separate thread to reach an agreement on
this topic?
