Hi Mayank,
Sorry for joining this FLIP discussion late. I can see that it significantly
reduces complexity and improves security when creating tables.
However, I have a question regarding its usage in the context of catalogs:
In the current FLIP, when creating a connection within a catalog, we use:
CREATE [TEMPORARY] [SYSTEM] CONNECTION [IF NOT EXISTS]
[catalog_name.][db_name.]connection_name
[COMMENT connection_comment]
WITH (
'type' = 'basic' | 'bearer' | 'oauth' | 'custom_type',
'url' = 'endpoint_url',
'username' = 'user_name', -- Will be extracted to SecretStore
'password' = 'user_password', -- Will be extracted to SecretStore
...
)
But we still need to define the catalog itself with some connection information
upfront. Does this mean that users are still required to provide initial
credentials (e.g., username and password) when creating the catalog?
If so, isn't this inconsistent with the goal of centralizing and securing
credentials via the SecretStore?
On the other hand, if a table is created using a catalog that already has an
associated connection, it seems no additional connection properties need to be
specified — which is great. But then, why do we need per-connection credentials
at all? Could the catalog-level connection be sufficient in most cases?
Best
Hongshun
> 2025年4月29日 23:07，Mayank Juneja <mayankjunej...@gmail.com> 写道：
>
> Hi all,
>
> I would like to open up for discussion a new FLIP-529 [1].
>
> Motivation:
> Currently, Flink SQL handles external connectivity by defining endpoints
> and credentials in table configuration. This approach prevents reusability
> of these connections and makes table definition less secure by exposing
> sensitive information.
> We propose the introduction of a new "connection" resource in Flink. This
> will be a pluggable resource configured with a remote endpoint and
> associated access key. Once defined, connections can be reused across table
> definitions, and eventually for model definition (as discussed in FLIP-437)
> for inference, enabling seamless and secure integration with external
> systems.
> The connection resource will provide a new, optional way to manage external
> connectivity in Flink. Existing methods for table definitions will remain
> unchanged.
>
> [1] https://cwiki.apache.org/confluence/x/cYroF
>
> Best Regards,
> Mayank Juneja
