Thanks all for the feedback on the FLIP. If there are no more discussions, I will start voting on this tomorrow.
On Wed, May 21, 2025 at 6:00 AM Mayank Juneja <mayankjunej...@gmail.com> wrote: > Hi Ryan, > > Thanks for the feedback. > > Just a quick clarification—DESCRIBE TABLE in Flink does *not* show table > properties; I believe you might be thinking of SHOW CREATE TABLE, which > does include properties as part of the DDL output. The goal of SHOW > CREATE TABLE is to provide a fully reproducible statement, so if secrets > were hidden or omitted, that reproducibility would be lost. By referencing > a named CONNECTION, we avoid this issue—users can reconstruct the table > DDL without exposing secrets directly, since the sensitive information > lives in the reusable connection object. > > Regarding the naming: CONNECTION is not part of the SQL standard, but > neither are alternatives like SECRET or INTEGRATION. We opted for > CONNECTION because it conveys intent clearly and aligns with common > terminology in data platforms, even if it's not a standard keyword. > > That said, we're definitely open to refining the naming based on broader > community input. Appreciate you raising these points and helping move the > conversation forward. > > Best, > Mayank > > On Fri, May 16, 2025 at 7:15 PM Ryan van Huuksloot > <ryan.vanhuuksl...@shopify.com.invalid> wrote: > >> Hi Mayank, >> >> Overall I think the idea of having shared properties (secrets or >> otherwise) >> at the database layer is really interesting/useful. Even outside of the ML >> scope. >> >> However, I have a concern: >> When I describe a table, I would not like it if all properties did not >> appear in the returned results of the describe. It adds a hidden layer of >> properties. >> CONNECTION is not a standard SQL syntax (at least that I am aware of, >> please correct me if I am wrong) and it adds a FlinkSQL specific layer of >> complexity. >> >> I like the idea of Grants to hide secrets rather than nesting them in a >> Connection object. The Connection object could still be described. >> Something akin to Snowflake's Secrets ( >> https://docs.snowflake.com/en/sql-reference/sql/show-secrets) that can be >> referenced within table properties and could be obfuscated during a >> describe. >> >> Then if we hide secrets, we can figure out a different name than >> Connection >> for common properties and they can be shown as part of the table spec. >> >> Thoughts? >> >> Ryan van Huuksloot >> Sr. Production Engineer | Streaming Platform >> [image: Shopify] >> <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> >> >> >> On Fri, May 16, 2025 at 1:35 PM Yash Anand <yan...@confluent.io.invalid> >> wrote: >> >> > Hi Mayank, >> > >> > Thanks for the initiative! I think this functionality would be a really >> > great addition to Flink. >> > >> > +1 for the proposal. >> > >> > Thanks, >> > Yash >> > >> > On Tue, May 6, 2025 at 9:36 AM Mayank Juneja <mayankjunej...@gmail.com> >> > wrote: >> > >> > > Hi Ferenc, >> > > >> > > Thanks for your question! We agree that a CONNECTION could logically >> be a >> > > catalog-level resource, especially since it’s intended to be reused >> > across >> > > multiple tables. However, we think there’s value in defining it at the >> > > database level to introduce a layer of isolation and scoping. >> > > >> > > This is particularly useful in shared or multi-tenant environments >> where >> > > different databases within the same catalog might need to connect to >> > > different external systems or use distinct credentials. >> > > >> > > That said, we’re open to revisiting this if there's strong consensus >> > around >> > > catalog-level scoping. Appreciate the feedback! >> > > >> > > Regards, >> > > Mayank >> > > >> > > On Fri, May 2, 2025 at 12:12 AM Ferenc Csaky >> <ferenc.cs...@pm.me.invalid >> > > >> > > wrote: >> > > >> > > > Hi Mayank, >> > > > >> > > > Thank you for starting the discussion! In general, I think such >> > > > functionality >> > > > would be a really great addition to Flink. >> > > > >> > > > Could you pls. elaborate a bit more one what is the reason of >> defining >> > a >> > > > `connection` resource on the database level instead of the catalog >> > level? >> > > > If I think about `JdbcCatalog`, or `HiveCatalog`, the catalog is in >> > > 1-to-1 >> > > > mapping with an RDBMS, or a HiveMetastore, so my initial thinking is >> > > that a >> > > > `connection` seems more like a catalog level resource. >> > > > >> > > > WDYT? >> > > > >> > > > Thanks, >> > > > Ferenc >> > > > >> > > > >> > > > >> > > > On Tuesday, April 29th, 2025 at 17:08, Mayank Juneja < >> > > > mayankjunej...@gmail.com> wrote: >> > > > >> > > > > >> > > > > >> > > > > Hi all, >> > > > > >> > > > > I would like to open up for discussion a new FLIP-529 [1]. >> > > > > >> > > > > Motivation: >> > > > > Currently, Flink SQL handles external connectivity by defining >> > > endpoints >> > > > > and credentials in table configuration. This approach prevents >> > > > reusability >> > > > > of these connections and makes table definition less secure by >> > exposing >> > > > > sensitive information. >> > > > > We propose the introduction of a new "connection" resource in >> Flink. >> > > This >> > > > > will be a pluggable resource configured with a remote endpoint and >> > > > > associated access key. Once defined, connections can be reused >> across >> > > > table >> > > > > definitions, and eventually for model definition (as discussed in >> > > > FLIP-437) >> > > > > for inference, enabling seamless and secure integration with >> external >> > > > > systems. >> > > > > The connection resource will provide a new, optional way to manage >> > > > external >> > > > > connectivity in Flink. Existing methods for table definitions will >> > > remain >> > > > > unchanged. >> > > > > >> > > > > [1] https://cwiki.apache.org/confluence/x/cYroF >> > > > > >> > > > > Best Regards, >> > > > > Mayank Juneja >> > > > >> > > >> > > >> > > -- >> > > *Mayank Juneja* >> > > Product Manager | Data Streaming and AI >> > > >> > >> > > > -- > *Mayank Juneja* > Product Manager | Data Streaming and AI > -- *Mayank Juneja* Product Manager | Data Streaming and AI
