[
https://issues.apache.org/jira/browse/FLINK-41599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-41599:
-----------------------------------
Labels: pull-request-available (was: )
> CatalogPropertiesUtil drops table options whose key starts with "schema"
> (e.g. 'schema-name') when deserializing a catalog table
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-41599
> URL: https://issues.apache.org/jira/browse/FLINK-41599
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
> Affects Versions: 2.2.0, 2.3.0, 2.2.1
> Reporter: Purushottam Sinha
> Priority: Major
> Labels: pull-request-available
>
> h3. Problem
> Since Flink 2.2.0, a catalog table that is serialized with
> {{ResolvedCatalogTable#toProperties}} and read back with
> {{CatalogTable.fromProperties}} loses every connector option whose key starts
> with {{schema}}. Only Flink's own schema entries ({{schema.<...>}}) should be
> excluded.
> Catalogs that persist tables this way therefore lose such options on reload.
> For example, the Flink CDC {{postgres-cdc}}, {{oracle-cdc}} and {{db2-cdc}}
> sources require {{'schema-name'}}. After a reload the table can no longer be
> used:
> {noformat}
> Unable to create a source for reading table '...'.
> ...
> Missing required options are:
> schema-name
> {noformat}
> {{CREATE TABLE}} succeeds; the first query on the table fails. Catalogs that
> keep table objects in memory ({{GenericInMemoryCatalog}}) are not affected.
> h3. Cause
> FLINK-38311 (commit 2e7cac1906, apache/flink#26990) merged the two
> {{deserializeOptions}} overloads and dropped the separator for the schema
> prefix:
> {code:java}
> // up to 2.1
> key.startsWith(schemaKey + SEPARATOR) // "schema."
> // 2.2, 2.3, master
> !key.startsWith(DISTRIBUTION + SEPARATOR) // "distribution."
> && !key.startsWith(PARTITION_KEYS + SEPARATOR) // "partition.keys."
> && !key.startsWith(SCHEMA) // "schema" -- separator
> missing
> {code}
> Every schema property Flink writes is {{schema.<...>}} ({{compoundKey(SCHEMA,
> ...)}}), so the missing separator only removes user options.
> h3. Reproduction
> {code:java}
> CatalogTable table = CatalogTable.newBuilder()
> .schema(Schema.newBuilder().column("id", DataTypes.INT()).build())
> .options(Map.of("connector", "postgres-cdc", "schema-name", "public",
> "table-name", "orders"))
> .build();
> ResolvedCatalogTable resolved = new ResolvedCatalogTable(
> table, ResolvedSchema.of(Column.physical("id", DataTypes.INT())));
> CatalogTable.fromProperties(resolved.toProperties(DefaultSqlFactory.INSTANCE)).getOptions();
> {code}
> Options read back, with {{flink-table-api-java}} from Maven Central:
> * 1.19.3 / 2.0.2 / 2.1.3: {{connector, schema-name, table-name}}
> * 2.2.1 / 2.3.0: {{connector, table-name}} ({{schema-name}} is lost)
> h3. Proposed fix
> Use {{!key.startsWith(SCHEMA + SEPARATOR)}} in
> {{CatalogPropertiesUtil#deserializeOptions}}. Add a case to
> {{CatalogPropertiesUtilTest}} with an option such as {{schema-name}} that
> must survive the round trip. Backport to release-2.2 and release-2.3.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)