Purushottam Sinha created FLINK-41599:
-----------------------------------------

             Summary: CatalogPropertiesUtil drops table options whose key 
starts with "schema" (e.g. 'schema-name') when deserializing a catalog table
                 Key: FLINK-41599
                 URL: https://issues.apache.org/jira/browse/FLINK-41599
             Project: Flink
          Issue Type: Bug
          Components: Table SQL / API
    Affects Versions: 2.2.1, 2.2.0, 2.3.0
            Reporter: Purushottam Sinha


h3. Problem
Since Flink 2.2.0, a catalog table that is serialized with 
{{ResolvedCatalogTable#toProperties}} and read back with 
{{CatalogTable.fromProperties}} loses every connector option whose key starts 
with {{schema}}. Only Flink's own schema entries ({{schema.<...>}}) should be 
excluded.

Catalogs that persist tables this way therefore lose such options on reload. 
For example, the Flink CDC {{postgres-cdc}}, {{oracle-cdc}} and {{db2-cdc}} 
sources require {{'schema-name'}}. After a reload the table can no longer be 
used:
{noformat}
Unable to create a source for reading table '...'.
...
Missing required options are:
schema-name
{noformat}
{{CREATE TABLE}} succeeds; the first query on the table fails. Catalogs that 
keep table objects in memory ({{GenericInMemoryCatalog}}) are not affected.

h3. Cause
FLINK-38311 (commit 2e7cac1906, apache/flink#26990) merged the two 
{{deserializeOptions}} overloads and dropped the separator for the schema 
prefix:
{code:java}
// up to 2.1
key.startsWith(schemaKey + SEPARATOR)          // "schema."

// 2.2, 2.3, master
!key.startsWith(DISTRIBUTION + SEPARATOR)      // "distribution."
&& !key.startsWith(PARTITION_KEYS + SEPARATOR) // "partition.keys."
&& !key.startsWith(SCHEMA)                     // "schema" -- separator missing
{code}
Every schema property Flink writes is {{schema.<...>}} ({{compoundKey(SCHEMA, 
...)}}), so the missing separator only removes user options.

h3. Reproduction
{code:java}
CatalogTable table = CatalogTable.newBuilder()
    .schema(Schema.newBuilder().column("id", DataTypes.INT()).build())
    .options(Map.of("connector", "postgres-cdc", "schema-name", "public", 
"table-name", "orders"))
    .build();
ResolvedCatalogTable resolved = new ResolvedCatalogTable(
    table, ResolvedSchema.of(Column.physical("id", DataTypes.INT())));

CatalogTable.fromProperties(resolved.toProperties(DefaultSqlFactory.INSTANCE)).getOptions();
{code}
Options read back, with {{flink-table-api-java}} from Maven Central:
* 1.19.3 / 2.0.2 / 2.1.3: {{connector, schema-name, table-name}}
* 2.2.1 / 2.3.0: {{connector, table-name}} ({{schema-name}} is lost)

h3. Proposed fix
Use {{!key.startsWith(SCHEMA + SEPARATOR)}} in 
{{CatalogPropertiesUtil#deserializeOptions}}. Add a case to 
{{CatalogPropertiesUtilTest}} with an option such as {{schema-name}} that must 
survive the round trip. Backport to release-2.2 and release-2.3.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to