[ 
https://issues.apache.org/jira/browse/FLINK-41599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purushottam Sinha updated FLINK-41599:
--------------------------------------
    Description: 
h3. Problem
Since Flink 2.2.0, a catalog table that is serialized with 
{{ResolvedCatalogTable#toProperties}} and read back with 
{{CatalogTable.fromProperties}} loses every connector option whose key starts 
with {{schema}}. Only Flink's own schema entries ({{schema.<...>}}) should be 
excluded.

Catalogs that persist tables this way therefore lose such options on reload. 
For example, the Flink CDC {{postgres-cdc}} and {{oracle-cdc}} sources require 
{{'schema-name'}}. After a reload the table can no longer be used:
{noformat}
Unable to create a source for reading table '...'.
...
Missing required options are:
schema-name
{noformat}
{{CREATE TABLE}} succeeds; the first query on the table fails. Catalogs that 
keep table objects in memory ({{GenericInMemoryCatalog}}) are not affected.

h3. Cause
FLINK-38311 (commit 2e7cac1906, apache/flink#26990) merged the two 
{{deserializeOptions}} overloads and dropped the separator for the schema 
prefix:
{code:java}
// up to 2.1
key.startsWith(schemaKey + SEPARATOR)          // "schema."

// 2.2, 2.3, master
!key.startsWith(DISTRIBUTION + SEPARATOR)      // "distribution."
&& !key.startsWith(PARTITION_KEYS + SEPARATOR) // "partition.keys."
&& !key.startsWith(SCHEMA)                     // "schema" -- separator missing
{code}
Every schema property Flink writes is {{schema.<...>}} ({{compoundKey(SCHEMA, 
...)}}), so the missing separator only removes user options.

h3. Reproduction
{code:java}
CatalogTable table = CatalogTable.newBuilder()
    .schema(Schema.newBuilder().column("id", DataTypes.INT()).build())
    .options(Map.of("connector", "postgres-cdc", "schema-name", "public", 
"table-name", "orders"))
    .build();
ResolvedCatalogTable resolved = new ResolvedCatalogTable(
    table, ResolvedSchema.of(Column.physical("id", DataTypes.INT())));

CatalogTable.fromProperties(resolved.toProperties(DefaultSqlFactory.INSTANCE)).getOptions();
{code}
Options read back, with {{flink-table-api-java}} from Maven Central:
* 1.19.3 / 2.0.2 / 2.1.3: {{connector, schema-name, table-name}}
* 2.2.1 / 2.3.0: {{connector, table-name}} ({{schema-name}} is lost)

h3. Proposed fix
Use {{!key.startsWith(SCHEMA + SEPARATOR)}} in 
{{CatalogPropertiesUtil#deserializeOptions}}. Add a case to 
{{CatalogPropertiesUtilTest}} with an option such as {{schema-name}} that must 
survive the round trip. Backport to release-2.2 and release-2.3.

  was:
h3. Problem
Since Flink 2.2.0, a catalog table that is serialized with 
{{ResolvedCatalogTable#toProperties}} and read back with 
{{CatalogTable.fromProperties}} loses every connector option whose key starts 
with {{schema}}. Only Flink's own schema entries ({{schema.<...>}}) should be 
excluded.

Catalogs that persist tables this way therefore lose such options on reload. 
For example, the Flink CDC {{postgres-cdc}}, {{oracle-cdc}} and {{db2-cdc}} 
sources require {{'schema-name'}}. After a reload the table can no longer be 
used:
{noformat}
Unable to create a source for reading table '...'.
...
Missing required options are:
schema-name
{noformat}
{{CREATE TABLE}} succeeds; the first query on the table fails. Catalogs that 
keep table objects in memory ({{GenericInMemoryCatalog}}) are not affected.

h3. Cause
FLINK-38311 (commit 2e7cac1906, apache/flink#26990) merged the two 
{{deserializeOptions}} overloads and dropped the separator for the schema 
prefix:
{code:java}
// up to 2.1
key.startsWith(schemaKey + SEPARATOR)          // "schema."

// 2.2, 2.3, master
!key.startsWith(DISTRIBUTION + SEPARATOR)      // "distribution."
&& !key.startsWith(PARTITION_KEYS + SEPARATOR) // "partition.keys."
&& !key.startsWith(SCHEMA)                     // "schema" -- separator missing
{code}
Every schema property Flink writes is {{schema.<...>}} ({{compoundKey(SCHEMA, 
...)}}), so the missing separator only removes user options.

h3. Reproduction
{code:java}
CatalogTable table = CatalogTable.newBuilder()
    .schema(Schema.newBuilder().column("id", DataTypes.INT()).build())
    .options(Map.of("connector", "postgres-cdc", "schema-name", "public", 
"table-name", "orders"))
    .build();
ResolvedCatalogTable resolved = new ResolvedCatalogTable(
    table, ResolvedSchema.of(Column.physical("id", DataTypes.INT())));

CatalogTable.fromProperties(resolved.toProperties(DefaultSqlFactory.INSTANCE)).getOptions();
{code}
Options read back, with {{flink-table-api-java}} from Maven Central:
* 1.19.3 / 2.0.2 / 2.1.3: {{connector, schema-name, table-name}}
* 2.2.1 / 2.3.0: {{connector, table-name}} ({{schema-name}} is lost)

h3. Proposed fix
Use {{!key.startsWith(SCHEMA + SEPARATOR)}} in 
{{CatalogPropertiesUtil#deserializeOptions}}. Add a case to 
{{CatalogPropertiesUtilTest}} with an option such as {{schema-name}} that must 
survive the round trip. Backport to release-2.2 and release-2.3.



> CatalogPropertiesUtil drops table options whose key starts with "schema" 
> (e.g. 'schema-name') when deserializing a catalog table
> --------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-41599
>                 URL: https://issues.apache.org/jira/browse/FLINK-41599
>             Project: Flink
>          Issue Type: Bug
>          Components: Table SQL / API
>    Affects Versions: 2.2.0, 2.3.0, 2.2.1
>            Reporter: Purushottam Sinha
>            Priority: Major
>              Labels: pull-request-available
>
> h3. Problem
> Since Flink 2.2.0, a catalog table that is serialized with 
> {{ResolvedCatalogTable#toProperties}} and read back with 
> {{CatalogTable.fromProperties}} loses every connector option whose key starts 
> with {{schema}}. Only Flink's own schema entries ({{schema.<...>}}) should be 
> excluded.
> Catalogs that persist tables this way therefore lose such options on reload. 
> For example, the Flink CDC {{postgres-cdc}} and {{oracle-cdc}} sources 
> require {{'schema-name'}}. After a reload the table can no longer be used:
> {noformat}
> Unable to create a source for reading table '...'.
> ...
> Missing required options are:
> schema-name
> {noformat}
> {{CREATE TABLE}} succeeds; the first query on the table fails. Catalogs that 
> keep table objects in memory ({{GenericInMemoryCatalog}}) are not affected.
> h3. Cause
> FLINK-38311 (commit 2e7cac1906, apache/flink#26990) merged the two 
> {{deserializeOptions}} overloads and dropped the separator for the schema 
> prefix:
> {code:java}
> // up to 2.1
> key.startsWith(schemaKey + SEPARATOR)          // "schema."
> // 2.2, 2.3, master
> !key.startsWith(DISTRIBUTION + SEPARATOR)      // "distribution."
> && !key.startsWith(PARTITION_KEYS + SEPARATOR) // "partition.keys."
> && !key.startsWith(SCHEMA)                     // "schema" -- separator 
> missing
> {code}
> Every schema property Flink writes is {{schema.<...>}} ({{compoundKey(SCHEMA, 
> ...)}}), so the missing separator only removes user options.
> h3. Reproduction
> {code:java}
> CatalogTable table = CatalogTable.newBuilder()
>     .schema(Schema.newBuilder().column("id", DataTypes.INT()).build())
>     .options(Map.of("connector", "postgres-cdc", "schema-name", "public", 
> "table-name", "orders"))
>     .build();
> ResolvedCatalogTable resolved = new ResolvedCatalogTable(
>     table, ResolvedSchema.of(Column.physical("id", DataTypes.INT())));
> CatalogTable.fromProperties(resolved.toProperties(DefaultSqlFactory.INSTANCE)).getOptions();
> {code}
> Options read back, with {{flink-table-api-java}} from Maven Central:
> * 1.19.3 / 2.0.2 / 2.1.3: {{connector, schema-name, table-name}}
> * 2.2.1 / 2.3.0: {{connector, table-name}} ({{schema-name}} is lost)
> h3. Proposed fix
> Use {{!key.startsWith(SCHEMA + SEPARATOR)}} in 
> {{CatalogPropertiesUtil#deserializeOptions}}. Add a case to 
> {{CatalogPropertiesUtilTest}} with an option such as {{schema-name}} that 
> must survive the round trip. Backport to release-2.2 and release-2.3.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to