Purushottam Sinha created FLINK-41599:
-----------------------------------------
Summary: CatalogPropertiesUtil drops table options whose key
starts with "schema" (e.g. 'schema-name') when deserializing a catalog table
Key: FLINK-41599
URL: https://issues.apache.org/jira/browse/FLINK-41599
Project: Flink
Issue Type: Bug
Components: Table SQL / API
Affects Versions: 2.2.1, 2.2.0, 2.3.0
Reporter: Purushottam Sinha
h3. Problem
Since Flink 2.2.0, a catalog table that is serialized with
{{ResolvedCatalogTable#toProperties}} and read back with
{{CatalogTable.fromProperties}} loses every connector option whose key starts
with {{schema}}. Only Flink's own schema entries ({{schema.<...>}}) should be
excluded.
Catalogs that persist tables this way therefore lose such options on reload.
For example, the Flink CDC {{postgres-cdc}}, {{oracle-cdc}} and {{db2-cdc}}
sources require {{'schema-name'}}. After a reload the table can no longer be
used:
{noformat}
Unable to create a source for reading table '...'.
...
Missing required options are:
schema-name
{noformat}
{{CREATE TABLE}} succeeds; the first query on the table fails. Catalogs that
keep table objects in memory ({{GenericInMemoryCatalog}}) are not affected.
h3. Cause
FLINK-38311 (commit 2e7cac1906, apache/flink#26990) merged the two
{{deserializeOptions}} overloads and dropped the separator for the schema
prefix:
{code:java}
// up to 2.1
key.startsWith(schemaKey + SEPARATOR) // "schema."
// 2.2, 2.3, master
!key.startsWith(DISTRIBUTION + SEPARATOR) // "distribution."
&& !key.startsWith(PARTITION_KEYS + SEPARATOR) // "partition.keys."
&& !key.startsWith(SCHEMA) // "schema" -- separator missing
{code}
Every schema property Flink writes is {{schema.<...>}} ({{compoundKey(SCHEMA,
...)}}), so the missing separator only removes user options.
h3. Reproduction
{code:java}
CatalogTable table = CatalogTable.newBuilder()
.schema(Schema.newBuilder().column("id", DataTypes.INT()).build())
.options(Map.of("connector", "postgres-cdc", "schema-name", "public",
"table-name", "orders"))
.build();
ResolvedCatalogTable resolved = new ResolvedCatalogTable(
table, ResolvedSchema.of(Column.physical("id", DataTypes.INT())));
CatalogTable.fromProperties(resolved.toProperties(DefaultSqlFactory.INSTANCE)).getOptions();
{code}
Options read back, with {{flink-table-api-java}} from Maven Central:
* 1.19.3 / 2.0.2 / 2.1.3: {{connector, schema-name, table-name}}
* 2.2.1 / 2.3.0: {{connector, table-name}} ({{schema-name}} is lost)
h3. Proposed fix
Use {{!key.startsWith(SCHEMA + SEPARATOR)}} in
{{CatalogPropertiesUtil#deserializeOptions}}. Add a case to
{{CatalogPropertiesUtilTest}} with an option such as {{schema-name}} that must
survive the round trip. Backport to release-2.2 and release-2.3.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)