Hao Yu created FLINK-41522:
------------------------------
Summary: FileCatalogStore changes YAML-sensitive string options
when reloading catalogs
Key: FLINK-41522
URL: https://issues.apache.org/jira/browse/FLINK-41522
Project: Flink
Issue Type: Bug
Components: Table SQL / API
Affects Versions: 2.4.0
Environment: Flink 2.4
Reporter: Hao Yu
h2. Description
FileCatalogStore does not preserve catalog option strings that require YAML
quoting or escaping.
For example, a generic_in_memory catalog configured with default-database="123"
initially uses the database name 123. After its descriptor is persisted and
loaded from a new FileCatalogStore instance, the default database becomes
'123', including literal single quotation marks.
Other affected values include boolean-looking strings such as "true",
null-looking strings such as "null", "*", empty strings, multiline strings, and
strings containing ": ".
This affects file-backed catalog persistence and reuse across sessions. An
already initialized catalog can continue to work with the original options, so
the changed value may only become visible when the catalog is loaded again.
h2. Reproduction
Configure a file-backed catalog store:
table.catalog-store.kind: file
table.catalog-store.file.path: /tmp/flink-catalog-store
Create a catalog:
CREATE CATALOG demo WITH (
'type' = 'generic_in_memory',
'default-database' = '123'
);
Load the catalog in a fresh session using the same catalog-store directory,
then inspect its databases:
USE CATALOG demo;
SHOW DATABASES;
The underlying failure has been reproduced with a deterministic regression test
that:
Stores a CatalogDescriptor in FileCatalogStore.
Closes the store and opens a new instance pointing to the same directory.
Retrieves the descriptor and creates the catalog through FactoryUtil.
Checks the default database name and whether database "123" exists.
The full SQL Gateway session sequence above has not been exercised as an
end-to-end process test.
h2. Expected behavior
Catalog option strings should survive persistence unchanged.
The reloaded catalog should have default database "123", without additional
quotation marks.
Actual behavior
The default database assertion fails:
expected: "123"
but was: "'123'"
The single quotation marks become part of the configuration value rather than
serving as YAML syntax.
h2. Root cause
FileCatalogStore.storeCatalog passes Configuration.toFileWritableMap() to
YAMLMapper.writeValue().
toFileWritableMap() already converts each value into YAML scalar text.
YAMLMapper then serializes that text again, treating the quoting or escaping
introduced by the first conversion as part of the string.
getCatalog() decodes only the outer YAML layer and passes the resulting strings
to Configuration.fromMap(), retaining the extra quoting or escaping.
h2. Compatibility
The proposed fix corrects newly persisted catalog definitions. It does not
automatically repair definitions already written with extra YAML encoding.
Affected definitions may need to be recreated from their original option values.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)