Hao Yu created FLINK-41522:
------------------------------

             Summary: FileCatalogStore changes YAML-sensitive string options 
when reloading catalogs
                 Key: FLINK-41522
                 URL: https://issues.apache.org/jira/browse/FLINK-41522
             Project: Flink
          Issue Type: Bug
          Components: Table SQL / API
    Affects Versions: 2.4.0
         Environment: Flink 2.4
            Reporter: Hao Yu


h2. Description

FileCatalogStore does not preserve catalog option strings that require YAML 
quoting or escaping.

For example, a generic_in_memory catalog configured with default-database="123" 
initially uses the database name 123. After its descriptor is persisted and 
loaded from a new FileCatalogStore instance, the default database becomes 
'123', including literal single quotation marks.

Other affected values include boolean-looking strings such as "true", 
null-looking strings such as "null", "*", empty strings, multiline strings, and 
strings containing ": ".

This affects file-backed catalog persistence and reuse across sessions. An 
already initialized catalog can continue to work with the original options, so 
the changed value may only become visible when the catalog is loaded again.
h2. Reproduction

Configure a file-backed catalog store:

table.catalog-store.kind: file
table.catalog-store.file.path: /tmp/flink-catalog-store

Create a catalog:

CREATE CATALOG demo WITH (
'type' = 'generic_in_memory',
'default-database' = '123'
);

Load the catalog in a fresh session using the same catalog-store directory, 
then inspect its databases:

USE CATALOG demo;
SHOW DATABASES;

The underlying failure has been reproduced with a deterministic regression test 
that:

Stores a CatalogDescriptor in FileCatalogStore.
Closes the store and opens a new instance pointing to the same directory.
Retrieves the descriptor and creates the catalog through FactoryUtil.
Checks the default database name and whether database "123" exists.

The full SQL Gateway session sequence above has not been exercised as an 
end-to-end process test.
h2. Expected behavior

Catalog option strings should survive persistence unchanged.

The reloaded catalog should have default database "123", without additional 
quotation marks.

Actual behavior

The default database assertion fails:

expected: "123"
but was: "'123'"

The single quotation marks become part of the configuration value rather than 
serving as YAML syntax.
h2. Root cause

FileCatalogStore.storeCatalog passes Configuration.toFileWritableMap() to 
YAMLMapper.writeValue().

toFileWritableMap() already converts each value into YAML scalar text. 
YAMLMapper then serializes that text again, treating the quoting or escaping 
introduced by the first conversion as part of the string.

getCatalog() decodes only the outer YAML layer and passes the resulting strings 
to Configuration.fromMap(), retaining the extra quoting or escaping.
h2. Compatibility

The proposed fix corrects newly persisted catalog definitions. It does not 
automatically repair definitions already written with extra YAML encoding. 
Affected definitions may need to be recreated from their original option values.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to