Hongshun Wang created FLINK-40349:
-------------------------------------
Summary: Add Fluss sink schema compatibility option to avoid
silent data loss
Key: FLINK-40349
URL: https://issues.apache.org/jira/browse/FLINK-40349
Project: Flink
Issue Type: Improvement
Components: Flink CDC
Affects Versions: cdc-3.5.0
Reporter: Hongshun Wang
Fix For: cdc-3.6.0
h2. {{Motivation}}
{{Currently, the Fluss sink supports schema mapping between the input schema
and the target Fluss table schema. This behavior is useful because Fluss only
supports limited schema evolution today, so the upstream and downstream schemas
may not always be exactly the same.}}
{{}}
{{However, the current default behavior may silently drop data when the
upstream input schema contains columns that do not exist in the downstream
Fluss table schema. For example, if the upstream schema is wider than the
target Fluss table schema, the extra input columns can be ignored during
sink-side schema mapping, and users may not notice that data is being lost.
We should provide an explicit sink option to control how strictly the sink
validates schema compatibility.}}
{{}}
h2.
h2. {{Proposal}}
{{Introduce a new Flink sink option, tentatively named:
schema-validation-mode.}}
{{}}
The option controls the allowed relationship between the input schema and the
target Fluss table schema.
Supported values:
* {{permissive: }}
Default value. Keeps the current behavior for backward compatibility.
The sink may perform schema mapping, projection, and padding based on the
target Fluss table schema. This allows existing jobs to continue working as
before.
*
{{target-superset:}}
The target Fluss table schema must contain all fields from the input schema by
field name.
This mode allows the target schema to be wider than the input schema, which is
useful for schema evolution such as adding nullable columns to the Fluss table.
However, it rejects the case where the input schema contains columns that do
not exist in the target Fluss table schema, preventing silent data loss.
*
{{strict: }}
The input schema and target Fluss table schema must be strictly identical.
The validation should check field count, field names, field order, and field
types. If any mismatch is found, the sink should fail during initialization
with a clear validation error.
{{{}{}}}{{{}{}}}
{{{}{}}}{{{}{}}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
