[
https://issues.apache.org/jira/browse/FLINK-41107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18130803#comment-18130803
]
Xin Chen commented on FLINK-41107:
----------------------------------
h3. Impact
For the MaxCompute sink, this can cause destructive operations to be applied
even when users explicitly exclude them:
* {{TruncateTableEvent}} invokes
{{{}SchemaEvolutionUtils.truncateTable(...){}}}.
* {{DropTableEvent}} invokes {{{}SchemaEvolutionUtils.dropTable(...){}}}.
For the Hudi sink, {{TruncateTableEvent}} and {{DropTableEvent}} are not
supported. Since excluded events are still forwarded to the sink, the pipeline
may fail with an unsupported-operation exception instead of silently ignoring
the excluded event.
Other pipeline sinks, such as Doris, StarRocks, Paimon, Iceberg, Kafka, and
Fluss, already implement accepted schema evolution type filtering.
> schema.change.behavior: lenient or exclude.schema.changes is ignored by Hudi
> and MaxCompute sinks because their MetadataApplier implementations do not
> honor accepted schema evolution types
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-41107
> URL: https://issues.apache.org/jira/browse/FLINK-41107
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
> Affects Versions: cdc-3.6.0, cdc-3.7.0
> Reporter: Xin Chen
> Priority: Major
>
> HudiMetadataApplier and MaxComputeMetadataApplier do not honor the schema
> evolution event types configured through include.schema.changes and
> exclude.schema.changes.
> In the Pipeline runtime, the resolved allowed schema change event types are
> passed to each sink through:
>
> {code:java}
> MetadataApplier#setAcceptedSchemaEvolutionTypes(...){code}
>
>
> Before forwarding an event to the sink, SchemaDerivator checks:
>
> {code:java}
> metadataApplier.acceptsSchemaEvolutionType(schemaChangeEvent.getType()){code}
>
> However, both Hudi and MaxCompute metadata appliers rely on the default
> methods in MetadataApplier:
>
> {code:java}
> default MetadataApplier setAcceptedSchemaEvolutionTypes(
> Set<SchemaChangeEventType> schemaEvolutionTypes) {
> return this;
> }
> default boolean acceptsSchemaEvolutionType(
> SchemaChangeEventType schemaChangeEventType) {
> return true;
> }
> {code}
>
> As a result, the configured accepted event types are discarded and every
> schema change event is accepted.
> This makes configurations such as the following ineffective:
>
> {code:java}
> sink:
> exclude.schema.changes:
> - truncate.table
> - drop.table{code}
> or
> {code:java}
> schema.change.behavior: lenient{code}
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
