[
https://issues.apache.org/jira/browse/FLINK-40303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qilong Wang updated FLINK-40303:
--------------------------------
Description:
*Problem*
For an upsert pipeline where the source produces key-only DELETE records and
the sink accepts key-only DELETE records, ChangelogNormalize can be eliminated.
A key-only DELETE contains values only for primary key fields. All non-key
fields are set to null, regardless of their NOT NULL constraints. However, the
sink constraint enforcer currently validates every NOT NULL field and rejects
such records.
For example, given a sink schema with:
- `id INT PRIMARY KEY NOT ENFORCED`
- `name STRING NOT NULL`
the following valid key-only DELETE fails:
`-D[1, null]`
with an error stating that column `name` must not be null.
*Expected Behavior*
For key-only DELETE records, NOT NULL constraints should only be enforced on
primary key fields. Non-key fields have no value semantics and should not be
validated.
INSERT, UPDATE, and full DELETE records should continue to enforce all NOT NULL
constraints.
*Root Cause*
`CommonExecSink` did not pass the input changelog's key-only DELETE semantic or
the sink primary key indices to `ConstraintEnforcerExecutor`.
As a result, `NotNullConstraint` could not distinguish null placeholders in
key-only DELETE records from actual null values.
*Proposed Fix*
- Pass `ChangelogMode.keyOnlyDeletes()` and the sink primary key indices from
`CommonExecSink` to `ConstraintEnforcerExecutor`.
- Let `NotNullConstraint` skip non-key fields when processing a key-only DELETE.
- Continue enforcing NOT NULL constraints on primary key fields and on all
fields for other changelog modes.
- Add a semantic test covering a key-only DELETE written to a sink with a NOT
NULL non-key column.
*Verification*
Extend `DeletesByKeySemanticTests` with a source that emits `-D[1, null]` and a
delete-by-key sink whose non-key column is declared NOT NULL.v
was:
# Problem
For an upsert pipeline where the source produces key-only DELETE records and
the sink accepts key-only DELETE records, ChangelogNormalize can be eliminated.
A key-only DELETE contains values only for primary key fields. All non-key
fields are set to null, regardless of their NOT NULL constraints. However, the
sink constraint enforcer currently validates every NOT NULL field and rejects
such records.
For example, given a sink schema with:
- `id INT PRIMARY KEY NOT ENFORCED`
- `name STRING NOT NULL`
the following valid key-only DELETE fails:
`-D[1, null]`
with an error stating that column `name` must not be null.
# Expected Behavior
For key-only DELETE records, NOT NULL constraints should only be enforced on
primary key fields. Non-key fields have no value semantics and should not be
validated.
INSERT, UPDATE, and full DELETE records should continue to enforce all NOT NULL
constraints.
# Root Cause
`CommonExecSink` did not pass the input changelog's key-only DELETE semantic or
the sink primary key indices to `ConstraintEnforcerExecutor`.
As a result, `NotNullConstraint` could not distinguish null placeholders in
key-only DELETE records from actual null values.
# Proposed Fix
- Pass `ChangelogMode.keyOnlyDeletes()` and the sink primary key indices from
`CommonExecSink` to `ConstraintEnforcerExecutor`.
- Let `NotNullConstraint` skip non-key fields when processing a key-only DELETE.
- Continue enforcing NOT NULL constraints on primary key fields and on all
fields for other changelog modes.
- Add a semantic test covering a key-only DELETE written to a sink with a NOT
NULL non-key column.
# Verification
Extend `DeletesByKeySemanticTests` with a source that emits `-D[1, null]` and a
delete-by-key sink whose non-key column is declared NOT NULL.
> Sink constraint enforcer rejects key-only DELETE records with NOT NULL
> non-key columns
> --------------------------------------------------------------------------------------
>
> Key: FLINK-40303
> URL: https://issues.apache.org/jira/browse/FLINK-40303
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner, Table SQL / Runtime
> Affects Versions: 2.4.0
> Reporter: Qilong Wang
> Priority: Major
> Fix For: 2.4.0
>
>
> *Problem*
> For an upsert pipeline where the source produces key-only DELETE records and
> the sink accepts key-only DELETE records, ChangelogNormalize can be
> eliminated.
> A key-only DELETE contains values only for primary key fields. All non-key
> fields are set to null, regardless of their NOT NULL constraints. However,
> the sink constraint enforcer currently validates every NOT NULL field and
> rejects such records.
> For example, given a sink schema with:
> - `id INT PRIMARY KEY NOT ENFORCED`
> - `name STRING NOT NULL`
> the following valid key-only DELETE fails:
> `-D[1, null]`
> with an error stating that column `name` must not be null.
> *Expected Behavior*
> For key-only DELETE records, NOT NULL constraints should only be enforced on
> primary key fields. Non-key fields have no value semantics and should not be
> validated.
> INSERT, UPDATE, and full DELETE records should continue to enforce all NOT
> NULL constraints.
> *Root Cause*
> `CommonExecSink` did not pass the input changelog's key-only DELETE semantic
> or the sink primary key indices to `ConstraintEnforcerExecutor`.
> As a result, `NotNullConstraint` could not distinguish null placeholders in
> key-only DELETE records from actual null values.
> *Proposed Fix*
> - Pass `ChangelogMode.keyOnlyDeletes()` and the sink primary key indices from
> `CommonExecSink` to `ConstraintEnforcerExecutor`.
> - Let `NotNullConstraint` skip non-key fields when processing a key-only
> DELETE.
> - Continue enforcing NOT NULL constraints on primary key fields and on all
> fields for other changelog modes.
> - Add a semantic test covering a key-only DELETE written to a sink with a NOT
> NULL non-key column.
> *Verification*
> Extend `DeletesByKeySemanticTests` with a source that emits `-D[1, null]` and
> a delete-by-key sink whose non-key column is declared NOT NULL.v
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
