sepuri sai krishna created FLINK-41007:
------------------------------------------
Summary: ON CONFLICT clause is silently ignored when
table.exec.sink.upsert-materialize is NONE
Key: FLINK-41007
URL: https://issues.apache.org/jira/browse/FLINK-41007
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Affects Versions: 2.3.0, 2.4.0
Reporter: sepuri sai krishna
With {{table.exec.sink.upsert-materialize}} set to {{NONE}}, an {{ON CONFLICT}}
clause is accepted
by the planner and then has no effect. {{DO ERROR}} does not fail on a primary
key conflict,
{{DO NOTHING}} does not keep the first record, and {{DO DEDUPLICATE}} does not
deduplicate.
h3. Reproducer
Same query and sink, only {{table.exec.sink.upsert-materialize}} differs.
{code:sql}
CREATE TABLE src (a INT, b BIGINT, ts TIMESTAMP_LTZ(3), WATERMARK FOR ts AS ts)
WITH ('connector' = 'datagen');
CREATE TABLE snk (id INT, cnt BIGINT, PRIMARY KEY (id) NOT ENFORCED)
WITH ('connector' = 'blackhole');
INSERT INTO snk SELECT MAX(a), COUNT(*) FROM src GROUP BY b ON CONFLICT DO
ERROR;
{code}
With {{AUTO}}, the default:
{noformat}
Sink(table=[...snk], fields=[EXPR$0, EXPR$1], upsertMaterialize=[true],
conflictStrategy=[ERROR])
+- Calc(select=[EXPR$0, EXPR$1])
+- GroupAggregate(groupBy=[b], select=[b, MAX(a) AS EXPR$0, COUNT(*) AS
EXPR$1])
{noformat}
With {{NONE}}:
{noformat}
Sink(table=[...snk], fields=[EXPR$0, EXPR$1], conflictStrategy=[ERROR])
+- Calc(select=[EXPR$0, EXPR$1])
+- GroupAggregate(groupBy=[b], select=[b, MAX(a) AS EXPR$0, COUNT(*) AS
EXPR$1])
{noformat}
The strategy is still recorded on the sink, but there is no operator left in
the plan that could
apply it, so the clause is a no-op and conflicts pass through unresolved.
h3. The clause is still validated under NONE
{{DO ERROR}} and {{DO NOTHING}} keep their watermark requirement under
{{NONE}}. With a source
that has no watermark, {{AUTO}} and {{NONE}} both fail with the identical
message:
{noformat}
ON CONFLICT DO ERROR requires all source tables to define watermarks, but the
following source(s)
do not: default_catalog.default_database.src. Please add a WATERMARK
declaration to these tables.
{noformat}
So a user is required to add watermarks to satisfy a clause that will not be
applied.
h3. Expected
Either the combination is rejected at planning time, the way an {{ON CONFLICT}}
clause is already
rejected for a sink that cannot use it, or the documentation states that the
clause has no effect
when the materializer is disabled.
h3. Affects
{{master}} and {{release-2.3}}.
Related to FLINK-40924, which covers the {{FORCE}} side of the same validation,
and to FLINK-40899.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
