From my understanding, for a sink, if its schema includes a primary key,
we can assume it has
the ability to process delete messages (with '-D') and perform deletions
by key (PK). If it does not
include a PK, we would implicitly treat it as a log-structured table that
supports full row deletions.
I am afraid this assumption is too far going. PK is information about
columns uniqueness and that's it. It does not tell us what is required to
perform a DELETE operation. I agree the assumption would most often hold,
but I am afraid it is not guaranteed. E.g. In a log based systems one may
just want to have full information encoded in the DELETE messages. (e.g. in
a debezium message)
Same holds for sources. Even though theoretically, if there is a PK,
deletes could contain only the key information, but the source may just as
well produce DELETEs with all fields set.
Given that you mentioned `PARTIAL_DELETE`, should I interpret this as
referring to a scenario
similar to wide tables, where if the sink has a PK, some columns are
deleted (set to null or through other operations) while others remain
unchanged?
No. The effect is the same. That the ROW is deleted/disappears. The
difference is what is required to perform the deletion. In some cases it
may be enough to have the PK to perform the deletion and then we don't need
the information about other columns, but there may be systems that require
all columns to be set.
By the way, since the flag applies both for sources and sinks to tell what
is the expected format of DELETE records produced/consumed I renamed the
flag in the FLIP:
supportsDeleteByKey -> deletesByKeOnly.
Let me know if there are other questions. If there are none, I'd like to
start a vote in the upcoming days.
Best,
Dawid
On Mon, 3 Mar 2025 at 07:29, Xuyang <xyzhong...@163.com> wrote:
Hi, Dawid.
Thanks for your response. I believe I've identified a key point, but I’m a
bit unclear about the
following you said. Could you please provide an example for clarification?
```
The only missing information is if the external sink can consume deletes
by key and if a source
produces full deletes or deletes by key.
```
From my understanding, for a sink, if its schema includes a primary key,
we can assume it has
the ability to process delete messages (with '-D') and perform deletions
by key (PK). If it does not
include a PK, we would implicitly treat it as a log-structured table that
supports full row deletions.
Given that you mentioned `PARTIAL_DELETE`, should I interpret this as
referring to a scenario
similar to wide tables, where if the sink has a PK, some columns are
deleted (set to null or through
other operations) while others remain unchanged?
Looking forward your reply.
--
Best！
Xuyang
At 2025-02-28 19:16:12, "Dawid Wysakowicz" <wysakowicz.da...@gmail.com>
wrote:
Hey Xuyang,
Ad. 1
Yes, you're right, but we already do that for determining if we need
UPDATE_BEFORE or not. FlinkChangelogModeInferenceProgram already deals
with
that.
Ad. 2
Unfortunately it is. This is also the only reason I need a FLIP. We can
determine internally for every internal operator if we can work with
partial deletes or if we need full deletes. The only missing information
is
if the external sink can consume deletes by key and if a source produces
full deletes or deletes by key. Unfortunately this is information that
comes from a connector implementation and thus needs to be provided via a
public API.
Ad. 3
With ChangelogMode#kinds -> to some degree yes. We theoretically could
split RowKind#DELETE to RowKind#DELETE_BY_KEY and RowKind#FULL_DELETE.
However, that change would 1) be much more involved 2) we would need to
encode that information in every single message, which I think is not
necessary. I don't think it has much to do with PK.
Ad.4
I don't think so. PK information is part of Schema not about the kind of
messages. We don't have PK information for UPDATE_BEFORE/UPDATE_AFTER and
they also apply per key. If the name containing `DELETE_BY_KEY` is
confusing I am happy to rename it to e.g. PARTIAL_DELETE, therefore I'd
add
`supportsPartialDeletes`
Best,
Dawid
On Fri, 28 Feb 2025 at 04:43, Xuyang <xyzhong...@163.com> wrote:
Hi Dawid.
Big +1 for this FLIP. After reading through it, I have a few questions
and
would appreciate your responses:
1. IIUC, we only need to provide additional information in the
`FlinkChangelogModeInferenceProgram` to enable the
inference program to determine whether it is safe to remove
`ChangelogNormalize`. My first instinct is that we need to
know if all subsequent output-side nodes consuming Upsert Keys include
the
Upsert Keys provided by the input-side operator (source).
If this condition is met, we can safely eliminate `ChangelogNormalize`.
Perhaps, I have missed some important points, so please feel
free to correct me if necessary.
2. The introduction of `supportsDeleteByKey` in ChangelogMode seems to
exist solely as auxiliary information for the
`FlinkChangelogModeInferenceProgram`. If that's the case, it doesn't
seem
necessary to expose it in the public API, does it?
3. If the purpose of introducing `supportsDeleteByKey` in ChangelogMode
is
to facilitate support for `#fromChangelogStream`
and `#toChangelogStream`, it appears that `supportsDeleteByKey` might
overlap with ChangelogMode#kinds and Schema#PK
to some extent, right?
4. Regarding supportsDeleteByKey, as part of a complete ChangelogMode
entity, should we also store the specific key information?
--
Best！
Xuyang
在 2025-02-28 04:27:19，"Martijn Visser" <martijnvis...@apache.org> 写道：
Hi Dawid,
Thanks for the FLIP, looks like a good improvement for me that will
bring
a
lot of benefits. +1
Best regards,
Martijn
On Tue, Feb 25, 2025 at 6:51 AM Sergey Nuyanzin <snuyan...@gmail.com>
wrote:
+1 for such improvement
On Mon, Feb 24, 2025 at 12:01 PM Dawid Wysakowicz
<wysakowicz.da...@gmail.com> wrote:
Hi everyone,
I would like to initiate a discussion for the FLIP-510[1] below,
which
aims
on optimising certain use cases in SQL which at the moment add
ChangelogNormalize, but don't necessarily need to do it.
Looking forward to hearing from you.
[1] https://cwiki.apache.org/confluence/x/7o5EF
--
Best regards,
Sergey
