[
https://issues.apache.org/jira/browse/FLINK-33182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17773229#comment-17773229
]
Timo Walther edited comment on FLINK-33182 at 10/9/23 10:22 AM:
----------------------------------------------------------------
[~lincoln.86xy] not sure if I understand your comment correctly. I understand
that -D is dangerous and could cause non-determinism. However,
ChangelogNormalize does not use the message value of -D. It reconstructs the -D
from the previously stored row as can be seen in
`DeduplicateFunctionHelper#processLastRowOnChangelog`. So metadata columns
cannot affect determinism in this case.
Note that only the primary key is used for equality. Metadata columns are not
supported as primary key columns.
was (Author: twalthr):
[~lincoln.86xy] not sure if I understand your comment correctly. I understand
that -D is dangerous and could cause non-determinism. However,
ChangelogNormalize does not use the message value of -D. It reconstructs the -D
from the previously stored row as can be seen in
`DeduplicateFunctionHelper#processLastRowOnChangelog`. So metadata columns
cannot affect determinism in this case.
> Allow metadata columns in NduAnalyzer with ChangelogNormalize
> -------------------------------------------------------------
>
> Key: FLINK-33182
> URL: https://issues.apache.org/jira/browse/FLINK-33182
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
> Reporter: Timo Walther
> Priority: Major
>
> Currently, the NduAnalyzer is very strict about metadata columns in updating
> sources. However, for upsert sources (like Kafka) that contain an incomplete
> changelog, the planner always adds a ChangelogNormalize node.
> ChangelogNormalize will make sure that metadata columns can be considered
> deterministic. So the NduAnalyzer should be satisfied in this case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
