[
https://issues.apache.org/jira/browse/FLINK-24608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17440522#comment-17440522
]
Marios Trivyzas commented on FLINK-24608:
-----------------------------------------
After discussion with [~twalthr] it seems there are 2 ways to implement this.
There is an issue with *SinkUpsertMaterializer* which keeps a state, and its
state is a leas of `RowData` so any information on the *StreamRecord* is lost,
so:
# Keep any constraint validation operator before the *SinkUpsertMaterializer*
and add a new operator at the end (after the {*}SinkUpsertMaterializer{*}) to
set the timestamp on the *StreamRecord*
# Combine the constraint validation operator with this new one which sets the
timestamp, and apply it after the *SinkUpsertMaterializer*
(Currently the constraint validation in place is only the
{*}SinkNotNullEnforcer{*})
With *1* we could already filter out records, or shorter the length of
char/varchar fields of a record (soon to be implemented) before the
*SinkUpsertMaterializer* thus decreasing the size of its state.
With *2* we loose this possible memory gain from *1* but we avoid introducing
yet another operator in the pipeline.
[~fabian.paul], [~jark] Could you please share your opinion on this?
> Sinks built with the unified sink framework do not receive timestamps when
> used in Table API
> --------------------------------------------------------------------------------------------
>
> Key: FLINK-24608
> URL: https://issues.apache.org/jira/browse/FLINK-24608
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common, Table SQL / Planner
> Affects Versions: 1.14.0, 1.13.3, 1.15.0
> Reporter: Fabian Paul
> Assignee: Marios Trivyzas
> Priority: Critical
>
> All sinks built with the unified sink framework extract the timestamp from
> the internal {{StreamRecord}}. The Table API does not facilitate the
> timestamp field in the {{StreamRecord}} but extracts the timestamp from the
> actual data.
> We either have to use a dedicated operator before all the sinks to simulate
> the behavior or allow a customizable timestamp extraction during the sink
> translation.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
