[
https://issues.apache.org/jira/browse/FLINK-40987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gustavo de Morais updated FLINK-40987:
--------------------------------------
Description:
A table's watermark can only be changed through DDL (ALTER TABLE ...
MODIFY/DROP WATERMARK, or CREATE TABLE ... LIKE). This affects every query on
the table or requires a second table. The OPTIONS hint (FLIP-296) can tune
watermark emission, but not the definition itself.
Motivation: the same table is often read by queries with different needs. A
dashboard wants a short delay, a billing job tolerates an hour of late data, a
copy job needs no watermark. Connectors are also starting to override the
watermark inside the source, without the planner knowing (apache/fluss#3265).
Idea: let the OPTIONS hint override the watermark for one table reference, with
the same semantics as the DDL:
{code:java}
SELECT * FROM orders /*+ OPTIONS('scan.watermark.delay' = '1 h') */;
SELECT * FROM orders /*+ OPTIONS('scan.watermark.column' = 'ship_time') */;
SELECT * FROM orders /*+ OPTIONS('scan.watermark.enabled' = 'false') */;{code}
No new semantics: each hint behaves like MODIFY/DROP WATERMARK, scoped to one
table reference. The hinted watermark would be applied during validation,
reusing the CREATE TABLE / LIKE schema resolution, so planner and source stay
consistent.
This ticket is to collect feedback. Option names are placeholders (a single
watermark-expression option is an alternative).
was:
A table's watermark definition can only be changed through DDL (ALTER TABLE ...
MODIFY/DROP WATERMARK, or CREATE TABLE ... LIKE), which affects every query on
that table or requires a second table.
The OPTIONS hint already allows tuning watermark emission per table reference
(scan.watermark.emit.strategy, idle-timeout, alignment; FLIP-296). It cannot
change the delay or turn the watermark off.
Idea: two more options handled by the planner, for example:
{code:java}
SELECT * FROM orders /*+ OPTIONS('scan.watermark.delay' = '1 h') */;
SELECT * FROM orders /*+ OPTIONS('scan.watermark.enabled' = 'false') */;{code}
This adds no new semantics. The result is the same as MODIFY/DROP WATERMARK,
scoped to one table reference in one statement. Changing the watermark column
is out of scope.
Motivation: connectors are starting to implement this themselves
([apache/fluss#3265|https://github.com/apache/fluss/pull/3266]), outside the
planner.
This ticket is to collect feedback on the idea and scope. Option names are
placeholders.
> Allow overriding a table's watermark per table reference via OPTIONS hint
> -------------------------------------------------------------------------
>
> Key: FLINK-40987
> URL: https://issues.apache.org/jira/browse/FLINK-40987
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
> Reporter: Gustavo de Morais
> Priority: Major
>
> A table's watermark can only be changed through DDL (ALTER TABLE ...
> MODIFY/DROP WATERMARK, or CREATE TABLE ... LIKE). This affects every query on
> the table or requires a second table. The OPTIONS hint (FLIP-296) can tune
> watermark emission, but not the definition itself.
> Motivation: the same table is often read by queries with different needs. A
> dashboard wants a short delay, a billing job tolerates an hour of late data,
> a copy job needs no watermark. Connectors are also starting to override the
> watermark inside the source, without the planner knowing (apache/fluss#3265).
> Idea: let the OPTIONS hint override the watermark for one table reference,
> with the same semantics as the DDL:
> {code:java}
> SELECT * FROM orders /*+ OPTIONS('scan.watermark.delay' = '1 h') */;
> SELECT * FROM orders /*+ OPTIONS('scan.watermark.column' = 'ship_time') */;
> SELECT * FROM orders /*+ OPTIONS('scan.watermark.enabled' = 'false') */;{code}
> No new semantics: each hint behaves like MODIFY/DROP WATERMARK, scoped to one
> table reference. The hinted watermark would be applied during validation,
> reusing the CREATE TABLE / LIKE schema resolution, so planner and source stay
> consistent.
> This ticket is to collect feedback. Option names are placeholders (a single
> watermark-expression option is an alternative).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
