KevinyhZou opened a new issue, #12300:
URL: https://github.com/apache/gluten/issues/12300
### Description
In current `Gluten-Flink`, `watermark` is defined as a single operator, for
instance, in the sql below
```
CREATE TABLE kafka (
event_type int,
person ROW<
......,
`dateTime` TIMESTAMP(3)>,
auction ROW<
......,
`dateTime` TIMESTAMP(3)>,
bid ROW<
.......,
`dateTime` TIMESTAMP(3)>,
`dateTime` AS
CASE
WHEN event_type = 0 THEN person.`dateTime`
WHEN event_type = 1 THEN auction.`dateTime`
ELSE bid.`dateTime`
END,
WATERMARK FOR `dateTime` AS `dateTime` - INTERVAL '4' SECOND
) WITH (
.....
);
...
create view bid as select * ....
insert into $sink_table select .... from bid
```
the execution plan would be `Source -> WatermarkAssigner -> .... -> Sink`,
`watermark` operator is transformed into a single operator `WatermarkAssigner`,
which maybe inefficient and different from current flink's implementation.
`Watermark` should be push down to streaming source operator, and generate
by source. By doing this, there would be less operators to be scheduled, less
data to be serialized / deserialized, and peformance would be better.
### Gluten version
main branch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
