[
https://issues.apache.org/jira/browse/CALCITE-4146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17170512#comment-17170512
]
liupengcheng edited comment on CALCITE-4146 at 8/4/20, 3:02 AM:
----------------------------------------------------------------
>hmmm the EMIT will be at end of SQL query and it will propagate through every
>relational operator. However, technically, I feel like >only
>TableFuncitonScanRel that works with a stream will be affected. You can think
>of that a macro batch streaming system >implements it. In that system, EMTI
>controls the size and frequency of each macro batch (from
>TableFuncitonScanRel), and all >other join, sort, aggregate are applied on
>each macro batch.
Hi, [~amaliujia],
Yes, I know this worked like a micro batch, but in real implementation for
streams, some operators upon a TableFunctionScanRel should know where is end of
the micro batch and when it should perform the calculation and EMIT data to
it's upstream(e.g. two stream join of windowTableFunctionScanRel), but others
may not(e.g., map like operations). So my doubt is that will the EMIT syntax
affect both the TableFunctionScanRel and all it's downstream operators?
E.g. The following example:
```
select * from windowTableFunctionScanRel1 t1 join windowTableFunctionScanRel2
t2 on t1.id = t2.id EMIT after watermark;
```
There are probably two implementations:
1. The EMIT is only bind to the window join and the windowTableFunctionScanRel
just worked like a map(append some extra window attributes)?
2. The EMIT is bind to windowTableFunctionScanRel and all the downstream
operators of windowTableFunctionScanRel . So The windowTableFunctionScanRel
will buffer data and EMIT as the specified strategy(e.g. every 1 minute), and
so do the window join.
which one is preferred?
was (Author: liupengcheng):
>hmmm the EMIT will be at end of SQL query and it will propagate through every
>relational operator. However, technically, I feel like >only
>TableFuncitonScanRel that works with a stream will be affected. You can think
>of that a macro batch streaming system >implements it. In that system, EMTI
>controls the size and frequency of each macro batch (from
>TableFuncitonScanRel), and all >other join, sort, aggregate are applied on
>each macro batch.
Hi, [~amaliujia],
Yes, I know this worked like a micro batch, but in real implementation for
streams, operators upon a TableFunctionScanRel should know where is end of the
micro batch and when it should perform the calculation and EMIT data to it's
upstream(e.g. two stream join of windowTableFunctionScanRel), so my doubt is
that will the EMIT syntax affect both the TableFunctionScanRel and it's
downstream operators?
E.g. The following example:
```
select * from windowTableFunctionScanRel1 t1 join windowTableFunctionScanRel2
t2 where t1.id = t2.id EMIT after watermark;
```
There are probably two implementations:
1. The EMIT is only bind to the window join and the windowTableFunctionScanRel
just worked like a map(append some extra window attributes)?
2. The EMIT is bind to windowTableFunctionScanRel and all the downstream
operators of windowTableFunctionScanRel . So The windowTableFunctionScanRel
will buffer data and EMIT as the specified strategy(e.g. every 1 minute), and
so do the window join.
which one is preferred?
> Implement EMIT Syntax
> ---------------------
>
> Key: CALCITE-4146
> URL: https://issues.apache.org/jira/browse/CALCITE-4146
> Project: Calcite
> Issue Type: New Feature
> Reporter: Rui Wang
> Assignee: Rui Wang
> Priority: Major
>
> The goal is to support the following syntax:
> {code:sql}
> SELECT clause
> FROM TUMBLE/HOP/SESSION
> EMIT AFTER WATERMARK
> {code}
> note that "EMIT AFTER WATERMARK" is the new thing.
> "EMIT AFTER WATERMARK" is proposed in [One SQL to Rule Them
> All|https://arxiv.org/pdf/1905.12133.pdf]. This idea proposes a way to allow
> streaming SQL queries control materialization latency. More specifically, it
> means emit elements in a window once the watermark passes the end of that
> window.
> There are more context discussed in
> [CALCITE-3272|https://issues.apache.org/jira/browse/CALCITE-3272?focusedCommentId=17166580&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17166580]
> and the [EMIT syntax proposal for event-timestamp semantic
> windowing|https://lists.apache.org/thread.html/r5bd9a6f7af2c0cd81aecd4de512fd889fbf15f112cc3704f188b1d4f%40%3Cdev.calcite.apache.org%3E]
> email thread.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
