[
https://issues.apache.org/jira/browse/FLINK-22781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17369433#comment-17369433
]
JING ZHANG edited comment on FLINK-22781 at 6/25/21, 12:09 PM:
---------------------------------------------------------------
[~jark] Thanks a lot for reply.
I would like to summarize the current solution (without introducing extra
mechanism) to handle late UB message for window aggregate upon
changelog(contains update message).
User needs to set all the following 3 parameters:
(1) enable late fire by setting
{code:java}
table.exec.emit.late-fire.enabled : true{code}
(2) set per record emit behavior for late records by setting
{code:java}
table.exec.emit.late-fire.delay : 0 s
{code}
(3) keep window state for extra time after window is fired by setting
{code:java}
table.exec.emit.allow-latenss : 1 h
// 或者
table.exec.state.ttl: 1h{code}
The solution has two disadvantages:
(1) User may not realize that UB message maybe dropped as late event, so they
will not set related parameters.
(2) When use look for a solution to solve dropped UB message problem, current
solution is a bit of unconvinient for them because they need to set all the 3
parameters.
I agree with you provides recommended configurations in docs could help.
Besides, could we simplify the 3 parameters a little. for example, user only
need set allow-lateness, framework could atom set
`table.exec.emit.late-fire.enabled` to true and
set`table.exec.emit.late-fire.delay ` to 0s.
What do you think?
was (Author: qingru zhang):
[~jark] Thanks a lot for reply.
I would like to conclude the current solution (without introducing extra
mechanism) to take late UB message into consideration for window aggregate upon
changlog(contains update message).
User needs to set all the following 3 parameters:
(1) enable late fire by setting
{code:java}
table.exec.emit.late-fire.enabled : true{code}
(2) set per record emit behavior for late records by setting
{code:java}
table.exec.emit.late-fire.delay : 0 s
{code}
(3) keep window state for extra time after window is fired by setting
{code:java}
table.exec.emit.allow-latenss : 1 h
// 或者
table.exec.state.ttl: 1h{code}
The solution has two disadvantages:
(1) User may not realize that UB message maybe dropped as late event, so they
will not set related parameters.
(2) When use look for a solution to solve dropped UB message problem, current
solution is a bit of unconvinient for them because they need to set all the 3
parameters.
I agree with you provides recommended configurations in docs could help.
Besides, could we simplify the 3 parameters a little. for example, user only
need set allow-lateness, framework could atom set
`table.exec.emit.late-fire.enabled` to true and
set`table.exec.emit.late-fire.delay ` to 0s.
What do you think?
> Incorrect result for group window aggregate when mini-batch is enabled
> ----------------------------------------------------------------------
>
> Key: FLINK-22781
> URL: https://issues.apache.org/jira/browse/FLINK-22781
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
> Affects Versions: 1.14.0
> Reporter: godfrey he
> Assignee: JING ZHANG
> Priority: Critical
> Labels: pull-request-available
> Fix For: 1.14.0
>
>
> We can reproduce this issue through adding the following code to
> {{GroupWindowITCase#testWindowAggregateOnUpsertSource}} method:
> {code:java}
> tEnv.getConfig.getConfiguration.setBoolean(
> ExecutionConfigOptions.TABLE_EXEC_MINIBATCH_ENABLED, true)
> tEnv.getConfig.getConfiguration.set(
> ExecutionConfigOptions.TABLE_EXEC_MINIBATCH_ALLOW_LATENCY,
> Duration.ofSeconds(1))
> tEnv.getConfig.getConfiguration.setLong(
> ExecutionConfigOptions.TABLE_EXEC_MINIBATCH_SIZE, 10L)
> {code}
> The reason is the group window without any data (the data may be retracted)
> should not send any record.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
