[
https://issues.apache.org/jira/browse/FLINK-41506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18146354#comment-18146354
]
Stefan Wang commented on FLINK-41506:
-------------------------------------
I've opened https://github.com/apache/flink/pull/29961 with a fix. Could a
committer assign this ticket to me?
> Batch sliding group window planned as a one-phase SortWindowAggregate reads
> the pane time at the input row's time index
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-41506
> URL: https://issues.apache.org/jira/browse/FLINK-41506
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner, Table SQL / Runtime
> Affects Versions: 2.3.0
> Reporter: Yaoxuan Wu
> Priority: Major
> Labels: correctness, pull-request-available
>
> {code:python}
> from pyflink.table import TableEnvironment, EnvironmentSettings, DataTypes
> from pyflink.table import expressions as T
> from pyflink.table.window import Slide
> t_env = TableEnvironment.create(EnvironmentSettings.in_batch_mode())
> cfg = t_env.get_config()
> cfg.set('parallelism.default', '1')
> cfg.set('table.local-time-zone', 'UTC')
> cfg.set('table.optimizer.agg-phase-strategy', 'ONE_PHASE')
> src = t_env.from_elements([(1, 7000)], DataTypes.ROW([DataTypes.FIELD('k',
> DataTypes.INT()),
> DataTypes.FIELD('v',
> DataTypes.BIGINT())]))
> t = src.add_columns(T.call_sql("TIMESTAMP '2001-09-09
> 01:46:40.000'").alias('rt')) # (k, v, rt)
> r =
> (t.window(Slide.over(T.lit(12).seconds).every(T.lit(3).seconds).on(T.col('rt')).alias('w'))
> .group_by(T.col('w'), T.col('k'))
> .select(T.col('k'), T.col('w').start, T.col('w').end, T.col('v').sum))
> for row in sorted(r.execute().collect(), key=str):
> print(row)
> # expected: 4 windows starting 2001-09-09 01:46:30, :33, :36, :39
> # actual: 4 windows starting 1969-12-31 23:59:57, 1970-01-01 00:00:00,
> 00:00:03, 00:00:06
> {code}
> Same on 2.0.1, 2.2.1, 2.3.0 and a 2.4-SNAPSHOT build. The row is placed at
> 7000 ms, the value of the {{SUM(v)}} buffer. With the input columns in the
> order {{(rt, k, v)}} it is placed at 1 ms (the key). With {{(k, rt, v)}} the
> result is right. With a fourth column after {{rt}}, the job fails with
> {{java.lang.IndexOutOfBoundsException}} in {{WindowsGrouping.getTimeValue}}.
> Tumbling windows and the two-phase plan are not affected.
> The one-phase plan is not only chosen by ONE_PHASE: with the default
> strategy, the same happens when the window input comes from two aggregates,
> e.g.
> {code:python}
> a2 = (s2.group_by(T.col('a')).select(T.col('a'), T.lit(1).count.alias('c'))
> .group_by(T.col('c')).select(T.col('c').alias('k'), (T.col('c').sum *
> 7000).alias('v'))
> .add_columns(T.call_sql("TIMESTAMP '2001-09-09
> 01:46:40.000'").alias('rt')))
> # s2 = (7, 100), (8, 200): one row (k=1, v=14000); Slide 12 s every 3 s ->
> windows around 1970-01-01 00:00:14
> {code}
> Cause (planner, master):
> - {{BatchPhysicalWindowAggregateRule.transformTimeSlidingWindow}}, one-phase
> branch: a sliding window with pane optimization ({{enableAssignPane = true}})
> becomes a {{BatchPhysicalSortWindowAggregate}} with {{isMerge = false}} and
> {{inputTimeFieldIndex}} = the time attribute's index in the input row.
> - {{SortWindowCodeGenerator}}: {{enablePreAcc = choosePreAcc || isMerge}},
> and {{choosePreAcc}} is true when {{enableAssignPane}} is set. Input rows are
> then pre-accumulated into pane elements laid out as {{[group keys,
> assignedTs$, agg buffers]}}
> ({{WindowCodeGenerator.getWindowsGroupingElementInfo}}).
> - {{WindowCodeGenerator.genWindowAggCodes}} passes the same input-row index
> to {{genCreateWindowsGroupingCode}}, so {{HeapWindowsGrouping}} reads a pane
> element's time ({{WindowsGrouping.getTimeValue}}) at that index. For a
> pre-accumulated element the index should be the number of grouping keys (the
> {{assignedTs$}} field). The two-phase merge branch already passes
> {{newInputTimeFieldIndexFromLocal}}, which points there.
> The result is only right when the input's time index happens to equal the
> number of grouping keys. This is separate from FLINK-41287, which is about
> the same operator. Its reproducer has no key and the time field at index 0,
> so it is not affected by this issue.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)