[
https://issues.apache.org/jira/browse/SPARK-21153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shixiong Zhu resolved SPARK-21153.
----------------------------------
Resolution: Fixed
Fix Version/s: 2.3.0
> Time windowing for tumbling windows can use a project instead of expand +
> filter
> --------------------------------------------------------------------------------
>
> Key: SPARK-21153
> URL: https://issues.apache.org/jira/browse/SPARK-21153
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.1.1
> Reporter: Burak Yavuz
> Assignee: Burak Yavuz
> Fix For: 2.3.0
>
>
> Time windowing in Spark currently performs an Expand + Filter, because there
> is no way to guarantee the amount of windows a timestamp will fall in, in the
> general case. However, for tumbling windows, a record is guaranteed to fall
> into a single bucket. In this case, doubling the number of records with
> Expand is wasteful, and can be improved by using a simple Projection instead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
