[
https://issues.apache.org/jira/browse/SPARK-8640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Herman van Hovell updated SPARK-8640:
-------------------------------------
Attachment: perf_test_window_collapse.scala
Some benchmarking result for this ticket. The benchmark shows the result of
Seperate Windows (one for each frame) versus Collapsed Windows (all frames
together) for a number of partition sizes.
The performance improvements are small, but memory usage should be better. Also
not that the current benchmark only uses two frames, gains will be greater if
there are more frames. The benchmark is shown in the table below:
{noformat}
+------+----------------+-----------------+------+
| Size|Separate Windows|Collapsed Windows|Diff %|
+------+----------------+-----------------+------+
| 2| 46512| 46505| 1.000|
| 8| 46251| 46680| 1.009|
| 32| 46647| 45982| 0.986|
| 128| 44594| 44594| 1.000|
| 512| 44959| 43688| 0.972|
| 1024| 43641| 42489| 0.974|
| 4096| 42502| 41350| 0.973|
| 16192| 41788| 41196| 0.986|
| 64768| 40956| 39099| 0.955|
|259072| 38945| 37459| 0.962|
+------+----------------+-----------------+------+
Separate Windows/Collapsed Windows is the time in Milliseconds it took to
complete the entire opertation.
{noformat}
> Window Function Multiple Frame Processing in Single Processing Step
> -------------------------------------------------------------------
>
> Key: SPARK-8640
> URL: https://issues.apache.org/jira/browse/SPARK-8640
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 1.5.0
> Reporter: Herman van Hovell
> Attachments: perf_test_window_collapse.scala
>
>
> The new Window operator is capable of processing different frames for the
> same Window. In order to enable this the Catalyst Analyzer needs to be
> modified.
> PR will follow as soon as SPARK-8638 gets in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
