[
https://issues.apache.org/jira/browse/DRILL-1908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14261362#comment-14261362
]
Deneche A. Hakim commented on DRILL-1908:
-----------------------------------------
Here is a general overview of how the operator (window frame function) should
work:
when computing the window function value for the "current row" we need to
compute it's window frame. A window frame is one or more rows from the current
row's partition, that are aggregated when computing the window. No matter what,
window functions for a partition never contain rows from other partitions.
We have 2 possibilities:
* a) all rows of a partition share the same window frame, and thus the same
window function value. This window frame aggregates the window function over
all rows of the partition
* b) each row has a unique window frame that contain one or more rows from the
partition
the new operator works as follow:
* the operator cannot process a partition of rows (compute the window function
value for any row of the partition) until the end of the partition is met:
either a new partition is encountered or no more batches are available.
* if some rows of the incoming batch can't be processed yet, because the batch
contains at least an incomplete partition, we save the batch internally and
fetch the next batch
Let's assume we have the following 3 batches of data:
|b0|b1|b2|
|p0 p1|p1|p2 p3 p4|
batch b0 contains 2 partitions p0 and p1
batch b1 has partition p1
batch b2 has partitions p2, p3 and p4
This is how the operator will handle the batches, in the following "we" refers
to the window operator:
* when next() is called
** we call next(incoming) and receive b0,
we can process p0 but we can't process p1 yet
we save b0
** we call next(incoming) and receive b1
we still can't process p1
we save b1 along b0
** we call next(incoming) and receive b2
we can process p1, p2 and p3
we process b0 and remove it from the saved batches
we return the container downstream
* when next() is called
** we call next(incoming) and receive NONE
we can now process p4
we process b1 and remove it from the saved batches
we return the container downstream
* when next() is called
** we process b2 and remove it from the saved batches
we return the container downstream
* when next() is called
** we return NONE
> new window function implementation
> ----------------------------------
>
> Key: DRILL-1908
> URL: https://issues.apache.org/jira/browse/DRILL-1908
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Operators
> Reporter: Deneche A. Hakim
> Assignee: Deneche A. Hakim
> Priority: Critical
>
> In order to fix DRILL-1487 a complete rewrite of the
> StreamingWindowFrameRecordBatch may be needed. The purpose of this issue is
> to report my progress and share my thoughts with the community in order to
> get a proper implementation
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
