Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/7057#discussion_r33640715
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/Window.scala ---
@@ -19,17 +19,39 @@ package org.apache.spark.sql.execution
import java.util
-import org.apache.spark.rdd.RDD
+import org.apache.spark.annotation.DeveloperApi
import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.plans.physical.{AllTuples,
ClusteredDistribution, Distribution, Partitioning}
+import org.apache.spark.sql.catalyst.plans.physical._
+import org.apache.spark.sql.types.IntegerType
+import org.apache.spark.rdd.RDD
import org.apache.spark.util.collection.CompactBuffer
+import scala.collection.mutable
/**
* :: DeveloperApi ::
- * For every row, evaluates `windowExpression` containing Window Functions
and attaches
- * the results with other regular expressions (presented by `projectList`).
- * Evert operator handles a single Window Specification, `windowSpec`.
+ * This class calculates and outputs (windowed) aggregates over the rows
in a single sorted group.
+ * The aggregates are calculated for each row in the group. An aggregate
can take a few forms:
+ * - Global: The aggregate is calculated for the entire group. Every row
has the same value.
+ * - Rows: The aggregate is calculated based on a subset of the window,
and is unique for each
+ * row and depends on the position of the given row within the window. The
group must be sorted
+ * for this to produce sensible output. Examples are moving averages,
running sums and row
+ * numbers.
+ * - Range: The aggregate is calculated based on a subset of the window,
and is unique for each
+ * value of the order by clause and depends on its ordering. The group
must be sorted for this to
+ * produce sensible output.
+ * - Shifted: The aggregate is a displaced value relative to the position
of the given row.
+ * Examples are Lead and Lag.
--- End diff --
Seems we are mixing the concepts of (1) how a frame updates; and (2) how
the frame boundary is determined together at here. Let me summarize them
separately.
For frame boundary, we have two types, row and range.
For how frame updates, we have four types of frame:
* Entire partition: The frame is the entire partition, i.e. `UNBOUNDED
PRECEDING AND UNBOUNDED FOLLOWING`. For this case, window function will take
all rows as inputs and be evaluated once.
* Growing frame: We only add new rows into the frame, i.e. `UNBOUNDED
PRECEDING AND ...`. Every time we move to a new row to process, we add some
rows to the frame. We do not remove rows from this frame.
* Shrinking frame: We only remove rows from the frame, i.e. `... AND
UNBOUNDED FOLLOWING`. Every time we move to a new row to process, we remove
some rows from the frame. We do not add rows to this frame. The frame will
originally contain all rows of the partition.
* Moving frame: Every time we move to a new row to process, we remove some
rows from the frame and we add some rows to the frame. Examples are `1
PRECEDING AND CURRENT ROW` and `1 FOLLOWING AND 2 FOLLOWING`.
I feel summarizing these two concepts separately can help people understand
them. What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
