Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/7057#discussion_r34168240
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/Window.scala ---
@@ -19,17 +19,47 @@ package org.apache.spark.sql.execution
import java.util
-import org.apache.spark.rdd.RDD
+import org.apache.spark.annotation.DeveloperApi
import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.plans.physical.{AllTuples,
ClusteredDistribution, Distribution, Partitioning}
+import org.apache.spark.sql.catalyst.plans.physical._
+import org.apache.spark.sql.types.IntegerType
+import org.apache.spark.rdd.RDD
import org.apache.spark.util.collection.CompactBuffer
+import scala.collection.mutable
/**
* :: DeveloperApi ::
- * For every row, evaluates `windowExpression` containing Window Functions
and attaches
- * the results with other regular expressions (presented by `projectList`).
- * Evert operator handles a single Window Specification, `windowSpec`.
+ * This class calculates and outputs (windowed) aggregates over the rows
in a single (sorted)
+ * partition. The aggregates are calculated for each row in the group.
Special processing
+ * instructions, frames, are used to calculate these aggregates. Frames
are processed in the order
+ * specified in the window specification (the ORDER BY ... clause). There
are four different frame
+ * types:
+ * - Entire partition: The frame is the entire partition, i.e.
+ * UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING. For this case, window
function will take all
+ * rows as inputs and be evaluated once.
+ * - Growing frame: We only add new rows into the frame, i.e. UNBOUNDED
PRECEDING AND ....
+ * Every time we move to a new row to process, we add some rows to the
frame. We do not remove
+ * rows from this frame.
+ * - Shrinking frame: We only remove rows from the frame, i.e. ... AND
UNBOUNDED FOLLOWING.
+ * Every time we move to a new row to process, we remove some rows from
the frame. We do not add
+ * rows to this frame. The frame will originally contain all rows of the
partition.
--- End diff --
I made a mistake at here. When we have `n FOLLOWING AND UNBOUNDED
FOLLOWING`, the frame will contain all rows of the partition at the beginning.
So, we can remove `The frame will originally contain all rows of the
partition.`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
