nyingping created SPARK-38214:
---------------------------------
Summary: No need to filter data when the sliding window length is
not redundant
Key: SPARK-38214
URL: https://issues.apache.org/jira/browse/SPARK-38214
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 3.2.1
Reporter: nyingping
At present, the sliding window adopts the form of expand + filter, but in some
cases, filter is not necessary.
Filter is required if the sliding window is irregular. When the window length
is divided by the slide length the result is an integer (I believe this is also
the case for most work scenarios in practice for sliding window), there is no
need to filter, which can save calculation resources and improve performance.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
