jerrypeng commented on code in PR #58673:
URL: https://github.com/apache/spark/pull/58673#discussion_r3982018023
##########
docs/streaming/real-time-mode.md:
##########
@@ -34,10 +34,11 @@ personalization.
Real-time Mode in Apache Spark supports **stateless queries** -- projections,
filters and other
map-like operations, unions, and stream-static joins -- and, starting in Spark
4.3.0, a first set of
-**stateful queries**: streaming **deduplication** (`dropDuplicates`) and
streaming **aggregations**
-(`groupBy(...).agg(...)`), and the JVM (Scala/Java) **`transformWithState`**
operator. These stateful
-operations require a shuffle, which Real-time Mode runs as a *pipelined
shuffle* so that records
-still stream through without waiting for a batch boundary; see
+**stateful queries**: streaming **deduplication** (`dropDuplicates`, plus
+`dropDuplicatesWithinWatermark` starting in Spark 5.0.0), streaming
**aggregations**
Review Comment:
will fix
##########
docs/streaming/real-time-mode.md:
##########
@@ -602,8 +605,15 @@ spark
</div>
-This keeps every distinct key it has seen in the state store.
`dropDuplicatesWithinWatermark`, which
-bounds how long keys are retained, is not yet supported in Real-time Mode.
+This example uses `dropDuplicates`, which keeps every distinct key it has seen
in the state store.
+To bound state, define an event-time watermark and use
`dropDuplicatesWithinWatermark`. For example,
+Scala and Java use
+`withWatermark("eventTime", "10
minutes").dropDuplicatesWithinWatermark("id")`; in PySpark, pass the
Review Comment:
will fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
