Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9636#discussion_r44612127
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala
---
@@ -426,6 +428,48 @@ class JavaPairDStream[K, V](val dstream: DStream[(K,
V)])(
)
}
+ /**
+ * :: Experimental ::
+ * Return a new [[JavaDStream]] of data generated by combining the
key-value data in `this` stream
+ * with a continuously updated per-key state. The user-provided state
tracking function is
+ * applied on each keyed data item along with its corresponding state.
The function can choose to
+ * update/remove the state and return a transformed data, which forms the
+ * [[JavaTrackStateDStream]].
+ *
+ * The specifications of this transformation is made through the
+ * [[org.apache.spark.streaming.StateSpec StateSpec]] class. Besides the
tracking function, there
+ * are a number of optional parameters - initial state data, number of
partitions, timeouts, etc.
+ * See the [[org.apache.spark.streaming.StateSpec StateSpec]] for more
details.
+ *
+ * Example of using `trackStateByKey`:
+ * {{{
+ * // A tracking function that maintains an integer state and return a
String
+ * Function2<Optional<Integer>, State<Integer>, Optional<String>>
trackStateFunc =
+ * new Function2<Optional<Integer>, State<Integer>,
Optional<String>>() {
+ *
+ * @Override
+ * public Optional<String> call(Optional<Integer> one,
State<Integer> state) {
+ * // Check if state exists, accordingly update/remove state
and return transformed data
+ * }
+ * };
+ *
+ * JavaTrackStateDStream[Integer, Integer, Integer, String]
trackStateDStream =
--- End diff --
Make sure that the code would compile
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
