Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44233204
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/PairDStreamFunctions.scala
---
@@ -351,6 +351,50 @@ class PairDStreamFunctions[K, V](self: DStream[(K, V)])
}
/**
+ * :: Experimental ::
+ * Return a new DStream of data generated by combining the key-value
data in `this` stream
+ * with a continuously updated per-key state. The user-provided state
tracking function is
+ * applied on each keyed data item along with its corresponding state.
The function can choose to
+ * update/remove the state and return a transformed data, which forms the
+ * [[org.apache.spark.streaming.dstream.EmittedRecordsDStream]].
+ *
+ * The specifications of this transformation is made through the
+ * [[org.apache.spark.streaming.StateSpec StateSpec]] class. Besides the
tracking function, there
+ * are a number of optional parameters - initial state data, number of
partitions, timeouts, etc.
+ * See the [[org.apache.spark.streaming.StateSpec StateSpec spec docs]]
for more details.
+ *
+ * Scala example of using `trackStateByKey`:
+ * {{{
+ * def trackingFunction(key: String, data: Option[Int], wrappedState:
State[Int]): Option[Int] = {
+ * // Check state exists, accordingly update/remove state and
return data to emit
+ * }
+ *
+ * val spec = StateSpec(trackingFunction).numPartitions(10)
+ *
+ * val emittedRecordDStream =
keyValueDStream.trackStateByKey[StateType, EmittedDataType](spec)
+ *
+ * }}}
+ *
+ * Java example of using `trackStateByKey`:
+ * {{{
+ * TODO(@zsxwing)
+ * }}}
--- End diff --
Aah, yes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
