Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r44098629
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/PairDStreamFunctions.scala
---
@@ -350,6 +349,18 @@ class PairDStreamFunctions[K, V](self: DStream[(K, V)])
)
}
+ /** TODO: Add scala docs */
+ def trackStateByKey[S: ClassTag, T: ClassTag](
+ spec: TrackStateSpec[K, V, S, T]): EmittedRecordsDStream[K, V, S, T] =
{
+ new EmittedRecordsDStreamImpl[K, V, S, T](
+ new TrackStateDStream[K, V, S, T](
+ self,
+ spec.asInstanceOf[TrackStateSpecImpl[K, V, S, T]]
--- End diff --
The DataFrameReader does not seem to have any getters. So I am not sure how
to derive inspiration from it.
From offline discussion, the confusion was that the function is not part of
the API of `TrackStateSpec` class. And that is because there the function is
specified through `TrackStateSpec.apply(func)`. I had designed it that way
because the function is the only spec that must be provided, and the rest is
optional. So having the `TrackStateSpec.apply()` take the function ensures that
its concise (`TrackStateSpec(func)`) and dumb proof (the user has to specify
it).
But if this is confusing, then there are two options.
1. Make `function()` a method with which the user can specify the function.
So the user could write either `TrackStateSpec.function(func)` or
`TrackStateSpec(func)` and the `function` is visible in the API + docs of the
TrackStateSpec class.
2. Just leave it as is and make the docs of `TrackStateSpec` class (and
also other docs) clearly show the usage using the `TrackStateSpec` object.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
