Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/17179#discussion_r105821496
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -249,6 +250,43 @@ class KeyValueGroupedDataset[K, V] private[sql](
dataAttributes,
OutputMode.Update,
isMapGroupsWithState = true,
+ KeyedStateTimeout.none,
+ child = logicalPlan))
+ }
+
+ /**
+ * ::Experimental::
+ * (Scala-specific)
+ * Applies the given function to each group of data, while maintaining a
user-defined per-group
+ * state. The result Dataset will represent the objects returned by the
function.
+ * For a static batch Dataset, the function will be invoked once per
group. For a streaming
+ * Dataset, the function will be invoked for each group repeatedly in
every trigger, and
+ * updates to each group's state will be saved across invocations.
+ * See [[org.apache.spark.sql.streaming.KeyedState]] for more details.
+ *
+ * @tparam S The type of the user-defined state. Must be encodable to
Spark SQL types.
+ * @tparam U The type of the output objects. Must be encodable to Spark
SQL types.
+ * @param func Function to be called on every group.
+ * @param timeout Timeout information for groups that do not receive
data for a while
+ *
+ * See [[Encoder]] for more details on what types are encodable to Spark
SQL.
+ * @since 2.2.0
+ */
+ @Experimental
+ @InterfaceStability.Evolving
+ def mapGroupsWithState[S: Encoder, U: Encoder](
+ func: (K, Iterator[V], KeyedState[S]) => U,
+ timeout: KeyedStateTimeout): Dataset[U] = {
--- End diff --
`timeoutType`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
