tdas commented on a change in pull request #33336:
URL: https://github.com/apache/spark/pull/33336#discussion_r672689363
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/streaming/FlatMapGroupsWithStateSuite.scala
##########
@@ -1413,6 +1419,46 @@ class FlatMapGroupsWithStateSuite extends
StateStoreMetricsTest {
)
}
+ Seq("NoTimeout", "EventTime", "ProcessingTime").foreach { timeout =>
+ test(s"flatMapGroupsWithState - initial state - batch mode - timeout
${timeout}") {
+ // We will test them on different shuffle partition configuration to
make sure the
+ // grouping by key will still work. On higher number of shuffle
partitions its possible
+ // that all keys end up on different partitions.
+ val initialState = Seq(
+ (s"keyInStateAndData-1-$timeout", new RunningCount(1)),
+ ("keyInStateAndData-2", new RunningCount(2)),
+ ("keyNoUpdate", new RunningCount(2)), // state.update will not be
called
+ ("keyOnlyInState-1", new RunningCount(1))
+ ).toDS().groupByKey(x => x._1).mapValues(_._2)
+
+ val inputData = Seq(
+ ("keyOnlyInData"), ("keyInStateAndData-2")
+ )
+ val timeoutMode = timeout match {
+ case "NoTimeout" =>
Review comment:
Why are you doing this?? Why not create seq of objects directly
`Seq(NoTimeout(), EventTimeTimeout(), ProcessingTimeTimeout()).foreach {
timeout =>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
