This is an automated email from the ASF dual-hosted git repository.
yao pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new d76223c20e13 [SPARK-44843][TESTS] Double streamingTimeout for
StateStoreMetricsTest to make RocksDBStateStore related streaming tests reliable
d76223c20e13 is described below
commit d76223c20e13acb0a790efff5ac54a4cf58293cf
Author: Kent Yao <[email protected]>
AuthorDate: Sat Nov 4 01:16:32 2023 +0800
[SPARK-44843][TESTS] Double streamingTimeout for StateStoreMetricsTest to
make RocksDBStateStore related streaming tests reliable
### What changes were proposed in this pull request?
This PR increases streamingTimeout and the check interval for
StateStoreMetricsTest to make RocksDBStateStore-related streaming tests
reliable, hopefully.
### Why are the changes needed?
```
SPARK-35896: metrics in StateOperatorProgress are output correctly
(RocksDBStateStore with changelog checkpointing) *** FAILED *** (1 minute)
[info] Timed out waiting for stream: The code passed to failAfter did not
complete within 60 seconds.
[info] java.base/java.lang.Thread.getStackTrace(Thread.java:1619)
```
The probability of these tests failing is close to 100%, which seriously
affects the UX of making PRs for the contributors.
https://github.com/yaooqinn/spark/actions/runs/6744173341/job/18333952141
### Does this PR introduce _any_ user-facing change?
no, test only
### How was this patch tested?
this can be verified by `sql - slow test` job in CI
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #43647 from yaooqinn/SPARK-44843.
Authored-by: Kent Yao <[email protected]>
Signed-off-by: Kent Yao <[email protected]>
(cherry picked from commit afdce266f0ffeb068d47eca2f2af1bcba66b0e95)
Signed-off-by: Kent Yao <[email protected]>
---
.../org/apache/spark/sql/streaming/StateStoreMetricsTest.scala | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StateStoreMetricsTest.scala
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StateStoreMetricsTest.scala
index 0abc79a5515c..b7b0be5d5324 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StateStoreMetricsTest.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StateStoreMetricsTest.scala
@@ -17,6 +17,8 @@
package org.apache.spark.sql.streaming
+import org.scalatest.time.SpanSugar._
+
import org.apache.spark.sql.execution.streaming.StreamExecution
trait StateStoreMetricsTest extends StreamTest {
@@ -24,6 +26,8 @@ trait StateStoreMetricsTest extends StreamTest {
private var lastCheckedRecentProgressIndex = -1
private var lastQuery: StreamExecution = null
+ override val streamingTimeout = 120.seconds
+
override def beforeEach(): Unit = {
super.beforeEach()
lastCheckedRecentProgressIndex = -1
@@ -106,7 +110,7 @@ trait StateStoreMetricsTest extends StreamTest {
AssertOnQuery(s"Check operator progress metrics: operatorName =
$operatorName, " +
s"numShufflePartitions = $numShufflePartitions, " +
s"numStateStoreInstances = $numStateStoreInstances") { q =>
- eventually(timeout(streamingTimeout)) {
+ eventually(timeout(streamingTimeout), interval(200.milliseconds)) {
val (progressesSinceLastCheck, lastCheckedProgressIndex,
numStateOperators) =
retrieveProgressesSinceLastCheck(q)
assert(operatorIndex < numStateOperators, s"Invalid operator Index:
$operatorIndex")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
