This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new a78579ea915 [SPARK-41972][TESTS] Fix a flaky test in
StreamingQueryStatusListenerSuite
a78579ea915 is described below
commit a78579ea91562eb62fdd5f567a8201c78a219669
Author: Gengliang Wang <[email protected]>
AuthorDate: Tue Jan 10 23:13:08 2023 -0800
[SPARK-41972][TESTS] Fix a flaky test in StreamingQueryStatusListenerSuite
### What changes were proposed in this pull request?
Fix flaky test in StreamingQueryStatusListenerSuite.
Example failed test
https://github.com/gengliangwang/spark/actions/runs/3880260857/jobs/6629409770:
```
StreamingQueryStatusListenerSuite.test small retained queries
org.scalatest.exceptions.TestFailedDueToTimeoutException: The code passed
to eventually never returned normally. Attempted 638 times over 10.006178314
seconds. Last failure message: Set(301dad06-8fd7-4766-8d61-a58724d2c966,
fab95baf-25ad-47dc-89fd-ff49bf1a4c4e) did not equal
Set(60d3b5b0-7397-4212-8a18-ff67a17f362a, fab95baf-25ad-47dc-89fd-ff49bf1a4c4e).
```
The retention queries after eviction(by the ending timestamp) are different
from what is expected. I suspect that some of the input queries has the same
ending timestamp.
Another proof: with RocksDB as backend, the test case always pass since
writing to RocksDB takes more time.
So, I suggest adding `Thread.sleep` between the termination of these
queries to fix the flakiness.
### Why are the changes needed?
Fix flaky test
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
GA test
Closes #39494 from gengliangwang/fixFlaky.
Authored-by: Gengliang Wang <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../spark/sql/streaming/ui/StreamingQueryStatusListenerSuite.scala | 6 ++++++
1 file changed, 6 insertions(+)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatusListenerSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatusListenerSuite.scala
index ad4c1ff28d1..58e04eb285a 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatusListenerSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatusListenerSuite.scala
@@ -181,9 +181,15 @@ class StreamingQueryStatusListenerSuite extends StreamTest
{
val terminateEvent1 = new StreamingQueryListener.QueryTerminatedEvent(id1,
runId1, None)
listener.onQueryTerminated(terminateEvent1)
checkInactiveQueryStatus(1, Seq(id1))
+ // SPARK-41972: having a short sleep here to make sure the end time of
query 2 is larger than
+ // query 1.
+ Thread.sleep(20)
val terminateEvent2 = new StreamingQueryListener.QueryTerminatedEvent(id2,
runId2, None)
listener.onQueryTerminated(terminateEvent2)
checkInactiveQueryStatus(2, Seq(id1, id2))
+ // SPARK-41972: having a short sleep here to make sure the end time of
query 3 is larger than
+ // query 2.
+ Thread.sleep(20)
val terminateEvent3 = new StreamingQueryListener.QueryTerminatedEvent(id3,
runId3, None)
listener.onQueryTerminated(terminateEvent3)
checkInactiveQueryStatus(2, Seq(id2, id3))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
