This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new ba773fe9a286 [SPARK-47185][SS][TESTS] Increase timeout between actions
in KafkaContinuousSourceSuite
ba773fe9a286 is described below
commit ba773fe9a28640af6d2ddebd8104b05e28778f58
Author: Hyukjin Kwon <gurwls...@apache.org>
AuthorDate: Tue Feb 27 07:36:08 2024 -0800
[SPARK-47185][SS][TESTS] Increase timeout between actions in
KafkaContinuousSourceSuite
### What changes were proposed in this pull request?
This PR proposes to increase the timeout between between actions in
`KafkaContinuousSourceSuite`.
### Why are the changes needed?
In Mac OS build, those tests fail indeterministically, see
- https://github.com/apache/spark/actions/runs/8054862135/job/22000404856
- https://github.com/apache/spark/actions/runs/8040413156/job/21958488693
- https://github.com/apache/spark/actions/runs/8032862212/job/21942732320
- https://github.com/apache/spark/actions/runs/8024427919/job/21937366481
`KafkaContinuousSourceSuite` is specifically slow in Mac OS. Kafka
producers send the messages correctly, but the consumers can't get the messages
for some reasons. You can't get the offsets for long time. This is not an issue
in micro batch but I fail to identify the difference.
I just decided to increase the timeout between actions for now. This is
more just a workaround.
### Does this PR introduce _any_ user-facing change?
No, test-only.
### How was this patch tested?
Manually tested in my Mac.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #45283 from HyukjinKwon/SPARK-47185.
Authored-by: Hyukjin Kwon <gurwls...@apache.org>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
.../org/apache/spark/sql/kafka010/KafkaContinuousSourceSuite.scala | 3 +++
1 file changed, 3 insertions(+)
diff --git
a/connector/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaContinuousSourceSuite.scala
b/connector/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaContinuousSourceSuite.scala
index e42662c7a62b..fa1db6bfaccc 100644
---
a/connector/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaContinuousSourceSuite.scala
+++
b/connector/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaContinuousSourceSuite.scala
@@ -18,6 +18,7 @@
package org.apache.spark.sql.kafka010
import org.apache.kafka.clients.producer.ProducerRecord
+import org.scalatest.time.SpanSugar._
import org.apache.spark.sql.Dataset
import org.apache.spark.sql.execution.datasources.v2.ContinuousScanExec
@@ -28,6 +29,8 @@ import org.apache.spark.sql.streaming.Trigger
class KafkaContinuousSourceSuite extends KafkaSourceSuiteBase with
KafkaContinuousTest {
import testImplicits._
+ override val streamingTimeout = 60.seconds
+
test("read Kafka transactional messages: read_committed") {
val table = "kafka_continuous_source_test"
withTable(table) {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
