This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 175fcfd2106 [SPARK-43414][TESTS] Fix flakiness in Kafka RDD suites due
to port binding configuration issue
175fcfd2106 is described below
commit 175fcfd2106652baff53fdfb1be84638a4f6a9c9
Author: Josh Rosen <[email protected]>
AuthorDate: Mon May 8 16:06:15 2023 -0700
[SPARK-43414][TESTS] Fix flakiness in Kafka RDD suites due to port binding
configuration issue
### What changes were proposed in this pull request?
This PR addresses a test flakiness issue in Kafka connector RDD suites
https://github.com/apache/spark/pull/34089#pullrequestreview-872739182
(Spark 3.4.0) upgraded Spark to Kafka 3.1.0, which requires a different
configuration key for configuring the broker listening port. That PR updated
the `KafkaTestUtils.scala` used in SQL tests, but there's a near-duplicate of
that code in a different `KafkaTestUtils.scala` used by RDD API suites which
wasn't updated. As a result, the RDD suites began using Kafka's default port
9092 and this results in flakiness as [...]
This PR fixes that by simply copying the updated configuration from the SQL
copy of `KafkaTestUtils.scala`.
### Why are the changes needed?
Fix test flakiness due to port conflicts.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Ran 20 concurrent copies of
`org.apache.spark.streaming.kafka010.JavaKafkaRDDSuite` in my CI environment
and confirmed that this PR's changes resolve the test flakiness.
Closes #41095 from
JoshRosen/update-kafka-test-utils-to-fix-port-binding-flakiness.
Authored-by: Josh Rosen <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git
a/connector/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
b/connector/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
index 91fecacb6e7..1a7a6e9676a 100644
---
a/connector/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
+++
b/connector/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/KafkaTestUtils.scala
@@ -240,9 +240,7 @@ private[kafka010] class KafkaTestUtils extends Logging {
private def brokerConfiguration: Properties = {
val props = new Properties()
props.put("broker.id", "0")
- props.put("host.name", localHostNameForURI)
- props.put("advertised.host.name", localHostNameForURI)
- props.put("port", brokerPort.toString)
+ props.put("listeners", s"PLAINTEXT://$localHostNameForURI:$brokerPort")
props.put("log.dir", brokerLogDir)
props.put("zookeeper.connect", zkAddress)
props.put("zookeeper.connection.timeout.ms", "60000")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
