This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new e1dbeb4de862 [SPARK-49871][CORE][TESTS] Fix
`(Ssl)CoarseGrainedExecutorBackendSuite` to reduce test resource and increase
timeout
e1dbeb4de862 is described below
commit e1dbeb4de86214c4feeb00e1b76a7caa343109dd
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Oct 4 12:55:34 2024 +0900
[SPARK-49871][CORE][TESTS] Fix `(Ssl)CoarseGrainedExecutorBackendSuite` to
reduce test resource and increase timeout
### What changes were proposed in this pull request?
This PR aims to fix `(Ssl)CoarseGrainedExecutorBackendSuite` to reduce test
resource and increase timeout in order to reduce the flakiness in the resource
hungry GitHub Action environment.
### Why are the changes needed?
The test case of `(Ssl)CoarseGrainedExecutorBackendSuite` injects a faulty
plugin to check the executor restart. Sometime, it fails due to the timeout
during restarting the executors.
- https://github.com/apache/spark/actions/runs/11168954174/job/31048566907
```
- SPARK-40320 Executor should exit when initialization failed for fatal
error *** FAILED ***
The code passed to eventually never returned normally. Attempted 187
times over 15.001016415999999 seconds.
Last failure message: 1 was not greater than or equal to 2.
(CoarseGrainedExecutorBackendSuite.scala:602)
```
### Does this PR introduce _any_ user-facing change?
No, this is a test-only change.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #48345 from dongjoon-hyun/SPARK-49871.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../apache/spark/executor/CoarseGrainedExecutorBackendSuite.scala | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git
a/core/src/test/scala/org/apache/spark/executor/CoarseGrainedExecutorBackendSuite.scala
b/core/src/test/scala/org/apache/spark/executor/CoarseGrainedExecutorBackendSuite.scala
index 7bd84c810c42..8b98df103c01 100644
---
a/core/src/test/scala/org/apache/spark/executor/CoarseGrainedExecutorBackendSuite.scala
+++
b/core/src/test/scala/org/apache/spark/executor/CoarseGrainedExecutorBackendSuite.scala
@@ -37,7 +37,7 @@ import org.scalatestplus.mockito.MockitoSugar
import org.apache.spark._
import org.apache.spark.TestUtils._
import org.apache.spark.api.plugin.{DriverPlugin, ExecutorPlugin,
PluginContext, SparkPlugin}
-import org.apache.spark.internal.config.PLUGINS
+import org.apache.spark.internal.config.{EXECUTOR_MEMORY, PLUGINS}
import org.apache.spark.resource._
import org.apache.spark.resource.ResourceUtils._
import org.apache.spark.resource.TestResourceIDs._
@@ -581,7 +581,8 @@ class CoarseGrainedExecutorBackendSuite extends
SparkFunSuite
*/
test("SPARK-40320 Executor should exit when initialization failed for fatal
error") {
val conf = createSparkConf()
- .setMaster("local-cluster[1, 1, 1024]")
+ .setMaster("local-cluster[1, 1, 512]")
+ .set(EXECUTOR_MEMORY.key, "512m")
.set(PLUGINS, Seq(classOf[TestFatalErrorPlugin].getName))
.setAppName("test")
sc = new SparkContext(conf)
@@ -599,7 +600,7 @@ class CoarseGrainedExecutorBackendSuite extends
SparkFunSuite
}
try {
sc.addSparkListener(listener)
- eventually(timeout(15.seconds)) {
+ eventually(timeout(30.seconds)) {
assert(executorAddCounter.get() >= 2)
assert(executorRemovedCounter.get() >= 2)
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
