This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 124b5af [SPARK-34732][K8S][TESTS] Fix IndexOutOfBoundsException in
logForFailedTest when driver is not started
124b5af is described below
commit 124b5af11452aec7075e2c65200b9cdfa065f9d0
Author: “attilapiros” <[email protected]>
AuthorDate: Sat Mar 13 15:28:02 2021 -0800
[SPARK-34732][K8S][TESTS] Fix IndexOutOfBoundsException in logForFailedTest
when driver is not started
### What changes were proposed in this pull request?
Fixing `IndexOutOfBoundsException` in `logForFailedTest` method when driver
is not started.
### Why are the changes needed?
Before this PR when the driver is not started an
`IndexOutOfBoundsException` as the first item is tried to be accessed from an
empty list:
```
- PVs with local storage *** FAILED ***
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:659)
at java.util.ArrayList.get(ArrayList.java:435)
at
org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.logForFailedTest(KubernetesSuite.scala:83)
at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:181)
at
org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:188)
at
org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:200)
at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
at
org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:200)
at
org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:182)
at
org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:61)
...
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Running integration tests.
After this changes the above error become:
```
- PVs with local storage *** FAILED ***
java.io.IOException: No such file or directory
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createTempFile(File.java:2026)
at
org.apache.spark.deploy.k8s.integrationtest.Utils$.createTempFile(Utils.scala:103)
at
org.apache.spark.deploy.k8s.integrationtest.PVTestsSuite.$anonfun$$init$$1(PVTestsSuite.scala:135)
at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at
org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:190)
...
```
Closes #31824 from attilapiros/SPARK-34732.
Authored-by: “attilapiros” <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../deploy/k8s/integrationtest/KubernetesSuite.scala | 19 +++++++++++--------
1 file changed, 11 insertions(+), 8 deletions(-)
diff --git
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
index f4ca5da..174030d 100644
---
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
+++
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
@@ -74,19 +74,22 @@ class KubernetesSuite extends SparkFunSuite
protected override def logForFailedTest(): Unit = {
logInfo("\n\n===== EXTRA LOGS FOR THE FAILED TEST\n")
- val driverPod = kubernetesTestComponents.kubernetesClient
+ val driverPodOption = kubernetesTestComponents.kubernetesClient
.pods()
.withLabel("spark-app-locator", appLocator)
.withLabel("spark-role", "driver")
.list()
.getItems
- .get(0)
- logInfo("BEGIN driver POD log\n" +
- kubernetesTestComponents.kubernetesClient
- .pods()
- .withName(driverPod.getMetadata.getName)
- .getLog)
- logInfo("END driver POD log")
+ .asScala
+ .headOption
+ driverPodOption.foreach { driverPod =>
+ logInfo("BEGIN driver POD log\n" +
+ kubernetesTestComponents.kubernetesClient
+ .pods()
+ .withName(driverPod.getMetadata.getName)
+ .getLog)
+ logInfo("END driver POD log")
+ }
kubernetesTestComponents.kubernetesClient
.pods()
.withLabel("spark-app-locator", appLocator)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
