[jira] [Commented] (SPARK-32617) Upgrade kubernetes client version to support latest minikube version.
[
https://issues.apache.org/jira/browse/SPARK-32617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17291326#comment-17291326
]
Dongjoon Hyun commented on SPARK-32617:
---
BTW, the issue title is wrong. K8s IT works with all the latest Minikube
including v1.16 and v1.17 (as of today).
> Upgrade kubernetes client version to support latest minikube version.
> -
>
> Key: SPARK-32617
> URL: https://issues.apache.org/jira/browse/SPARK-32617
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 3.1.0
>Reporter: Prashant Sharma
>Priority: Major
>
> Following error comes, when the k8s integration tests are run against the
> minikube cluster with version 1.2.1
> {code:java}
> Run starting. Expected test count is: 18
> KubernetesSuite:
> org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite *** ABORTED ***
> io.fabric8.kubernetes.client.KubernetesClientException: An error has
> occurred.
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:196)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:62)
> at io.fabric8.kubernetes.client.BaseClient.(BaseClient.java:51)
> at
> io.fabric8.kubernetes.client.DefaultKubernetesClient.(DefaultKubernetesClient.java:105)
> at
> org.apache.spark.deploy.k8s.integrationtest.backend.minikube.Minikube$.getKubernetesClient(Minikube.scala:81)
> at
> org.apache.spark.deploy.k8s.integrationtest.backend.minikube.MinikubeTestBackend$.initialize(MinikubeTestBackend.scala:33)
> at
> org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.beforeAll(KubernetesSuite.scala:131)
> at
> org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
> ...
> Cause: java.nio.file.NoSuchFileException: /root/.minikube/apiserver.crt
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
> at java.nio.file.Files.newByteChannel(Files.java:361)
> at java.nio.file.Files.newByteChannel(Files.java:407)
> at java.nio.file.Files.readAllBytes(Files.java:3152)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.getInputStreamFromDataOrFile(CertUtils.java:72)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:242)
> at
> io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:128)
> ...
> Run completed in 1 second, 821 milliseconds.
> Total number of tests run: 0
> Suites: completed 1, aborted 1
> Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
> *** 1 SUITE ABORTED ***
> [INFO]
>
> [INFO] Reactor Summary for Spark Project Parent POM 3.1.0-SNAPSHOT:
> [INFO]
> [INFO] Spark Project Parent POM ... SUCCESS [ 4.454
> s]
> [INFO] Spark Project Tags . SUCCESS [ 4.768
> s]
> [INFO] Spark Project Local DB . SUCCESS [ 2.961
> s]
> [INFO] Spark Project Networking ... SUCCESS [ 4.258
> s]
> [INFO] Spark Project Shuffle Streaming Service SUCCESS [ 5.703
> s]
> [INFO] Spark Project Unsafe ... SUCCESS [ 3.239
> s]
> [INFO] Spark Project Launcher . SUCCESS [ 3.224
> s]
> [INFO] Spark Project Core . SUCCESS [02:25
> min]
> [INFO] Spark Project Kubernetes Integration Tests . FAILURE [ 17.244
> s]
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 03:12 min
> [INFO] Finished at: 2020-08-11T06:26:15-05:00
> [INFO]
>
> [ERROR] Failed to execute goal
> org.scalatest:scalatest-maven-plugin:2.0.0:test (integration-test) on project
> spark-kubernetes-integration-tests_2.12: There are test failures -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors
[jira] [Commented] (SPARK-32617) Upgrade kubernetes client version to support latest minikube version.
[
https://issues.apache.org/jira/browse/SPARK-32617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17291325#comment-17291325
]
Dongjoon Hyun commented on SPARK-32617:
---
The commit is reverted because it breaks IT when `minikube kubectl config view`
returns multiple configs.
I tried to do follow-up first with `kubectl config view --minify`. However, it
turns out that `minikube` doesn't support `minikube kubectl config view
--minify`.
> Upgrade kubernetes client version to support latest minikube version.
> -
>
> Key: SPARK-32617
> URL: https://issues.apache.org/jira/browse/SPARK-32617
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 3.1.0
>Reporter: Prashant Sharma
>Priority: Major
>
> Following error comes, when the k8s integration tests are run against the
> minikube cluster with version 1.2.1
> {code:java}
> Run starting. Expected test count is: 18
> KubernetesSuite:
> org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite *** ABORTED ***
> io.fabric8.kubernetes.client.KubernetesClientException: An error has
> occurred.
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:196)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:62)
> at io.fabric8.kubernetes.client.BaseClient.(BaseClient.java:51)
> at
> io.fabric8.kubernetes.client.DefaultKubernetesClient.(DefaultKubernetesClient.java:105)
> at
> org.apache.spark.deploy.k8s.integrationtest.backend.minikube.Minikube$.getKubernetesClient(Minikube.scala:81)
> at
> org.apache.spark.deploy.k8s.integrationtest.backend.minikube.MinikubeTestBackend$.initialize(MinikubeTestBackend.scala:33)
> at
> org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.beforeAll(KubernetesSuite.scala:131)
> at
> org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
> ...
> Cause: java.nio.file.NoSuchFileException: /root/.minikube/apiserver.crt
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
> at java.nio.file.Files.newByteChannel(Files.java:361)
> at java.nio.file.Files.newByteChannel(Files.java:407)
> at java.nio.file.Files.readAllBytes(Files.java:3152)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.getInputStreamFromDataOrFile(CertUtils.java:72)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:242)
> at
> io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:128)
> ...
> Run completed in 1 second, 821 milliseconds.
> Total number of tests run: 0
> Suites: completed 1, aborted 1
> Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
> *** 1 SUITE ABORTED ***
> [INFO]
>
> [INFO] Reactor Summary for Spark Project Parent POM 3.1.0-SNAPSHOT:
> [INFO]
> [INFO] Spark Project Parent POM ... SUCCESS [ 4.454
> s]
> [INFO] Spark Project Tags . SUCCESS [ 4.768
> s]
> [INFO] Spark Project Local DB . SUCCESS [ 2.961
> s]
> [INFO] Spark Project Networking ... SUCCESS [ 4.258
> s]
> [INFO] Spark Project Shuffle Streaming Service SUCCESS [ 5.703
> s]
> [INFO] Spark Project Unsafe ... SUCCESS [ 3.239
> s]
> [INFO] Spark Project Launcher . SUCCESS [ 3.224
> s]
> [INFO] Spark Project Core . SUCCESS [02:25
> min]
> [INFO] Spark Project Kubernetes Integration Tests . FAILURE [ 17.244
> s]
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 03:12 min
> [INFO] Finished at: 2020-08-11T06:26:15-05:00
> [INFO]
>
> [ERROR] Failed to execute goal
> org.scalatest:scalatest-maven-plugin:2.0.0:test (integration-test) on project
> spark-kubernetes-integration-tests_2.12: There are test failures -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run
[jira] [Commented] (SPARK-32617) Upgrade kubernetes client version to support latest minikube version.
[
https://issues.apache.org/jira/browse/SPARK-32617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17291321#comment-17291321
]
Apache Spark commented on SPARK-32617:
--
User 'dongjoon-hyun' has created a pull request for this issue:
https://github.com/apache/spark/pull/31655
> Upgrade kubernetes client version to support latest minikube version.
> -
>
> Key: SPARK-32617
> URL: https://issues.apache.org/jira/browse/SPARK-32617
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 3.1.0
>Reporter: Prashant Sharma
>Assignee: Attila Zsolt Piros
>Priority: Major
> Fix For: 3.2.0
>
>
> Following error comes, when the k8s integration tests are run against the
> minikube cluster with version 1.2.1
> {code:java}
> Run starting. Expected test count is: 18
> KubernetesSuite:
> org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite *** ABORTED ***
> io.fabric8.kubernetes.client.KubernetesClientException: An error has
> occurred.
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:196)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:62)
> at io.fabric8.kubernetes.client.BaseClient.(BaseClient.java:51)
> at
> io.fabric8.kubernetes.client.DefaultKubernetesClient.(DefaultKubernetesClient.java:105)
> at
> org.apache.spark.deploy.k8s.integrationtest.backend.minikube.Minikube$.getKubernetesClient(Minikube.scala:81)
> at
> org.apache.spark.deploy.k8s.integrationtest.backend.minikube.MinikubeTestBackend$.initialize(MinikubeTestBackend.scala:33)
> at
> org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.beforeAll(KubernetesSuite.scala:131)
> at
> org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
> ...
> Cause: java.nio.file.NoSuchFileException: /root/.minikube/apiserver.crt
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
> at java.nio.file.Files.newByteChannel(Files.java:361)
> at java.nio.file.Files.newByteChannel(Files.java:407)
> at java.nio.file.Files.readAllBytes(Files.java:3152)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.getInputStreamFromDataOrFile(CertUtils.java:72)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:242)
> at
> io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:128)
> ...
> Run completed in 1 second, 821 milliseconds.
> Total number of tests run: 0
> Suites: completed 1, aborted 1
> Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
> *** 1 SUITE ABORTED ***
> [INFO]
>
> [INFO] Reactor Summary for Spark Project Parent POM 3.1.0-SNAPSHOT:
> [INFO]
> [INFO] Spark Project Parent POM ... SUCCESS [ 4.454
> s]
> [INFO] Spark Project Tags . SUCCESS [ 4.768
> s]
> [INFO] Spark Project Local DB . SUCCESS [ 2.961
> s]
> [INFO] Spark Project Networking ... SUCCESS [ 4.258
> s]
> [INFO] Spark Project Shuffle Streaming Service SUCCESS [ 5.703
> s]
> [INFO] Spark Project Unsafe ... SUCCESS [ 3.239
> s]
> [INFO] Spark Project Launcher . SUCCESS [ 3.224
> s]
> [INFO] Spark Project Core . SUCCESS [02:25
> min]
> [INFO] Spark Project Kubernetes Integration Tests . FAILURE [ 17.244
> s]
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 03:12 min
> [INFO] Finished at: 2020-08-11T06:26:15-05:00
> [INFO]
>
> [ERROR] Failed to execute goal
> org.scalatest:scalatest-maven-plugin:2.0.0:test (integration-test) on project
> spark-kubernetes-integration-tests_2.12: There are test failures -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug
[jira] [Commented] (SPARK-32617) Upgrade kubernetes client version to support latest minikube version.
[
https://issues.apache.org/jira/browse/SPARK-32617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248523#comment-17248523
]
Apache Spark commented on SPARK-32617:
--
User 'attilapiros' has created a pull request for this issue:
https://github.com/apache/spark/pull/30751
> Upgrade kubernetes client version to support latest minikube version.
> -
>
> Key: SPARK-32617
> URL: https://issues.apache.org/jira/browse/SPARK-32617
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 3.1.0
>Reporter: Prashant Sharma
>Priority: Major
>
> Following error comes, when the k8s integration tests are run against the
> minikube cluster with version 1.2.1
> {code:java}
> Run starting. Expected test count is: 18
> KubernetesSuite:
> org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite *** ABORTED ***
> io.fabric8.kubernetes.client.KubernetesClientException: An error has
> occurred.
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:196)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:62)
> at io.fabric8.kubernetes.client.BaseClient.(BaseClient.java:51)
> at
> io.fabric8.kubernetes.client.DefaultKubernetesClient.(DefaultKubernetesClient.java:105)
> at
> org.apache.spark.deploy.k8s.integrationtest.backend.minikube.Minikube$.getKubernetesClient(Minikube.scala:81)
> at
> org.apache.spark.deploy.k8s.integrationtest.backend.minikube.MinikubeTestBackend$.initialize(MinikubeTestBackend.scala:33)
> at
> org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.beforeAll(KubernetesSuite.scala:131)
> at
> org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
> ...
> Cause: java.nio.file.NoSuchFileException: /root/.minikube/apiserver.crt
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
> at java.nio.file.Files.newByteChannel(Files.java:361)
> at java.nio.file.Files.newByteChannel(Files.java:407)
> at java.nio.file.Files.readAllBytes(Files.java:3152)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.getInputStreamFromDataOrFile(CertUtils.java:72)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:242)
> at
> io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:128)
> ...
> Run completed in 1 second, 821 milliseconds.
> Total number of tests run: 0
> Suites: completed 1, aborted 1
> Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
> *** 1 SUITE ABORTED ***
> [INFO]
>
> [INFO] Reactor Summary for Spark Project Parent POM 3.1.0-SNAPSHOT:
> [INFO]
> [INFO] Spark Project Parent POM ... SUCCESS [ 4.454
> s]
> [INFO] Spark Project Tags . SUCCESS [ 4.768
> s]
> [INFO] Spark Project Local DB . SUCCESS [ 2.961
> s]
> [INFO] Spark Project Networking ... SUCCESS [ 4.258
> s]
> [INFO] Spark Project Shuffle Streaming Service SUCCESS [ 5.703
> s]
> [INFO] Spark Project Unsafe ... SUCCESS [ 3.239
> s]
> [INFO] Spark Project Launcher . SUCCESS [ 3.224
> s]
> [INFO] Spark Project Core . SUCCESS [02:25
> min]
> [INFO] Spark Project Kubernetes Integration Tests . FAILURE [ 17.244
> s]
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 03:12 min
> [INFO] Finished at: 2020-08-11T06:26:15-05:00
> [INFO]
>
> [ERROR] Failed to execute goal
> org.scalatest:scalatest-maven-plugin:2.0.0:test (integration-test) on project
> spark-kubernetes-integration-tests_2.12: There are test failures -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and
[jira] [Commented] (SPARK-32617) Upgrade kubernetes client version to support latest minikube version.
[
https://issues.apache.org/jira/browse/SPARK-32617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248522#comment-17248522
]
Apache Spark commented on SPARK-32617:
--
User 'attilapiros' has created a pull request for this issue:
https://github.com/apache/spark/pull/30751
> Upgrade kubernetes client version to support latest minikube version.
> -
>
> Key: SPARK-32617
> URL: https://issues.apache.org/jira/browse/SPARK-32617
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 3.1.0
>Reporter: Prashant Sharma
>Priority: Major
>
> Following error comes, when the k8s integration tests are run against the
> minikube cluster with version 1.2.1
> {code:java}
> Run starting. Expected test count is: 18
> KubernetesSuite:
> org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite *** ABORTED ***
> io.fabric8.kubernetes.client.KubernetesClientException: An error has
> occurred.
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:196)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:62)
> at io.fabric8.kubernetes.client.BaseClient.(BaseClient.java:51)
> at
> io.fabric8.kubernetes.client.DefaultKubernetesClient.(DefaultKubernetesClient.java:105)
> at
> org.apache.spark.deploy.k8s.integrationtest.backend.minikube.Minikube$.getKubernetesClient(Minikube.scala:81)
> at
> org.apache.spark.deploy.k8s.integrationtest.backend.minikube.MinikubeTestBackend$.initialize(MinikubeTestBackend.scala:33)
> at
> org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.beforeAll(KubernetesSuite.scala:131)
> at
> org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
> ...
> Cause: java.nio.file.NoSuchFileException: /root/.minikube/apiserver.crt
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
> at java.nio.file.Files.newByteChannel(Files.java:361)
> at java.nio.file.Files.newByteChannel(Files.java:407)
> at java.nio.file.Files.readAllBytes(Files.java:3152)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.getInputStreamFromDataOrFile(CertUtils.java:72)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:242)
> at
> io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:128)
> ...
> Run completed in 1 second, 821 milliseconds.
> Total number of tests run: 0
> Suites: completed 1, aborted 1
> Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
> *** 1 SUITE ABORTED ***
> [INFO]
>
> [INFO] Reactor Summary for Spark Project Parent POM 3.1.0-SNAPSHOT:
> [INFO]
> [INFO] Spark Project Parent POM ... SUCCESS [ 4.454
> s]
> [INFO] Spark Project Tags . SUCCESS [ 4.768
> s]
> [INFO] Spark Project Local DB . SUCCESS [ 2.961
> s]
> [INFO] Spark Project Networking ... SUCCESS [ 4.258
> s]
> [INFO] Spark Project Shuffle Streaming Service SUCCESS [ 5.703
> s]
> [INFO] Spark Project Unsafe ... SUCCESS [ 3.239
> s]
> [INFO] Spark Project Launcher . SUCCESS [ 3.224
> s]
> [INFO] Spark Project Core . SUCCESS [02:25
> min]
> [INFO] Spark Project Kubernetes Integration Tests . FAILURE [ 17.244
> s]
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 03:12 min
> [INFO] Finished at: 2020-08-11T06:26:15-05:00
> [INFO]
>
> [ERROR] Failed to execute goal
> org.scalatest:scalatest-maven-plugin:2.0.0:test (integration-test) on project
> spark-kubernetes-integration-tests_2.12: There are test failures -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and
[jira] [Commented] (SPARK-32617) Upgrade kubernetes client version to support latest minikube version.
[
https://issues.apache.org/jira/browse/SPARK-32617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248168#comment-17248168
]
Attila Zsolt Piros commented on SPARK-32617:
I will have PR for this soon.
> Upgrade kubernetes client version to support latest minikube version.
> -
>
> Key: SPARK-32617
> URL: https://issues.apache.org/jira/browse/SPARK-32617
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 3.1.0
>Reporter: Prashant Sharma
>Priority: Major
>
> Following error comes, when the k8s integration tests are run against the
> minikube cluster with version 1.2.1
> {code:java}
> Run starting. Expected test count is: 18
> KubernetesSuite:
> org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite *** ABORTED ***
> io.fabric8.kubernetes.client.KubernetesClientException: An error has
> occurred.
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:196)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:62)
> at io.fabric8.kubernetes.client.BaseClient.(BaseClient.java:51)
> at
> io.fabric8.kubernetes.client.DefaultKubernetesClient.(DefaultKubernetesClient.java:105)
> at
> org.apache.spark.deploy.k8s.integrationtest.backend.minikube.Minikube$.getKubernetesClient(Minikube.scala:81)
> at
> org.apache.spark.deploy.k8s.integrationtest.backend.minikube.MinikubeTestBackend$.initialize(MinikubeTestBackend.scala:33)
> at
> org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.beforeAll(KubernetesSuite.scala:131)
> at
> org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
> ...
> Cause: java.nio.file.NoSuchFileException: /root/.minikube/apiserver.crt
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
> at java.nio.file.Files.newByteChannel(Files.java:361)
> at java.nio.file.Files.newByteChannel(Files.java:407)
> at java.nio.file.Files.readAllBytes(Files.java:3152)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.getInputStreamFromDataOrFile(CertUtils.java:72)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:242)
> at
> io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:128)
> ...
> Run completed in 1 second, 821 milliseconds.
> Total number of tests run: 0
> Suites: completed 1, aborted 1
> Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
> *** 1 SUITE ABORTED ***
> [INFO]
>
> [INFO] Reactor Summary for Spark Project Parent POM 3.1.0-SNAPSHOT:
> [INFO]
> [INFO] Spark Project Parent POM ... SUCCESS [ 4.454
> s]
> [INFO] Spark Project Tags . SUCCESS [ 4.768
> s]
> [INFO] Spark Project Local DB . SUCCESS [ 2.961
> s]
> [INFO] Spark Project Networking ... SUCCESS [ 4.258
> s]
> [INFO] Spark Project Shuffle Streaming Service SUCCESS [ 5.703
> s]
> [INFO] Spark Project Unsafe ... SUCCESS [ 3.239
> s]
> [INFO] Spark Project Launcher . SUCCESS [ 3.224
> s]
> [INFO] Spark Project Core . SUCCESS [02:25
> min]
> [INFO] Spark Project Kubernetes Integration Tests . FAILURE [ 17.244
> s]
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 03:12 min
> [INFO] Finished at: 2020-08-11T06:26:15-05:00
> [INFO]
>
> [ERROR] Failed to execute goal
> org.scalatest:scalatest-maven-plugin:2.0.0:test (integration-test) on project
> spark-kubernetes-integration-tests_2.12: There are test failures -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR]
[jira] [Commented] (SPARK-32617) Upgrade kubernetes client version to support latest minikube version.
[
https://issues.apache.org/jira/browse/SPARK-32617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17182945#comment-17182945
]
Daniel Moore commented on SPARK-32617:
--
Is there a minikube support matrix?
> Upgrade kubernetes client version to support latest minikube version.
> -
>
> Key: SPARK-32617
> URL: https://issues.apache.org/jira/browse/SPARK-32617
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
>Affects Versions: 3.1.0
>Reporter: Prashant Sharma
>Priority: Major
>
> Following error comes, when the k8s integration tests are run against the
> minikube cluster with version 1.2.1
> {code:java}
> Run starting. Expected test count is: 18
> KubernetesSuite:
> org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite *** ABORTED ***
> io.fabric8.kubernetes.client.KubernetesClientException: An error has
> occurred.
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
> at
> io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:196)
> at
> io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:62)
> at io.fabric8.kubernetes.client.BaseClient.(BaseClient.java:51)
> at
> io.fabric8.kubernetes.client.DefaultKubernetesClient.(DefaultKubernetesClient.java:105)
> at
> org.apache.spark.deploy.k8s.integrationtest.backend.minikube.Minikube$.getKubernetesClient(Minikube.scala:81)
> at
> org.apache.spark.deploy.k8s.integrationtest.backend.minikube.MinikubeTestBackend$.initialize(MinikubeTestBackend.scala:33)
> at
> org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.beforeAll(KubernetesSuite.scala:131)
> at
> org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
> ...
> Cause: java.nio.file.NoSuchFileException: /root/.minikube/apiserver.crt
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
> at java.nio.file.Files.newByteChannel(Files.java:361)
> at java.nio.file.Files.newByteChannel(Files.java:407)
> at java.nio.file.Files.readAllBytes(Files.java:3152)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.getInputStreamFromDataOrFile(CertUtils.java:72)
> at
> io.fabric8.kubernetes.client.internal.CertUtils.createKeyStore(CertUtils.java:242)
> at
> io.fabric8.kubernetes.client.internal.SSLUtils.keyManagers(SSLUtils.java:128)
> ...
> Run completed in 1 second, 821 milliseconds.
> Total number of tests run: 0
> Suites: completed 1, aborted 1
> Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
> *** 1 SUITE ABORTED ***
> [INFO]
>
> [INFO] Reactor Summary for Spark Project Parent POM 3.1.0-SNAPSHOT:
> [INFO]
> [INFO] Spark Project Parent POM ... SUCCESS [ 4.454
> s]
> [INFO] Spark Project Tags . SUCCESS [ 4.768
> s]
> [INFO] Spark Project Local DB . SUCCESS [ 2.961
> s]
> [INFO] Spark Project Networking ... SUCCESS [ 4.258
> s]
> [INFO] Spark Project Shuffle Streaming Service SUCCESS [ 5.703
> s]
> [INFO] Spark Project Unsafe ... SUCCESS [ 3.239
> s]
> [INFO] Spark Project Launcher . SUCCESS [ 3.224
> s]
> [INFO] Spark Project Core . SUCCESS [02:25
> min]
> [INFO] Spark Project Kubernetes Integration Tests . FAILURE [ 17.244
> s]
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 03:12 min
> [INFO] Finished at: 2020-08-11T06:26:15-05:00
> [INFO]
>
> [ERROR] Failed to execute goal
> org.scalatest:scalatest-maven-plugin:2.0.0:test (integration-test) on project
> spark-kubernetes-integration-tests_2.12: There are test failures -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help
