[
https://issues.apache.org/jira/browse/SPARK-34738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shane Knapp updated SPARK-34738:
--------------------------------
Description:
[~shaneknapp] as we discussed [on the mailing
list|http://apache-spark-developers-list.1001551.n3.nabble.com/minikube-and-kubernetes-cluster-versions-for-integration-testing-td30856.html]
Minikube can be upgraded to the latest (v1.18.1) and kubernetes version should
be v1.17.3 (`minikube config set kubernetes-version v1.17.3`).
[Here|https://github.com/apache/spark/pull/31829] is my PR which uses a new
method to configure the kubernetes client. Thanks in advance to use it for
testing on the Jenkins after the Minikube version is updated.
Added by Shane:
we also need to move from the kvm2 virtualization layer to docker. docker is a
recommended driver w/the latest versions of minikube, and this will allow devs
to more easily recreate the minikube/k8s env on their local workstations and
run the integration tests in an identical environment as jenkins.
the TL;DR is that upgrading to docker works, except that the PV integration
tests are failing due to a couple of possible reasons:
1) the 'spark-kubernetes-driver' isn't properly being loaded
(https://issues.apache.org/jira/browse/SPARK-34738?focusedCommentId=17312517&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17312517)
2) /
was:
[~shaneknapp] as we discussed [on the mailing
list|http://apache-spark-developers-list.1001551.n3.nabble.com/minikube-and-kubernetes-cluster-versions-for-integration-testing-td30856.html]
Minikube can be upgraded to the latest (v1.18.1) and kubernetes version should
be v1.17.3 (`minikube config set kubernetes-version v1.17.3`).
[Here|https://github.com/apache/spark/pull/31829] is my PR which uses a new
method to configure the kubernetes client. Thanks in advance to use it for
testing on the Jenkins after the Minikube version is updated.
Added by Shane:
we also need to move from the kvm2 virtualization layer to docker. docker is a
recommended driver w/the latest versions of minikube, and this will allow devs
to more easily recreate the minikube/k8s env on their local workstations and
run the integration tests in an identical environment as jenkins.
the TL;DR is that the PV integration tests are failing due to
> Upgrade Minikube and kubernetes and move to docker 'virtualization' layer
> -------------------------------------------------------------------------
>
> Key: SPARK-34738
> URL: https://issues.apache.org/jira/browse/SPARK-34738
> Project: Spark
> Issue Type: Task
> Components: jenkins, Kubernetes
> Affects Versions: 3.2.0
> Reporter: Attila Zsolt Piros
> Assignee: Shane Knapp
> Priority: Major
> Attachments: integration-tests.log
>
>
> [~shaneknapp] as we discussed [on the mailing
> list|http://apache-spark-developers-list.1001551.n3.nabble.com/minikube-and-kubernetes-cluster-versions-for-integration-testing-td30856.html]
> Minikube can be upgraded to the latest (v1.18.1) and kubernetes version
> should be v1.17.3 (`minikube config set kubernetes-version v1.17.3`).
> [Here|https://github.com/apache/spark/pull/31829] is my PR which uses a new
> method to configure the kubernetes client. Thanks in advance to use it for
> testing on the Jenkins after the Minikube version is updated.
>
> Added by Shane:
> we also need to move from the kvm2 virtualization layer to docker. docker is
> a recommended driver w/the latest versions of minikube, and this will allow
> devs to more easily recreate the minikube/k8s env on their local workstations
> and run the integration tests in an identical environment as jenkins.
> the TL;DR is that upgrading to docker works, except that the PV integration
> tests are failing due to a couple of possible reasons:
> 1) the 'spark-kubernetes-driver' isn't properly being loaded
> (https://issues.apache.org/jira/browse/SPARK-34738?focusedCommentId=17312517&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17312517)
> 2) /
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
