[
https://issues.apache.org/jira/browse/SPARK-34738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17342018#comment-17342018
]
Attila Zsolt Piros commented on SPARK-34738:
--------------------------------------------
[~shaneknapp] I have merged my commit this means "PVs with local storage" test
is skipped.
Could you please check whether the tests on Minikube with docker driver are
passing now successfully on jenkins?
> Upgrade Minikube and kubernetes and move to docker 'virtualization' layer
> -------------------------------------------------------------------------
>
> Key: SPARK-34738
> URL: https://issues.apache.org/jira/browse/SPARK-34738
> Project: Spark
> Issue Type: Task
> Components: jenkins, Kubernetes
> Affects Versions: 3.2.0
> Reporter: Attila Zsolt Piros
> Assignee: Shane Knapp
> Priority: Major
> Attachments: integration-tests.log
>
>
> [~shaneknapp] as we discussed [on the mailing
> list|http://apache-spark-developers-list.1001551.n3.nabble.com/minikube-and-kubernetes-cluster-versions-for-integration-testing-td30856.html]
> Minikube can be upgraded to the latest (v1.18.1) and kubernetes version
> should be v1.17.3 (`minikube config set kubernetes-version v1.17.3`).
> [Here|https://github.com/apache/spark/pull/31829] is my PR which uses a new
> method to configure the kubernetes client. Thanks in advance to use it for
> testing on the Jenkins after the Minikube version is updated.
>
> Added by Shane:
> we also need to move from the kvm2 virtualization layer to docker. docker is
> a recommended driver w/the latest versions of minikube, and this will allow
> devs to more easily recreate the minikube/k8s env on their local workstations
> and run the integration tests in an identical environment as jenkins.
> the TL;DR is that upgrading to docker works, except that the PV integration
> tests are failing due to a couple of possible reasons:
> 1) the 'spark-kubernetes-driver' isn't properly being loaded
> (https://issues.apache.org/jira/browse/SPARK-34738?focusedCommentId=17312517&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17312517)
> 2) during the PV test run, the error message 'Given path
> (/opt/spark/pv-tests/tmp4595937990978494271.txt) does not exist' shows up in
> the logs. however, the mk cluster *does* mount successfully to the local
> bare-metal filesystem *and* if i 'minikube ssh' in to it, i can see the mount
> and read/write successfully to it
> (https://issues.apache.org/jira/browse/SPARK-34738?focusedCommentId=17312548&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17312548)
> i could really use some help, and if it's useful, i can create some local
> accounts manually and allow ssh access for a couple of people to assist me.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
