[spark] branch master updated (d5563f3 -> 4e1e33b)
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from d5563f3 [SPARK-36956][MLLIB] model prediction in .mllib avoid conversion to breeze vector add 4e1e33b [SPARK-37011][PYTHON][BUILD] update flake8 on jenkins workers No new revisions were added by this update. Summary of changes: .../files/python_environments/py36.txt | 49 -- .../files/python_environments/spark-py36-spec.txt | 18 dev/lint-python| 3 +- 3 files changed, 10 insertions(+), 60 deletions(-) delete mode 100644 dev/ansible-for-test-node/roles/jenkins-worker/files/python_environments/py36.txt - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark-website] branch asf-site updated: updating local k8s/minikube testing instructions
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new d60611b updating local k8s/minikube testing instructions d60611b is described below commit d60611b5c464db0fb99ca072a9d8b55e824ca7c2 Author: shane knapp AuthorDate: Mon Aug 2 10:15:46 2021 -0700 updating local k8s/minikube testing instructions a small update to the k8s/minikube integration test instructions Author: shane knapp Closes #351 from shaneknapp/updating-k8s-docs. --- developer-tools.md| 9 ++--- site/developer-tools.html | 9 ++--- 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/developer-tools.md b/developer-tools.md index 9551533..bf3ee6e 100644 --- a/developer-tools.md +++ b/developer-tools.md @@ -169,8 +169,8 @@ Please check other available options via `python/run-tests[-with-coverage] --hel If you have made changes to the K8S bindings in Apache Spark, it would behoove you to test locally before submitting a PR. This is relatively simple to do, but it will require a local (to you) installation of [minikube](https://kubernetes.io/docs/setup/minikube/). Due to how minikube interacts with the host system, please be sure to set things up as follows: -- minikube version v1.7.3 (or greater) -- You must use a VM driver! Running minikube with the `--vm-driver=none` option requires that the user launching minikube/k8s have root access. Our Jenkins workers use the [kvm2](https://minikube.sigs.k8s.io/docs/drivers/kvm2/) drivers. More details [here](https://minikube.sigs.k8s.io/docs/drivers/). +- minikube version v1.18.1 (or greater) +- You must use a VM driver! Running minikube with the `--vm-driver=none` option requires that the user launching minikube/k8s have root access, which could impact how the tests are run. Our Jenkins workers use the default [docker](https://minikube.sigs.k8s.io/docs/drivers/docker/) drivers. More details [here](https://minikube.sigs.k8s.io/docs/drivers/). - kubernetes version v1.17.3 (can be set by executing `minikube config set kubernetes-version v1.17.3`) - the current kubernetes context must be minikube's default context (called 'minikube'). This can be selected by `minikube kubectl -- config use-context minikube`. This is only needed when after minikube is started another kubernetes context is selected. @@ -196,8 +196,11 @@ PVC_TMP_DIR=$(mktemp -d) export PVC_TESTS_HOST_PATH=$PVC_TMP_DIR export PVC_TESTS_VM_PATH=$PVC_TMP_DIR -minikube --vm-driver= start --memory 6000 --cpus 8 minikube config set kubernetes-version v1.17.3 +minikube --vm-driver= start --memory 6000 --cpus 8 + +# for macos only (see https://github.com/apache/spark/pull/32793): +# minikube ssh "sudo useradd spark -u 185 -g 0 -m -s /bin/bash" minikube mount ${PVC_TESTS_HOST_PATH}:${PVC_TESTS_VM_PATH} --9p-version=9p2000.L --gid=0 --uid=185 &; MOUNT_PID=$! diff --git a/site/developer-tools.html b/site/developer-tools.html index 5558945..81d64c7 100644 --- a/site/developer-tools.html +++ b/site/developer-tools.html @@ -348,8 +348,8 @@ Generating HTML files for PySpark coverage under /.../spark/python/test_coverage If you have made changes to the K8S bindings in Apache Spark, it would behoove you to test locally before submitting a PR. This is relatively simple to do, but it will require a local (to you) installation of https://kubernetes.io/docs/setup/minikube/";>minikube. Due to how minikube interacts with the host system, please be sure to set things up as follows: - minikube version v1.7.3 (or greater) - You must use a VM driver! Running minikube with the --vm-driver=none option requires that the user launching minikube/k8s have root access. Our Jenkins workers use the https://minikube.sigs.k8s.io/docs/drivers/kvm2/";>kvm2 drivers. More details https://minikube.sigs.k8s.io/docs/drivers/";>here. + minikube version v1.18.1 (or greater) + You must use a VM driver! Running minikube with the --vm-driver=none option requires that the user launching minikube/k8s have root access, which could impact how the tests are run. Our Jenkins workers use the default https://minikube.sigs.k8s.io/docs/drivers/docker/";>docker drivers. More details https://minikube.sigs.k8s.io/docs/drivers/";>here. kubernetes version v1.17.3 (can be set by executing minikube config set kubernetes-version v1.17.3) the current kubernetes context must be minikube’s default context (called ‘minikube’). This can be selected by minikube kubectl -- config use-context minikube. This is only needed when after minikube is started another kubernetes context is selected. @@ -374,8 +374,11 @@ export TARBALL_TO_TEST=($(pwd)/spark-*${DATE}-${REVISION}.tgz) export PVC_TESTS_HOST
[spark] branch master updated: [SPARK-35430][K8S] Switch on "PVs with local storage" integration test on Docker driver
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 7b90fd2 [SPARK-35430][K8S] Switch on "PVs with local storage" integration test on Docker driver 7b90fd2 is described below commit 7b90fd2ca79b9a1fec5fca0bdcc169c7962ad880 Author: attilapiros AuthorDate: Mon Aug 2 09:17:29 2021 -0700 [SPARK-35430][K8S] Switch on "PVs with local storage" integration test on Docker driver ### What changes were proposed in this pull request? Switching back the "PVs with local storage" integration test on Docker driver. I have analyzed why this test was failing on my machine (I hope the root cause of the problem is OS agnostic). It failed because of the mounting of the host directory into the Minikube node using the `--uid=185` (Spark user user id): ``` $ minikube mount ${PVC_TESTS_HOST_PATH}:${PVC_TESTS_VM_PATH} --9p-version=9p2000.L --gid=0 --uid=185 &; MOUNT_PID=$! ``` Are referring to a nonexistent user. See the the number of occurence of 185 in "/etc/passwd": ``` $ minikube ssh "grep -c 185 /etc/passwd" 0 ``` This leads to a permission denied. Skipping the `--uid=185` won't help although the path will listable before the test execution: ``` ╭─attilazsoltpirosapiros-MBP16 ~/git/attilapiros/spark ‹SPARK-35430*› ╰─$ 📁 Mounting host path /var/folders/t_/fr_vqcyx23vftk81ftz1k5hwgn/T/tmp.k9X4Gecv into VM as /var/folders/t_/fr_vqcyx23vftk81ftz1k5hwgn/T/tmp.k9X4Gecv ... ▪ Mount type: ▪ User ID: docker ▪ Group ID: 0 ▪ Version: 9p2000.L ▪ Message Size: 262144 ▪ Permissions: 755 (-rwxr-xr-x) ▪ Options: map[] ▪ Bind Address: 127.0.0.1:51740 🚀 Userspace file server: ufs starting ╭─attilazsoltpirosapiros-MBP16 ~/git/attilapiros/spark ‹SPARK-35430*› ╰─$ minikube ssh "ls /var/folders/t_/fr_vqcyx23vftk81ftz1k5hwgn/T/tmp.k9X4Gecv" ╭─attilazsoltpirosapiros-MBP16 ~/git/attilapiros/spark ‹SPARK-35430*› ╰─$ ``` But the test will fail and after its execution the `dmesg` shows the following error: ``` [13670.493359] bpfilter: Loaded bpfilter_umh pid 66153 [13670.493363] bpfilter: write fail -32 [13670.530737] bpfilter: Loaded bpfilter_umh pid 66155 ... ``` This `bpfilter` is a firewall module and we are back to a permission denied when we want to list the mounted directory. The solution is to add a spark user with 185 uid when the minikube is started. **So this must be added to Jenkins job (and the mount should use --gid=0 --uid=185)**: ``` $ minikube ssh "sudo useradd spark -u 185 -g 0 -m -s /bin/bash" ``` ### Why are the changes needed? This integration test is needed to validate the PVs feature. ### Does this PR introduce _any_ user-facing change? No. It is just testing. ### How was this patch tested? Running the test locally: ``` KubernetesSuite: - Run SparkPi with no resources - Run SparkPi with a very long application name. - Use SparkLauncher.NO_RESOURCE - Run SparkPi with a master URL without a scheme. - Run SparkPi with an argument. - Run SparkPi with custom labels, annotations, and environment variables. - All pods have the same service account by default - Run extraJVMOptions check on driver - Run SparkRemoteFileTest using a remote data file - Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j.properties - Run SparkPi with env and mount secrets. - Run PySpark on simple pi.py example - Run PySpark to test a pyfiles example - Run PySpark with memory customization - Run in client mode. - Start pod creation from template - PVs with local storage ``` The "PVs with local storage" was successful but the next test `Launcher client dependencies` the minio stops the test executions on Mac (only on Mac): ``` 21/06/29 04:33:32.449 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: 🏃 Starting tunnel for service minio-s3. 21/06/29 04:33:33.425 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: |--|--|-|| 21/06/29 04:33:33.426 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: |NAMESPACE | NAME | TARGET PORT | URL | 21/06/29 04:33:33.426 ScalaTest-main-running-KubernetesSuite INFO ProcessUtils: |--|--|-|
[spark] branch master updated (d506815 -> ad528a0)
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from d506815 [SPARK-36188][PYTHON] Add categories setter to CategoricalAccessor and CategoricalIndex add ad528a0 [SPARK-32797][SPARK-32391][SPARK-33242][SPARK-32666][ANSIBLE] updating a bunch of python packages No new revisions were added by this update. Summary of changes: .../files/python_environments/spark-py36-spec.txt | 64 +- 1 file changed, 61 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (733e85f1 -> 2c94fbc)
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 733e85f1 [SPARK-35953][SQL] Support extracting date fields from timestamp without time zone add 2c94fbc initial commit for skeleton ansible for jenkins worker config No new revisions were added by this update. Summary of changes: dev/.rat-excludes | 1 + dev/ansible-for-test-node/README.md| 25 +++ .../deploy-jenkins-worker.yml | 8 + dev/ansible-for-test-node/roles/common/README.md | 4 + .../roles/common/tasks/main.yml| 4 + .../roles/common/tasks/setup_local_userspace.yml | 8 + .../roles/common/tasks/system_packages.yml | 73 .../roles/jenkins-worker/README.md | 15 ++ .../roles/jenkins-worker/defaults/main.yml | 30 .../files/python_environments/base-py3-pip.txt | 3 + .../files/python_environments/base-py3-spec.txt| 21 +++ .../files/python_environments/py36.txt | 49 ++ .../files/python_environments/spark-py2-pip.txt| 8 + .../files/python_environments/spark-py36-spec.txt | 61 +++ .../files/python_environments/spark-py3k-spec.txt | 42 + .../files/scripts/jenkins-gitcache-cron| 7 + .../files/util_scripts/kill_zinc_nailgun.py| 60 +++ .../files/util_scripts/post_github_pr_comment.py | 81 + .../files/util_scripts/session_lock_resource.py| 152 + .../roles/jenkins-worker/files/worker-limits.conf | 5 + .../roles/jenkins-worker/tasks/cleanup.yml | 12 ++ .../jenkins-worker/tasks/install_anaconda.yml | 79 + .../tasks/install_build_packages.yml | 21 +++ .../roles/jenkins-worker/tasks/install_docker.yml | 33 .../jenkins-worker/tasks/install_minikube.yml | 16 ++ .../tasks/install_spark_build_packages.yml | 183 + .../jenkins-worker/tasks/jenkins_userspace.yml | 119 ++ .../roles/jenkins-worker/tasks/main.yml| 22 +++ .../roles/jenkins-worker/vars/main.yml | 9 + dev/tox.ini| 4 +- 30 files changed, 1153 insertions(+), 2 deletions(-) create mode 100644 dev/ansible-for-test-node/README.md create mode 100644 dev/ansible-for-test-node/deploy-jenkins-worker.yml create mode 100644 dev/ansible-for-test-node/roles/common/README.md create mode 100644 dev/ansible-for-test-node/roles/common/tasks/main.yml create mode 100644 dev/ansible-for-test-node/roles/common/tasks/setup_local_userspace.yml create mode 100644 dev/ansible-for-test-node/roles/common/tasks/system_packages.yml create mode 100644 dev/ansible-for-test-node/roles/jenkins-worker/README.md create mode 100644 dev/ansible-for-test-node/roles/jenkins-worker/defaults/main.yml create mode 100644 dev/ansible-for-test-node/roles/jenkins-worker/files/python_environments/base-py3-pip.txt create mode 100644 dev/ansible-for-test-node/roles/jenkins-worker/files/python_environments/base-py3-spec.txt create mode 100644 dev/ansible-for-test-node/roles/jenkins-worker/files/python_environments/py36.txt create mode 100644 dev/ansible-for-test-node/roles/jenkins-worker/files/python_environments/spark-py2-pip.txt create mode 100644 dev/ansible-for-test-node/roles/jenkins-worker/files/python_environments/spark-py36-spec.txt create mode 100644 dev/ansible-for-test-node/roles/jenkins-worker/files/python_environments/spark-py3k-spec.txt create mode 100644 dev/ansible-for-test-node/roles/jenkins-worker/files/scripts/jenkins-gitcache-cron create mode 100755 dev/ansible-for-test-node/roles/jenkins-worker/files/util_scripts/kill_zinc_nailgun.py create mode 100755 dev/ansible-for-test-node/roles/jenkins-worker/files/util_scripts/post_github_pr_comment.py create mode 100755 dev/ansible-for-test-node/roles/jenkins-worker/files/util_scripts/session_lock_resource.py create mode 100644 dev/ansible-for-test-node/roles/jenkins-worker/files/worker-limits.conf create mode 100644 dev/ansible-for-test-node/roles/jenkins-worker/tasks/cleanup.yml create mode 100644 dev/ansible-for-test-node/roles/jenkins-worker/tasks/install_anaconda.yml create mode 100644 dev/ansible-for-test-node/roles/jenkins-worker/tasks/install_build_packages.yml create mode 100644 dev/ansible-for-test-node/roles/jenkins-worker/tasks/install_docker.yml create mode 100644 dev/ansible-for-test-node/roles/jenkins-worker/tasks/install_minikube.yml create mode 100644 dev/ansible-for-test-node/roles/jenkins-worker/tasks/install_spark_build_packages.yml create mode 100644 dev/ansible-for-test-node/roles/jenkins-worker/tasks/jenkins_userspace.yml create mode 100644 dev/ansible-for-test-node/roles/jenkins-worker/tasks/main.yml create mode 100644 dev/ansible-for-test-node
[spark] branch branch-3.0 updated (efae8b6 -> 8eedc41)
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from efae8b6 [SPARK-33535][INFRA][TESTS] Export LANG to en_US.UTF-8 in run-tests-jenkins script add 8eedc41 [SPARK-33565][PYTHON][BUILD][3.0] Remove py38 spark3 No new revisions were added by this update. Summary of changes: python/run-tests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1de3fc4 -> c529426)
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1de3fc4 [SPARK-33525][SQL] Update hive-service-rpc to 3.1.2 add c529426 [SPARK-33565][BUILD][PYTHON] remove python3.8 and fix breakage No new revisions were added by this update. Summary of changes: python/run-tests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated (8bde6ed -> 7f522d5)
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a change to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git. from 8bde6ed [SPARK-31839][TESTS] Delete duplicate code in castsuit add 7f522d5 [BUILD][INFRA] bump the timeout to match the jenkins PRB No new revisions were added by this update. Summary of changes: dev/run-tests-jenkins.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated (8bde6ed -> 7f522d5)
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a change to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git. from 8bde6ed [SPARK-31839][TESTS] Delete duplicate code in castsuit add 7f522d5 [BUILD][INFRA] bump the timeout to match the jenkins PRB No new revisions were added by this update. Summary of changes: dev/run-tests-jenkins.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated (8bde6ed -> 7f522d5)
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a change to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git. from 8bde6ed [SPARK-31839][TESTS] Delete duplicate code in castsuit add 7f522d5 [BUILD][INFRA] bump the timeout to match the jenkins PRB No new revisions were added by this update. Summary of changes: dev/run-tests-jenkins.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [BUILD][INFRA] bump the timeout to match the jenkins PRB
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 9580266 [BUILD][INFRA] bump the timeout to match the jenkins PRB 9580266 is described below commit 9580266c8ee910dbf9a37dd6ff8baa9a94bca38e Author: shane knapp AuthorDate: Thu May 28 14:25:49 2020 -0700 [BUILD][INFRA] bump the timeout to match the jenkins PRB ### What changes were proposed in this pull request? bump the timeout to match what's set in jenkins ### Why are the changes needed? tests be timing out! ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? via jenkins Closes #28666 from shaneknapp/increase-jenkins-timeout. Authored-by: shane knapp Signed-off-by: shane knapp (cherry picked from commit 9e68affd13a3875b92f0700b8ab7c9d902f1a08c) Signed-off-by: shane knapp --- dev/run-tests-jenkins.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev/run-tests-jenkins.py b/dev/run-tests-jenkins.py index 72e32d4..13be959 100755 --- a/dev/run-tests-jenkins.py +++ b/dev/run-tests-jenkins.py @@ -198,7 +198,7 @@ def main(): # format: http://linux.die.net/man/1/timeout # must be less than the timeout configured on Jenkins. Usually Jenkins's timeout is higher # then this. Please consult with the build manager or a committer when it should be increased. -tests_timeout = "400m" +tests_timeout = "500m" # Array to capture all test names to run on the pull request. These tests are represented # by their file equivalents in the dev/tests/ directory. - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated (8bde6ed -> 7f522d5)
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a change to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git. from 8bde6ed [SPARK-31839][TESTS] Delete duplicate code in castsuit add 7f522d5 [BUILD][INFRA] bump the timeout to match the jenkins PRB No new revisions were added by this update. Summary of changes: dev/run-tests-jenkins.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [BUILD][INFRA] bump the timeout to match the jenkins PRB
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 9580266 [BUILD][INFRA] bump the timeout to match the jenkins PRB 9580266 is described below commit 9580266c8ee910dbf9a37dd6ff8baa9a94bca38e Author: shane knapp AuthorDate: Thu May 28 14:25:49 2020 -0700 [BUILD][INFRA] bump the timeout to match the jenkins PRB ### What changes were proposed in this pull request? bump the timeout to match what's set in jenkins ### Why are the changes needed? tests be timing out! ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? via jenkins Closes #28666 from shaneknapp/increase-jenkins-timeout. Authored-by: shane knapp Signed-off-by: shane knapp (cherry picked from commit 9e68affd13a3875b92f0700b8ab7c9d902f1a08c) Signed-off-by: shane knapp --- dev/run-tests-jenkins.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev/run-tests-jenkins.py b/dev/run-tests-jenkins.py index 72e32d4..13be959 100755 --- a/dev/run-tests-jenkins.py +++ b/dev/run-tests-jenkins.py @@ -198,7 +198,7 @@ def main(): # format: http://linux.die.net/man/1/timeout # must be less than the timeout configured on Jenkins. Usually Jenkins's timeout is higher # then this. Please consult with the build manager or a committer when it should be increased. -tests_timeout = "400m" +tests_timeout = "500m" # Array to capture all test names to run on the pull request. These tests are represented # by their file equivalents in the dev/tests/ directory. - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8bbb666 -> 9e68aff)
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8bbb666 [SPARK-25351][PYTHON][TEST][FOLLOWUP] Fix test assertions to be consistent add 9e68aff [BUILD][INFRA] bump the timeout to match the jenkins PRB No new revisions were added by this update. Summary of changes: dev/run-tests-jenkins.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8bbb666 -> 9e68aff)
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8bbb666 [SPARK-25351][PYTHON][TEST][FOLLOWUP] Fix test assertions to be consistent add 9e68aff [BUILD][INFRA] bump the timeout to match the jenkins PRB No new revisions were added by this update. Summary of changes: dev/run-tests-jenkins.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (c0e9f9f -> 4d23938)
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from c0e9f9f [SPARK-30459][SQL] Fix ignoreMissingFiles/ignoreCorruptFiles in data source v2 add 4d23938 [MINOR][SQL][TEST-HIVE1.2] Fix scalastyle error due to length line in hive-1.2 profile No new revisions were added by this update. Summary of changes: .../apache/spark/sql/execution/datasources/orc/OrcFilterSuite.scala| 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (c0e9f9f -> 4d23938)
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from c0e9f9f [SPARK-30459][SQL] Fix ignoreMissingFiles/ignoreCorruptFiles in data source v2 add 4d23938 [MINOR][SQL][TEST-HIVE1.2] Fix scalastyle error due to length line in hive-1.2 profile No new revisions were added by this update. Summary of changes: .../apache/spark/sql/execution/datasources/orc/OrcFilterSuite.scala| 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (5496e98 -> 708cf16)
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 5496e98 [SPARK-30109][ML] PCA use BLAS.gemv for sparse vectors add 708cf16 [SPARK-30111][K8S] Apt-get update to fix debian issues No new revisions were added by this update. Summary of changes: .../kubernetes/docker/src/main/dockerfiles/spark/bindings/R/Dockerfile | 2 +- .../docker/src/main/dockerfiles/spark/bindings/python/Dockerfile | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (e46e487 -> 04e99c1)
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from e46e487 [SPARK-29682][SQL] Resolve conflicting attributes in Expand correctly add 04e99c1 [SPARK-29672][PYSPARK] update spark testing framework to use python3 No new revisions were added by this update. Summary of changes: dev/pip-sanity-check.py| 2 -- dev/run-pip-tests | 23 +-- dev/run-tests | 6 +++--- dev/run-tests-jenkins | 8 +--- dev/run-tests-jenkins.py | 3 +-- dev/run-tests.py | 5 ++--- dev/sparktestsupport/shellutils.py | 6 ++ python/pyspark/context.py | 2 -- python/pyspark/version.py | 2 +- python/run-tests | 8 +++- python/run-tests.py| 17 ++--- python/setup.py| 7 +++ 12 files changed, 43 insertions(+), 46 deletions(-) mode change 100644 => 100755 python/setup.py - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (e46e487 -> 04e99c1)
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from e46e487 [SPARK-29682][SQL] Resolve conflicting attributes in Expand correctly add 04e99c1 [SPARK-29672][PYSPARK] update spark testing framework to use python3 No new revisions were added by this update. Summary of changes: dev/pip-sanity-check.py| 2 -- dev/run-pip-tests | 23 +-- dev/run-tests | 6 +++--- dev/run-tests-jenkins | 8 +--- dev/run-tests-jenkins.py | 3 +-- dev/run-tests.py | 5 ++--- dev/sparktestsupport/shellutils.py | 6 ++ python/pyspark/context.py | 2 -- python/pyspark/version.py | 2 +- python/run-tests | 8 +++- python/run-tests.py| 17 ++--- python/setup.py| 7 +++ 12 files changed, 43 insertions(+), 46 deletions(-) mode change 100644 => 100755 python/setup.py - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (e696c36 -> 52186af)
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from e696c36 [SPARK-29442][SQL] Set `default` mode should override the existing mode add 52186af [SPARK-25152][K8S] Enable SparkR Integration Tests for Kubernetes No new revisions were added by this update. Summary of changes: .../integration-tests/scripts/setup-integration-test-env.sh | 4 ++-- .../org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-28701][INFRA][FOLLOWUP] Fix the key error when looking in os.environ
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 84d4f94 [SPARK-28701][INFRA][FOLLOWUP] Fix the key error when looking in os.environ 84d4f94 is described below commit 84d4f945969e199a5d3fb658864e494b88d15f3c Author: shane knapp AuthorDate: Mon Aug 26 12:40:31 2019 -0700 [SPARK-28701][INFRA][FOLLOWUP] Fix the key error when looking in os.environ ### What changes were proposed in this pull request? i broke run-tests.py for non-PRB builds in this PR: https://github.com/apache/spark/pull/25423 ### Why are the changes needed? to fix what i broke ### Does this PR introduce any user-facing change? no ### How was this patch tested? the build system will test this Closes #25585 from shaneknapp/fix-run-tests. Authored-by: shane knapp Signed-off-by: shane knapp --- dev/run-tests.py | 9 + 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/dev/run-tests.py b/dev/run-tests.py index a338667..ea51570 100755 --- a/dev/run-tests.py +++ b/dev/run-tests.py @@ -405,10 +405,11 @@ def run_scala_tests(build_tool, hadoop_version, test_modules, excluded_tags): test_profiles += ['-Dtest.exclude.tags=' + ",".join(excluded_tags)] # set up java11 env if this is a pull request build with 'test-java11' in the title -if "test-java11" in os.environ["ghprbPullTitle"].lower(): -os.environ["JAVA_HOME"] = "/usr/java/jdk-11.0.1" -os.environ["PATH"] = "%s/bin:%s" % (os.environ["JAVA_HOME"], os.environ["PATH"]) -test_profiles += ['-Djava.version=11'] +if "ghprbPullTitle" in os.environ: +if "test-java11" in os.environ["ghprbPullTitle"].lower(): +os.environ["JAVA_HOME"] = "/usr/java/jdk-11.0.1" +os.environ["PATH"] = "%s/bin:%s" % (os.environ["JAVA_HOME"], os.environ["PATH"]) +test_profiles += ['-Djava.version=11'] if build_tool == "maven": run_scala_tests_maven(test_profiles) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.3 updated: [SPARK-25079][PYTHON][BRANCH-2.3] update python3 executable to 3.6.x
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a commit to branch branch-2.3 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.3 by this push: new a85ab12 [SPARK-25079][PYTHON][BRANCH-2.3] update python3 executable to 3.6.x a85ab12 is described below commit a85ab120e3d29a323e6d28aa307d4c20ee5f2c6c Author: shane knapp AuthorDate: Fri Apr 19 09:45:40 2019 -0700 [SPARK-25079][PYTHON][BRANCH-2.3] update python3 executable to 3.6.x ## What changes were proposed in this pull request? have jenkins test against python3.6 (instead of 3.4). ## How was this patch tested? extensive testing on both the centos and ubuntu jenkins workers revealed that 2.3 probably doesn't like python 3.6... :( NOTE: this is just for branch-2.3 PLEASE DO NOT MERGE Author: shane knapp Closes #24380 from shaneknapp/update-python-executable-2.3. --- python/run-tests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/python/run-tests.py b/python/run-tests.py index 3539c76..d571855 100755 --- a/python/run-tests.py +++ b/python/run-tests.py @@ -114,7 +114,7 @@ def run_individual_python_test(test_name, pyspark_python): def get_default_python_executables(): -python_execs = [x for x in ["python2.7", "python3.4", "pypy"] if which(x)] +python_execs = [x for x in ["python2.7", "python3.6", "pypy"] if which(x)] if "python2.7" not in python_execs: LOGGER.warning("Not testing against `python2.7` because it could not be found; falling" " back to `python` instead") - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [SPARK-25079][PYTHON][BRANCH-2.4] update python3 executable to 3.6.x
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new eaa88ae [SPARK-25079][PYTHON][BRANCH-2.4] update python3 executable to 3.6.x eaa88ae is described below commit eaa88ae5237b23fb7497838f3897a64641efe383 Author: shane knapp AuthorDate: Fri Apr 19 09:44:06 2019 -0700 [SPARK-25079][PYTHON][BRANCH-2.4] update python3 executable to 3.6.x ## What changes were proposed in this pull request? have jenkins test against python3.6 (instead of 3.4). ## How was this patch tested? extensive testing on both the centos and ubuntu jenkins workers revealed that 2.4 doesn't like python 3.6... :( NOTE: this is just for branch-2.4 PLEASE DO NOT MERGE Closes #24379 from shaneknapp/update-python-executable. Authored-by: shane knapp Signed-off-by: shane knapp --- python/run-tests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/python/run-tests.py b/python/run-tests.py index ccbdfac..921fdc9 100755 --- a/python/run-tests.py +++ b/python/run-tests.py @@ -162,7 +162,7 @@ def run_individual_python_test(target_dir, test_name, pyspark_python): def get_default_python_executables(): -python_execs = [x for x in ["python2.7", "python3.4", "pypy"] if which(x)] +python_execs = [x for x in ["python2.7", "python3.6", "pypy"] if which(x)] if "python2.7" not in python_execs: LOGGER.warning("Not testing against `python2.7` because it could not be found; falling" " back to `python` instead") - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark-website] branch asf-site updated: testing how-to for k8s changes
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 5f4985d testing how-to for k8s changes 5f4985d is described below commit 5f4985db6340efa0107ebe34bb285ab8ceb74f65 Author: shane knapp AuthorDate: Thu Mar 28 11:03:09 2019 -0700 testing how-to for k8s changes i think that this will be quite useful. :) Author: shane knapp Closes #186 from shaneknapp/add-k8s-testing-instructions. --- developer-tools.md| 48 ++ site/developer-tools.html | 49 +++ 2 files changed, 97 insertions(+) diff --git a/developer-tools.md b/developer-tools.md index 43ad445..00d57cd 100644 --- a/developer-tools.md +++ b/developer-tools.md @@ -175,6 +175,54 @@ You can check the coverage report visually by HTMLs under `/.../spark/python/tes Please check other available options via `python/run-tests[-with-coverage] --help`. +Testing K8S + +If you have made changes to the K8S bindings in Apache Spark, it would behoove you to test locally before submitting a PR. This is relatively simple to do, but it will require a local (to you) installation of [minikube](https://kubernetes.io/docs/setup/minikube/). Due to how minikube interacts with the host system, please be sure to set things up as follows: + +- minikube version v0.34.1 (or greater, but backwards-compatibility between versions is spotty) +- You must use a VM driver! Running minikube with the `--vm-driver=none` option requires that the user launching minikube/k8s have root access. Our Jenkins workers use the [kvm2](https://github.com/kubernetes/minikube/blob/master/docs/drivers.md#kvm2-driver) drivers. More details [here](https://github.com/kubernetes/minikube/blob/master/docs/drivers.md). +- kubernetes version v1.13.3 (can be set by executing `minikube config set kubernetes-version v1.13.3`) + +Once you have minikube properly set up, and have successfully completed the [quick start](https://kubernetes.io/docs/setup/minikube/#quickstart), you can test your changes locally. All subsequent commands should be run from your root spark/ repo directory: + +1) Build a tarball to test against: + +``` +export DATE=`date "+%Y%m%d"` +export REVISION=`git rev-parse --short HEAD` +export ZINC_PORT=$(python -S -c "import random; print random.randrange(3030,4030)") + +./dev/make-distribution.sh --name ${DATE}-${REVISION} --pip --tgz -DzincPort=${ZINC_PORT} \ + -Phadoop-2.7 -Pkubernetes -Pkinesis-asl -Phive -Phive-thriftserver +``` + +2) Use that tarball and run the K8S integration tests: + +``` +PVC_TMP_DIR=$(mktemp -d) +export PVC_TESTS_HOST_PATH=$PVC_TMP_DIR +export PVC_TESTS_VM_PATH=$PVC_TMP_DIR + +minikube --vm-driver= start --memory 6000 --cpus 8 + +minikube mount ${PVC_TESTS_HOST_PATH}:${PVC_TESTS_VM_PATH} --9p-version=9p2000.L --gid=0 --uid=185 & + +MOUNT_PID=$(jobs -rp) + +kubectl create clusterrolebinding serviceaccounts-cluster-admin --clusterrole=cluster-admin --group=system:serviceaccounts || true + +./resource-managers/kubernetes/integration-tests/dev/dev-run-integration-tests.sh \ +--spark-tgz ${WORKSPACE}/spark-*.tgz + +kill -9 $MOUNT_PID +minikube stop +``` + +After the run is completed, the integration test logs are saved here: `./resource-managers/kubernetes/integration-tests/target/integration-tests.log` + +Getting logs from the pods and containers directly is an exercise left to the reader. + +Kubernetes, and more importantly, minikube have rapid release cycles, and point releases have been found to be buggy and/or break older and existing functionality. If you are having trouble getting tests to pass on Jenkins, but locally things work, don't hesitate to file a Jira issue. ScalaTest Issues diff --git a/site/developer-tools.html b/site/developer-tools.html index e676d7b..17da11c 100644 --- a/site/developer-tools.html +++ b/site/developer-tools.html @@ -353,6 +353,55 @@ Generating HTML files for PySpark coverage under /.../spark/python/test_coverage Please check other available options via python/run-tests[-with-coverage] --help. +Testing K8S + +If you have made changes to the K8S bindings in Apache Spark, it would behoove you to test locally before submitting a PR. This is relatively simple to do, but it will require a local (to you) installation of https://kubernetes.io/docs/setup/minikube/";>minikube. Due to how minikube interacts with the host system, please be sure to set things up as follows: + + + minikube version v0.34.1 (or greater, but backwards-compatibility between versions is spotty) + You must use a VM driver! Running minikube with the --vm-driver=none option requires that the user launching minikube/k8s have root acces
[spark] branch master updated: [SPARK-24902][K8S] Add PV integration tests
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 39577a2 [SPARK-24902][K8S] Add PV integration tests 39577a2 is described below commit 39577a27a0b58fd75b41d24b10012447748b7ee9 Author: Stavros Kontopoulos AuthorDate: Wed Mar 27 13:00:56 2019 -0700 [SPARK-24902][K8S] Add PV integration tests ## What changes were proposed in this pull request? - Adds persistent volume integration tests - Adds a custom tag to the test to exclude it if it is run against a cloud backend. - Assumes default fs type for the host, AFAIK that is ext4. ## How was this patch tested? Manually run the tests against minikube as usual: ``` [INFO] --- scalatest-maven-plugin:1.0:test (integration-test) spark-kubernetes-integration-tests_2.12 --- Discovery starting. Discovery completed in 192 milliseconds. Run starting. Expected test count is: 16 KubernetesSuite: - Run SparkPi with no resources - Run SparkPi with a very long application name. - Use SparkLauncher.NO_RESOURCE - Run SparkPi with a master URL without a scheme. - Run SparkPi with an argument. - Run SparkPi with custom labels, annotations, and environment variables. - Run extraJVMOptions check on driver - Run SparkRemoteFileTest using a remote data file - Run SparkPi with env and mount secrets. - Run PySpark on simple pi.py example - Run PySpark with Python2 to test a pyfiles example - Run PySpark with Python3 to test a pyfiles example - Run PySpark with memory customization - Run in client mode. - Start pod creation from template - Test PVs with local storage ``` Closes #23514 from skonto/pvctests. Authored-by: Stavros Kontopoulos Signed-off-by: shane knapp --- .../apache/spark/examples/DFSReadWriteTest.scala | 12 +- .../k8s/integrationtest/KubernetesSuite.scala | 35 +++- .../integrationtest/KubernetesTestComponents.scala | 3 +- .../deploy/k8s/integrationtest/PVTestsSuite.scala | 189 + .../k8s/integrationtest/SecretsTestsSuite.scala| 27 +-- .../spark/deploy/k8s/integrationtest/Utils.scala | 22 +++ 6 files changed, 260 insertions(+), 28 deletions(-) diff --git a/examples/src/main/scala/org/apache/spark/examples/DFSReadWriteTest.scala b/examples/src/main/scala/org/apache/spark/examples/DFSReadWriteTest.scala index 1a77971..a738598 100644 --- a/examples/src/main/scala/org/apache/spark/examples/DFSReadWriteTest.scala +++ b/examples/src/main/scala/org/apache/spark/examples/DFSReadWriteTest.scala @@ -22,6 +22,9 @@ import java.io.File import scala.io.Source._ +import org.apache.hadoop.fs.FileSystem +import org.apache.hadoop.fs.Path + import org.apache.spark.sql.SparkSession /** @@ -107,6 +110,13 @@ object DFSReadWriteTest { println("Writing local file to DFS") val dfsFilename = s"$dfsDirPath/dfs_read_write_test" + +// delete file if exists +val fs = FileSystem.get(spark.sessionState.newHadoopConf()) +if (fs.exists(new Path(dfsFilename))) { +fs.delete(new Path(dfsFilename), true) +} + val fileRDD = spark.sparkContext.parallelize(fileContents) fileRDD.saveAsTextFile(dfsFilename) @@ -123,7 +133,6 @@ object DFSReadWriteTest { .sum spark.stop() - if (localWordCount == dfsWordCount) { println(s"Success! Local Word Count $localWordCount and " + s"DFS Word Count $dfsWordCount agree.") @@ -131,7 +140,6 @@ object DFSReadWriteTest { println(s"Failure! Local Word Count $localWordCount " + s"and DFS Word Count $dfsWordCount disagree.") } - } } // scalastyle:on println diff --git a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala index 91419e8..bc0bb20 100644 --- a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala +++ b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala @@ -40,7 +40,7 @@ import org.apache.spark.internal.config._ class KubernetesSuite extends SparkFunSuite with BeforeAndAfterAll with BeforeAndAfter with BasicTestsSuite with SecretsTestsSuite - with PythonTestsSuite with ClientModeTestsSuite with PodTemplateSuite + with PythonTestsSuite with ClientModeTestsSuite with PodTemplateSuite with PVTestsSuite with Logging with Eventually with Matchers { import KubernetesSuite._ @@ -178,6 +178,29 @@ class KubernetesSuite
[spark] branch master updated: [SPARK-27178][K8S] add nss to the spark/k8s Dockerfile
This is an automated email from the ASF dual-hosted git repository. shaneknapp pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 5564fe5 [SPARK-27178][K8S] add nss to the spark/k8s Dockerfile 5564fe5 is described below commit 5564fe51513f725d2526dbf9e25a2f2c40d19afc Author: shane knapp AuthorDate: Mon Mar 18 16:38:42 2019 -0700 [SPARK-27178][K8S] add nss to the spark/k8s Dockerfile ## What changes were proposed in this pull request? while performing some tests on our existing minikube and k8s infrastructure, i noticed that the integration tests were failing. i dug in and discovered the following message buried at the end of the stacktrace: ``` Caused by: java.io.FileNotFoundException: /usr/lib/libnss3.so at sun.security.pkcs11.Secmod.initialize(Secmod.java:193) at sun.security.pkcs11.SunPKCS11.(SunPKCS11.java:218) ... 81 more ``` after i added the `nss` package to `resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile`, everything worked. this is also impacting current builds. see: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8959/console ## How was this patch tested? i tested locally before pushing, and the build system will test the rest. Closes #24111 from shaneknapp/add-nss-package-to-dockerfile. Authored-by: shane knapp Signed-off-by: shane knapp --- .../kubernetes/docker/src/main/dockerfiles/spark/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile index 1d8ac3c..871d34b 100644 --- a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile +++ b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile @@ -29,7 +29,7 @@ ARG spark_uid=185 RUN set -ex && \ apk upgrade --no-cache && \ ln -s /lib /lib64 && \ -apk add --no-cache bash tini libc6-compat linux-pam krb5 krb5-libs && \ +apk add --no-cache bash tini libc6-compat linux-pam krb5 krb5-libs nss && \ mkdir -p /opt/spark && \ mkdir -p /opt/spark/examples && \ mkdir -p /opt/spark/work-dir && \ - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [BUILD] refactor dev/lint-python in to something readable
Repository: spark Updated Branches: refs/heads/master db136d360 -> 42c48387c [BUILD] refactor dev/lint-python in to something readable ## What changes were proposed in this pull request? `dev/lint-python` is a mess of nearly unreadable bash. i would like to fix that as best as i can. ## How was this patch tested? the build system will test this. Closes #22994 from shaneknapp/lint-python-refactor. Authored-by: shane knapp Signed-off-by: shane knapp Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/42c48387 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/42c48387 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/42c48387 Branch: refs/heads/master Commit: 42c48387c047d96154bcfeb95fcb816a43e60d7c Parents: db136d3 Author: shane knapp Authored: Tue Nov 20 12:38:40 2018 -0800 Committer: shane knapp Committed: Tue Nov 20 12:38:40 2018 -0800 -- dev/lint-python | 359 +++ 1 file changed, 220 insertions(+), 139 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/42c48387/dev/lint-python -- diff --git a/dev/lint-python b/dev/lint-python index 27d87f6..0681693 100755 --- a/dev/lint-python +++ b/dev/lint-python @@ -1,5 +1,4 @@ #!/usr/bin/env bash - # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with @@ -16,160 +15,242 @@ # See the License for the specific language governing permissions and # limitations under the License. # +# define test binaries + versions +PYDOCSTYLE_BUILD="pydocstyle" +MINIMUM_PYDOCSTYLE="3.0.0" -SCRIPT_DIR="$( cd "$( dirname "$0" )" && pwd )" -SPARK_ROOT_DIR="$(dirname "$SCRIPT_DIR")" -# Exclude auto-generated configuration file. -PATHS_TO_CHECK="$( cd "$SPARK_ROOT_DIR" && find . -name "*.py" )" -DOC_PATHS_TO_CHECK="$( cd "$SPARK_ROOT_DIR" && find . -name "*.py" | grep -vF 'functions.py' )" -PYCODESTYLE_REPORT_PATH="$SPARK_ROOT_DIR/dev/pycodestyle-report.txt" -PYDOCSTYLE_REPORT_PATH="$SPARK_ROOT_DIR/dev/pydocstyle-report.txt" -PYLINT_REPORT_PATH="$SPARK_ROOT_DIR/dev/pylint-report.txt" -PYLINT_INSTALL_INFO="$SPARK_ROOT_DIR/dev/pylint-info.txt" - -PYDOCSTYLEBUILD="pydocstyle" -MINIMUM_PYDOCSTYLEVERSION="3.0.0" - -FLAKE8BUILD="flake8" +FLAKE8_BUILD="flake8" MINIMUM_FLAKE8="3.5.0" -SPHINXBUILD=${SPHINXBUILD:=sphinx-build} -SPHINX_REPORT_PATH="$SPARK_ROOT_DIR/dev/sphinx-report.txt" +PYCODESTYLE_BUILD="pycodestyle" +MINIMUM_PYCODESTYLE="2.4.0" -cd "$SPARK_ROOT_DIR" +SPHINX_BUILD="sphinx-build" -# compileall: https://docs.python.org/2/library/compileall.html -python -B -m compileall -q -l $PATHS_TO_CHECK > "$PYCODESTYLE_REPORT_PATH" -compile_status="${PIPESTATUS[0]}" +function compile_python_test { +local COMPILE_STATUS= +local COMPILE_REPORT= + +if [[ ! "$1" ]]; then +echo "No python files found! Something is very wrong -- exiting." +exit 1; +fi -# Get pycodestyle at runtime so that we don't rely on it being installed on the build server. -# See: https://github.com/apache/spark/pull/1744#issuecomment-50982162 -# Updated to the latest official version of pep8. pep8 is formally renamed to pycodestyle. -PYCODESTYLE_VERSION="2.4.0" -PYCODESTYLE_SCRIPT_PATH="$SPARK_ROOT_DIR/dev/pycodestyle-$PYCODESTYLE_VERSION.py" -PYCODESTYLE_SCRIPT_REMOTE_PATH="https://raw.githubusercontent.com/PyCQA/pycodestyle/$PYCODESTYLE_VERSION/pycodestyle.py"; +# compileall: https://docs.python.org/2/library/compileall.html +echo "starting python compilation test..." +COMPILE_REPORT=$( (python -B -mcompileall -q -l $1) 2>&1) +COMPILE_STATUS=$? + +if [ $COMPILE_STATUS -ne 0 ]; then +echo "Python compilation failed with the following errors:" +echo "$COMPILE_REPORT" +echo "$COMPILE_STATUS" +exit "$COMPILE_STATUS" +else +echo "python compilation succeeded." +echo +fi +} -if [ ! -e "$PYCODESTYLE_SCRIPT_PATH" ]; then -curl --silent -o "$PYCODESTYLE_SCRIPT_PATH" "$PYCODESTYLE_SCRIPT_REMOTE_PATH" -curl_status="$?" +function pycodestyle_test { +local PYCODESTYLE_STATUS= +local PYCODESTYLE_REPORT= +local RUN_LOCAL_PYCODESTYLE= +local VERSION= +lo