GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/7031
[SPARK-5161] [WIP] Parallelize Python test execution
This is a WIP commit towards parallelizing Python unit test execution,
which should significantly reduce build times in Jenkins.
This is based on #6967; I'll rebase after that PR is merged.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark parallelize-python-tests
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7031.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7031
----
commit 1bdeb87de057187655abfaddf95491580629d7aa
Author: Josh Rosen <[email protected]>
Date: 2015-06-22T22:40:05Z
Move module definitions to separate file.
commit 311c6a99b313aa277cfc7d44be544a33c6550289
Author: Josh Rosen <[email protected]>
Date: 2015-06-22T22:46:32Z
Move shell utility functions to own module.
commit 32660fc94545f48e5a39785061261697fe4f447f
Author: Josh Rosen <[email protected]>
Date: 2015-06-23T05:58:06Z
Initial cut at Python test runner refactoring
commit dcc9c09719cafc172bf3d58b7e81c73ae4535eaa
Author: Josh Rosen <[email protected]>
Date: 2015-06-23T06:16:48Z
Fix time division
commit 4c9713653a05e66a279885c50c99b288ba7089fa
Author: Josh Rosen <[email protected]>
Date: 2015-06-23T07:28:18Z
PYTHONPATH fixes
commit 04015b9dd5d2a2f47c1ca48408bca8949709ab01
Author: Josh Rosen <[email protected]>
Date: 2015-06-24T01:06:48Z
First attempt at getting PySpark Kafka test to work in new runner script
commit aec0b8f8cdfe7017066a29b77b446c923abe3ab6
Author: Josh Rosen <[email protected]>
Date: 2015-06-24T01:12:01Z
Actually get the Kafka stuff to run properly
commit def2d8a951416514f975c9a19f9142c7f4086bfc
Author: Josh Rosen <[email protected]>
Date: 2015-06-24T01:38:05Z
Two minor fixes
commit d6a77d30fa33651ac70aa36ad2bb9c5ce85d01d9
Author: Josh Rosen <[email protected]>
Date: 2015-06-24T05:26:43Z
Fix the tests of dev/run-tests
commit caeb040b28c7151a5a3f0898b5b418fa063ff4c6
Author: Josh Rosen <[email protected]>
Date: 2015-06-24T16:53:23Z
Fixes to PySpark test module definitions
commit b2ab027852c596da6dcf18df2109386653beb97a
Author: Josh Rosen <[email protected]>
Date: 2015-06-24T20:28:13Z
Add command-line options for running individual suites in python/run-tests
commit 2efd594a6fb7c323c1a3d94e0398d6f5c6b66c58
Author: Josh Rosen <[email protected]>
Date: 2015-06-24T20:32:37Z
Update dev/run-tests to use new Python test runner flags
commit fff4d099f2e59b5ecccaa9107cb0b3becde65dac
Author: Josh Rosen <[email protected]>
Date: 2015-06-24T20:34:34Z
Add dev/sparktestsupport to pep8 checks
commit f542ac5a28902b4021641c61ff508b2ce0e1c145
Author: Josh Rosen <[email protected]>
Date: 2015-06-24T23:10:28Z
Fix lint check for Python 3
commit 8f3244c1127c7f4d30cf0018d904e88bc375e562
Author: Josh Rosen <[email protected]>
Date: 2015-06-25T05:58:27Z
Use universal_newlines to fix dev/run-tests doctest failures on Python 3.
commit 4f8902cc3f819a16354584d09ad7054072a4b1ea
Author: Josh Rosen <[email protected]>
Date: 2015-06-25T05:59:18Z
Python lint fixes.
commit d33e52503009256f3ff228be7e9c3c375a5b34d2
Author: Josh Rosen <[email protected]>
Date: 2015-06-25T06:22:29Z
Merge remote-tracking branch 'origin/master' into run-tests-python-modules
commit 9c80469099790fbb2aa3957c2054ba44e061a76d
Author: Josh Rosen <[email protected]>
Date: 2015-06-25T23:59:44Z
Fix passing of PYSPARK_PYTHON
commit 7fe7c70da37490e7fea69952d8a3c6c670cdf868
Author: Josh Rosen <[email protected]>
Date: 2015-06-26T02:31:15Z
Merge remote-tracking branch 'origin/master' into parallelize-python-tests
commit f0909379fb44095f74a236c7c5b8a167f6e585c6
Author: Josh Rosen <[email protected]>
Date: 2015-06-26T02:54:33Z
Initial attempt at parallelizing Python test execution
commit 78fd0be5c46b91601aa7ad9ea9eed11d582a20c8
Author: Josh Rosen <[email protected]>
Date: 2015-06-26T02:55:23Z
Temporarily disable JVM tests so we can test Python speedup in Jenkins.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
