GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/6967
[SPARK-8583] Refactor python/run-tests to integrate with dev/run-tests
module system
This patch refactors the `python/run-tests` script:
- It's now written in Python instead of Bash.
- The descriptions of the tests to run are now stored in `dev/run-tests`'s
modules. This allows the pull request builder to skip Python tests suites that
were not affected by the pull request's changes. For example, we can now skip
the PySpark Streaming test cases when only SQL files are changed.
- `dev/run-tests` has been split into multiple files: the module
definitions and test utility functions are now stored inside of a
`dev/sparktestsupport` Python module, allowing them to be re-used from the
Python test runner script.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark run-tests-python-modules
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/6967.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6967
----
commit 1bdeb87de057187655abfaddf95491580629d7aa
Author: Josh Rosen <[email protected]>
Date: 2015-06-22T22:40:05Z
Move module definitions to separate file.
commit 311c6a99b313aa277cfc7d44be544a33c6550289
Author: Josh Rosen <[email protected]>
Date: 2015-06-22T22:46:32Z
Move shell utility functions to own module.
commit 32660fc94545f48e5a39785061261697fe4f447f
Author: Josh Rosen <[email protected]>
Date: 2015-06-23T05:58:06Z
Initial cut at Python test runner refactoring
commit dcc9c09719cafc172bf3d58b7e81c73ae4535eaa
Author: Josh Rosen <[email protected]>
Date: 2015-06-23T06:16:48Z
Fix time division
commit 4c9713653a05e66a279885c50c99b288ba7089fa
Author: Josh Rosen <[email protected]>
Date: 2015-06-23T07:28:18Z
PYTHONPATH fixes
commit 04015b9dd5d2a2f47c1ca48408bca8949709ab01
Author: Josh Rosen <[email protected]>
Date: 2015-06-24T01:06:48Z
First attempt at getting PySpark Kafka test to work in new runner script
commit aec0b8f8cdfe7017066a29b77b446c923abe3ab6
Author: Josh Rosen <[email protected]>
Date: 2015-06-24T01:12:01Z
Actually get the Kafka stuff to run properly
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
