GitHub user HyukjinKwon opened a pull request:
https://github.com/apache/spark/pull/19665
[SPARK-22376][TESTS] Makes dev/run-tests.py script compatible with Python 3
## What changes were proposed in this pull request?
This PR proposes to fix `dev/run-tests.py` script to support Python 3.
Here are some backgrounds. Up to my knowledge,
In Python 2,
- `unicode` is NOT `str` in Python 2 (`type("foo") != type(u"foo")`).
- `str` has an alias, `bytes` in Python 2 (`type("foo") == type(b"foo")`).
In Python 3,
- `unicode` was (roughly) replaced by `str` in Python 3 (`type("foo") ==
type(u"foo")`).
- `str` is NOT `bytes` in Python 3 (`type("foo") != type(b"foo")`).
So, this PR fixes:
1. Use `b''` instead of `''` so that both `str` in Python 2 and `bytes`
in Python 3 can be hanlded. `sbt_proc.stdout.readline()` returns `str` (which
has an alias, `bytes`) in Python 2 and `bytes` in Python 3
2. Similarily, use `b''` instead of `''` so that both `str` in Python 2
and `bytes` in Python 3 can be hanlded. `re.compile` with `str` pattern does
not seem supporting to match `bytes` in Python 3:
Actually, this change is recommended up to my knowledge -
https://docs.python.org/3/howto/pyporting.html#text-versus-binary-data:
> Mark all binary literals with a b prefix, textual literals with a u prefix
## How was this patch tested?
I manually tested this via Python 3 with few additional changes to reduce
the elapsed time.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HyukjinKwon/spark SPARK-22376
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19665.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19665
----
commit 2c3d6f14a7f2884f5194475cf4f47bcdcb05b48e
Author: hyukjinkwon <[email protected]>
Date: 2017-11-06T09:13:53Z
Makes dev/run-tests.py script compatible with Python 3
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
