This is an automated email from the ASF dual-hosted git repository.
cutlerb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 75d8449 [SPARK-26676][PYTHON] Make
HiveContextSQLTests.test_unbounded_frames test compatible with Python 2 and PyPy
75d8449 is described below
commit 75d84498a4e649dba0b8c9ec35965066246d1bd2
Author: Hyukjin Kwon <gurwls...@apache.org>
AuthorDate: Mon Jan 21 14:27:17 2019 -0800
[SPARK-26676][PYTHON] Make HiveContextSQLTests.test_unbounded_frames test
compatible with Python 2 and PyPy
## What changes were proposed in this pull request?
This particular test is being skipped at PyPy and Python 2.
```
Skipped tests in pyspark.sql.tests.test_context with pypy:
test_unbounded_frames
(pyspark.sql.tests.test_context.HiveContextSQLTests) ... skipped "Unittest <
3.3 doesn't support mocking"
Skipped tests in pyspark.sql.tests.test_context with python2.7:
test_unbounded_frames
(pyspark.sql.tests.test_context.HiveContextSQLTests) ... skipped "Unittest <
3.3 doesn't support mocking"
```
We don't have to use unittest 3.3 module to mock. And looks the test itself
isn't compatible with Python 2.
This PR makes:
- Manually monkey-patch `sys.maxsize` to get rid of unittest 3.3 condition
- Use the built-in `reload` in Python 2, and `importlib.reload` in Python 3
## How was this patch tested?
Manually tested, and unit test is fixed.
Closes #23604 from HyukjinKwon/test-window.
Authored-by: Hyukjin Kwon <gurwls...@apache.org>
Signed-off-by: Bryan Cutler <cutl...@gmail.com>
---
python/pyspark/sql/tests/test_context.py | 36 +++++++++++++++-----------------
1 file changed, 17 insertions(+), 19 deletions(-)
diff --git a/python/pyspark/sql/tests/test_context.py
b/python/pyspark/sql/tests/test_context.py
index 918f4ad..dc81426 100644
--- a/python/pyspark/sql/tests/test_context.py
+++ b/python/pyspark/sql/tests/test_context.py
@@ -19,6 +19,11 @@ import shutil
import sys
import tempfile
import unittest
+try:
+ from importlib import reload # Python 3.4+ only.
+except ImportError:
+ # Otherwise, we will stick to Python 2's built-in reload.
+ pass
import py4j
@@ -216,12 +221,9 @@ class HiveContextSQLTests(ReusedPySparkTestCase):
parse_result =
df.select(functions.to_date(functions.col("dateCol"))).first()
self.assertEquals(date(2017, 1, 22),
parse_result['to_date(`dateCol`)'])
- @unittest.skipIf(sys.version_info < (3, 3), "Unittest < 3.3 doesn't
support mocking")
def test_unbounded_frames(self):
- from unittest.mock import patch
from pyspark.sql import functions as F
from pyspark.sql import window
- import importlib
df = self.spark.range(0, 3)
@@ -235,22 +237,18 @@ class HiveContextSQLTests(ReusedPySparkTestCase):
F.count("*").over(window.Window.rangeBetween(-sys.maxsize,
sys.maxsize))
).columns[0]
- with patch("sys.maxsize", 2 ** 31 - 1):
- importlib.reload(window)
- self.assertTrue(rows_frame_match())
- self.assertTrue(range_frame_match())
-
- with patch("sys.maxsize", 2 ** 63 - 1):
- importlib.reload(window)
- self.assertTrue(rows_frame_match())
- self.assertTrue(range_frame_match())
-
- with patch("sys.maxsize", 2 ** 127 - 1):
- importlib.reload(window)
- self.assertTrue(rows_frame_match())
- self.assertTrue(range_frame_match())
-
- importlib.reload(window)
+ for new_maxsize in [2 ** 31 - 1, 2 ** 63 - 1, 2 ** 127 - 1]:
+ old_maxsize = sys.maxsize
+ sys.maxsize = new_maxsize
+ try:
+ # Manually reload window module to use monkey-patched
sys.maxsize.
+ reload(window)
+ self.assertTrue(rows_frame_match())
+ self.assertTrue(range_frame_match())
+ finally:
+ sys.maxsize = old_maxsize
+
+ reload(window)
if __name__ == "__main__":
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
