Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/20487#discussion_r165795538
--- Diff: python/pyspark/sql/tests.py ---
@@ -48,19 +48,26 @@
else:
import unittest
-_have_pandas = False
-_have_old_pandas = False
+_pandas_requirement_message = None
try:
- import pandas
- try:
- from pyspark.sql.utils import require_minimum_pandas_version
- require_minimum_pandas_version()
- _have_pandas = True
- except:
- _have_old_pandas = True
-except:
- # No Pandas, but that's okay, we'll skip those tests
- pass
+ from pyspark.sql.utils import require_minimum_pandas_version
+ require_minimum_pandas_version()
+except ImportError as e:
+ from pyspark.util import _exception_message
+ # If Pandas version requirement is not satisfied, skip related tests.
+ _pandas_requirement_message = _exception_message(e)
+
+_pyarrow_requirement_message = None
+try:
+ from pyspark.sql.utils import require_minimum_pyarrow_version
+ require_minimum_pyarrow_version()
+except ImportError as e:
+ from pyspark.util import _exception_message
+ # If Arrow version requirement is not satisfied, skip related tests.
+ _pyarrow_requirement_message = _exception_message(e)
+
+_have_pandas = _pandas_requirement_message is None
+_have_pyarrow = _pyarrow_requirement_message is None
--- End diff --
Here is the logic I used:
`_pyarrow_requirement_message` contains error message for PyArrow
requirement if missing or version is not matched.
if `_pyarrow_requirement_message` contains the message, `_have_pyarrow`
becomes `False`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
