Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/20487#discussion_r165795538
  
    --- Diff: python/pyspark/sql/tests.py ---
    @@ -48,19 +48,26 @@
     else:
         import unittest
     
    -_have_pandas = False
    -_have_old_pandas = False
    +_pandas_requirement_message = None
     try:
    -    import pandas
    -    try:
    -        from pyspark.sql.utils import require_minimum_pandas_version
    -        require_minimum_pandas_version()
    -        _have_pandas = True
    -    except:
    -        _have_old_pandas = True
    -except:
    -    # No Pandas, but that's okay, we'll skip those tests
    -    pass
    +    from pyspark.sql.utils import require_minimum_pandas_version
    +    require_minimum_pandas_version()
    +except ImportError as e:
    +    from pyspark.util import _exception_message
    +    # If Pandas version requirement is not satisfied, skip related tests.
    +    _pandas_requirement_message = _exception_message(e)
    +
    +_pyarrow_requirement_message = None
    +try:
    +    from pyspark.sql.utils import require_minimum_pyarrow_version
    +    require_minimum_pyarrow_version()
    +except ImportError as e:
    +    from pyspark.util import _exception_message
    +    # If Arrow version requirement is not satisfied, skip related tests.
    +    _pyarrow_requirement_message = _exception_message(e)
    +
    +_have_pandas = _pandas_requirement_message is None
    +_have_pyarrow = _pyarrow_requirement_message is None
    --- End diff --
    
    Here is the logic I used:
    
    `_pyarrow_requirement_message` contains error message for PyArrow 
requirement if missing or version is not matched.
    
    if `_pyarrow_requirement_message` contains the message, `_have_pyarrow` 
becomes `False`.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to