Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/20839
### After Reworded Warnings
```
In [2]: spark.createDataFrame(pd.DataFrame([[{u'a': 1}]]), "a: map<string,
int>")
/home/bryan/git/spark/python/pyspark/sql/session.py:688: UserWarning:
createDataFrame attempted Arrow optimization because
'spark.sql.execution.arrow.enabled' is set to true, but has reached the error
below and will not continue because automatic fallback with
'spark.sql.execution.arrow.fallback.enabled' has been set to false.
PyArrow >= 0.8.0 must be installed; however, it was not found.
warnings.warn(msg)
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-2-ecc28a9b5e18> in <module>()
----> 1 spark.createDataFrame(pd.DataFrame([[{u'a': 1}]]), "a: map<string,
int>")
~/git/spark/python/pyspark/sql/session.py in createDataFrame(self, data,
schema, samplingRatio, verifySchema)
665 and len(data) > 0:
666 try:
--> 667 return
self._create_from_pandas_with_arrow(data, schema, timezone)
668 except Exception as e:
669 from pyspark.util import _exception_message
~/git/spark/python/pyspark/sql/session.py in
_create_from_pandas_with_arrow(self, pdf, schema, timezone)
508
509 require_minimum_pandas_version()
--> 510 require_minimum_pyarrow_version()
511
512 from pandas.api.types import is_datetime64_dtype,
is_datetime64tz_dtype
~/git/spark/python/pyspark/sql/utils.py in require_minimum_pyarrow_version()
147 if not have_arrow:
148 raise ImportError("PyArrow >= %s must be installed;
however, "
--> 149 "it was not found." %
minimum_pyarrow_version)
150 if LooseVersion(pyarrow.__version__) <
LooseVersion(minimum_pyarrow_version):
151 raise ImportError("PyArrow >= %s must be installed;
however, "
ImportError: PyArrow >= 0.8.0 must be installed; however, it was not found.
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
