[
https://issues.apache.org/jira/browse/SPARK-29875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-29875.
-----------------------------------
Fix Version/s: 2.4.5
Resolution: Fixed
Issue resolved by pull request 26501
[https://github.com/apache/spark/pull/26501]
> Avoid to use deprecated pyarrow.open_stream API in Spark 2.4.x
> --------------------------------------------------------------
>
> Key: SPARK-29875
> URL: https://issues.apache.org/jira/browse/SPARK-29875
> Project: Spark
> Issue Type: Bug
> Components: PySpark, SQL
> Affects Versions: 2.4.4
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Minor
> Fix For: 2.4.5
>
>
> In Spark 2.4.x, if we use PyArrow higher then 0.12.0, it shows a bunch of
> warnings as below:
> {code}
> from pyspark.sql.functions import pandas_udf, PandasUDFType
> @pandas_udf("integer", PandasUDFType.SCALAR) # doctest: +SKIP
> def add_one(x):
> return x + 1
> spark.range(100).select(add_one("id")).collect()
> {code}
> {code}
> UserWarning: pyarrow.open_stream is deprecated, please use
> pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning:
> pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning:
> pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning:
> pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning:
> pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning:
> pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning:
> pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning:
> pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning:
> pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning:
> pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning:
> pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning:
> pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
