[
https://issues.apache.org/jira/browse/SPARK-29875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun reassigned SPARK-29875:
-------------------------------------
Assignee: Hyukjin Kwon
> Avoid to use deprecated pyarrow.open_stream API in Spark 2.4.x
> --------------------------------------------------------------
>
> Key: SPARK-29875
> URL: https://issues.apache.org/jira/browse/SPARK-29875
> Project: Spark
> Issue Type: Bug
> Components: PySpark, SQL
> Affects Versions: 2.4.4
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Minor
>
> In Spark 2.4.x, if we use PyArrow higher then 0.12.0, it shows a bunch of
> warnings as below:
> {code}
> from pyspark.sql.functions import pandas_udf, PandasUDFType
> @pandas_udf("integer", PandasUDFType.SCALAR) # doctest: +SKIP
> def add_one(x):
> return x + 1
> spark.range(100).select(add_one("id")).collect()
> {code}
> {code}
> UserWarning: pyarrow.open_stream is deprecated, please use
> pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning:
> pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning:
> pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning:
> pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning:
> pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning:
> pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning:
> pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning:
> pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning:
> pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning:
> pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning:
> pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> /usr/local/lib/python3.7/site-packages/pyarrow/__init__.py:157: UserWarning:
> pyarrow.open_stream is deprecated, please use pyarrow.ipc.open_stream
> warnings.warn("pyarrow.open_stream is deprecated, please use "
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
