Tian Gao created SPARK-55294:
--------------------------------
Summary: Support pyarrow based data types
Key: SPARK-55294
URL: https://issues.apache.org/jira/browse/SPARK-55294
Project: Spark
Issue Type: Sub-task
Components: PySpark
Affects Versions: 4.2.0
Reporter: Tian Gao
Pandas 3 supports pyarrow based data types like `bool[pyarrow]`. We have no
code to work with it. The problem is if `pyarrow` is installed, `string` type
will automatically use pyarrow backend, which means the operations between
`string` types will result in pyarrow dtypes. e.g. if we compare two string
series, the result would be a `bool[pyarrow]` series.
We need some extra work to make this happen. We probable need to add more
arguments to `InternalField.from_struct_field` so pyspark knows if it should
convert the data back to a pyarrow based format.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
