[
https://issues.apache.org/jira/browse/SPARK-39405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xinrong Meng updated SPARK-39405:
---------------------------------
Description:
NumPy is the fundamental package for scientific computing with Python. It is
very commonly used, especially in the data science world. For example, Pandas
is backed by NumPy, and Tensors also supports interchangeable conversion
from/to NumPy arrays.
However, PySpark only supports Python native types with the exception of pandas
data in “SparkSession.createDataFrame(pandas.DataFrame)”.
“DataFrame.toPandas()”. NumPy input support has been requested multiple times,
see SPARK-2012, SPARK-37697, SPARK-31776, and SPARK-6857.
The ticket proposes to support NumPy input in PySpark.
See more design
[design|https://docs.google.com/document/d/e/2PACX-1vTD70k1_Ohd2QCY-dbVrqDTmyfmlyDgxA-0SVl0DcDyeePzc9JiaIjG9B_FFI2_MfGGyjPrWYCTXk5F/pub].
was:
NumPy is the fundamental package for scientific computing with Python. It is
very commonly used, especially in the data science world. For example, Pandas
is backed by NumPy, and Tensors also supports interchangeable conversion
from/to NumPy arrays.
However, PySpark only supports Python built-in types with the exception of
“SparkSession.createDataFrame(pandas.DataFrame)” and “DataFrame.toPandas”.
This issue has been raised multiple times internally and externally, see also
SPARK-2012, SPARK-37697, SPARK-31776, and SPARK-6857.
With the NumPy support in SQL, we expect more adaptations from naive data
scientists and newcomers leveraging their existing background and codebase with
NumPy.
See more
[https://docs.google.com/document/d/1WsBiHoQB3UWERP47C47n_frffxZ9YIoGRwXSwIeMank/edit#]
.
> NumPy Input Support in PySpark
> ------------------------------
>
> Key: SPARK-39405
> URL: https://issues.apache.org/jira/browse/SPARK-39405
> Project: Spark
> Issue Type: Umbrella
> Components: PySpark
> Affects Versions: 3.4.0
> Reporter: Xinrong Meng
> Assignee: Xinrong Meng
> Priority: Major
>
> NumPy is the fundamental package for scientific computing with Python. It is
> very commonly used, especially in the data science world. For example, Pandas
> is backed by NumPy, and Tensors also supports interchangeable conversion
> from/to NumPy arrays.
> However, PySpark only supports Python native types with the exception of
> pandas data in “SparkSession.createDataFrame(pandas.DataFrame)”.
> “DataFrame.toPandas()”. NumPy input support has been requested multiple
> times, see SPARK-2012, SPARK-37697, SPARK-31776, and SPARK-6857.
> The ticket proposes to support NumPy input in PySpark.
>
> See more design
> [design|https://docs.google.com/document/d/e/2PACX-1vTD70k1_Ohd2QCY-dbVrqDTmyfmlyDgxA-0SVl0DcDyeePzc9JiaIjG9B_FFI2_MfGGyjPrWYCTXk5F/pub].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
