[
https://issues.apache.org/jira/browse/SPARK-27759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-27759:
----------------------------------
Affects Version/s: (was: 3.0.0)
3.1.0
> Do not auto cast array<double> to np.array in vectorized udf
> ------------------------------------------------------------
>
> Key: SPARK-27759
> URL: https://issues.apache.org/jira/browse/SPARK-27759
> Project: Spark
> Issue Type: Improvement
> Components: PySpark, SQL
> Affects Versions: 3.1.0
> Reporter: colin fang
> Priority: Minor
>
> {code:java}
> pd_df = pd.DataFrame({'x': np.random.rand(11, 3, 5).tolist()})
> df = spark.createDataFrame(pd_df).cache()
> {code}
> Each element in x is a list of list, as expected.
> {code:java}
> df.toPandas()['x']
> # 0 [[0.08669612955959993, 0.32624430522634495, 0....
> # 1 [[0.29838166086156914, 0.008550172904516762, 0...
> # 2 [[0.641304534802928, 0.2392047548381877, 0.555...
> {code}
>
> {code:java}
> def my_udf(x):
> # Hack to see what's inside a udf
> raise Exception(x.values.shape, x.values[0].shape, x.values[0][0].shape,
> np.stack(x.values).shape)
> return pd.Series(x.values)
> my_udf = F.pandas_udf(my_udf, returnType=DoubleType())
> df.coalesce(1).withColumn('y', my_udf('x')).show(
> # Exception: ((11,), (3,), (5,), (11, 3)){code}
>
> A batch (11) of `x` is converted to pd.Series, however, each element in the
> pd.Series is now a numpy 1d array of numpy 1d array. It is inconvenient to
> work with nested 1d numpy array in practice in a udf.
>
> For example, I need a ndarray of shape (11, 3, 5) in udf, so that I can make
> use of the numpy vectorized operations. If I was given a list of list intact,
> I can simply do `np.stack(x.values)`. However, it doesn't work here as what I
> received is a nested numpy 1d array.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
