[
https://issues.apache.org/jira/browse/SPARK-27759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
colin fang updated SPARK-27759:
-------------------------------
Description:
{code:java}
pd_df = pd.DataFrame({'x': np.random.rand(11, 3, 5).tolist()})
df = spark.createDataFrame(pd_df).cache()
{code}
Each element in x is a list of list, as expected.
{code:java}
df.toPandas()['x']
# 0 [[0.08669612955959993, 0.32624430522634495, 0....
# 1 [[0.29838166086156914, 0.008550172904516762, 0...
# 2 [[0.641304534802928, 0.2392047548381877, 0.555...
{code}
{code:java}
def my_udf(x):
# Hack to see what's inside a udf
raise Exception(x.values.shape, x.values[0].shape, x.values[0][0].shape,
np.stack(x.values).shape)
return pd.Series(x.values)
my_udf = F.pandas_udf(my_udf, returnType=DoubleType())
df.coalesce(1).withColumn('y', my_udf('x')).show(
# Exception: ((11,), (3,), (5,), (11, 3)){code}
A batch (11) of `x` is converted to pd.Series, however, each element in the
pd.Series is now a numpy 1d array of numpy 1d array. It is inconvenient to work
with nested 1d numpy array in practice in a udf.
For example, I need a ndarray of shape (11, 3, 5) in udf, so that I can make
use of the numpy vectorized operations. If I was given a list of list intact, I
can simply do `np.stack(x.values)`. However, it doesn't work here as what I
received is a nested numpy 1d array.
was:
{code:java}
pd_df = pd.DataFrame(\{'x': np.random.rand(11, 3, 5).tolist()})
df = spark.createDataFrame(pd_df).cache()
{code}
Each element in x is a list of list, as expected.
{code:java}
df.toPandas()['x']
# 0 [[0.08669612955959993, 0.32624430522634495, 0....
# 1 [[0.29838166086156914, 0.008550172904516762, 0...
# 2 [[0.641304534802928, 0.2392047548381877, 0.555...
{code}
{code:java}
def my_udf(x):
# Hack to see what's inside a udf
raise Exception(x.values.shape, x.values[0].shape, x.values[0][0].shape,
np.stack(x.values).shape)
return pd.Series(x.values)
my_udf = pandas_udf(dot_product, returnType=DoubleType())
df.withColumn('y', my_udf('x')).show()
Exception: ((2,), (3,), (5,), (2, 3))
{code}
A batch (2) of `x` is converted to pd.Series, however, each element in the
pd.Series is now a numpy 1d array of numpy 1d array. It is inconvenient to work
with nested 1d numpy array in practice in a udf.
For example, I need a ndarray of shape (2, 3, 5) in udf, so that I can make use
of the numpy vectorized operations. If I was given a list of list intact, I can
simply do `np.stack(x.values)`. However, it doesn't work here as what I
received is a nested numpy 1d array.
> Do not auto cast array<double> to np.array in vectorized udf
> ------------------------------------------------------------
>
> Key: SPARK-27759
> URL: https://issues.apache.org/jira/browse/SPARK-27759
> Project: Spark
> Issue Type: Improvement
> Components: PySpark, SQL
> Affects Versions: 3.0.0
> Reporter: colin fang
> Priority: Minor
>
> {code:java}
> pd_df = pd.DataFrame({'x': np.random.rand(11, 3, 5).tolist()})
> df = spark.createDataFrame(pd_df).cache()
> {code}
> Each element in x is a list of list, as expected.
> {code:java}
> df.toPandas()['x']
> # 0 [[0.08669612955959993, 0.32624430522634495, 0....
> # 1 [[0.29838166086156914, 0.008550172904516762, 0...
> # 2 [[0.641304534802928, 0.2392047548381877, 0.555...
> {code}
>
> {code:java}
> def my_udf(x):
> # Hack to see what's inside a udf
> raise Exception(x.values.shape, x.values[0].shape, x.values[0][0].shape,
> np.stack(x.values).shape)
> return pd.Series(x.values)
> my_udf = F.pandas_udf(my_udf, returnType=DoubleType())
> df.coalesce(1).withColumn('y', my_udf('x')).show(
> # Exception: ((11,), (3,), (5,), (11, 3)){code}
>
> A batch (11) of `x` is converted to pd.Series, however, each element in the
> pd.Series is now a numpy 1d array of numpy 1d array. It is inconvenient to
> work with nested 1d numpy array in practice in a udf.
>
> For example, I need a ndarray of shape (11, 3, 5) in udf, so that I can make
> use of the numpy vectorized operations. If I was given a list of list intact,
> I can simply do `np.stack(x.values)`. However, it doesn't work here as what I
> received is a nested numpy 1d array.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
