[
https://issues.apache.org/jira/browse/SPARK-58013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18098955#comment-18098955
]
Vivek Gangavarapu commented on SPARK-58013:
-------------------------------------------
I ran the pandas-on-Spark test suite against pandas 3.0.5 and hit a single
failure: \{{test_frame_loc_setitem}} (subcase \{{op1}}, \{{check_ser=True}}) in
\{{test_indexing_loc.py}}.
The case is a scalar loc-assignment to a list of columns given in reverse
order, e.g.:
{code:python}
df.loc[["viper", "sidewinder"], ["shield", "max_speed"]] = 10
{code}
pandas 3 sets both columns to 10, but pandas-on-Spark silently drops the write
and leaves the frame unchanged.
Root cause is in \{{LocIndexerLike.setitem}} (\{{indexing.py}}). The CoW branch
added in SPARK-55296 returns early (no-op) when the selected columns aren't in
the frame's internal order, assuming that matches pandas 3. For a scalar
assignment column order doesn't matter, and pandas 3 does apply the write — so
the guard just drops valid writes. The CoW view-decoupling it was meant to
protect is already handled by the normal path (the same test with natural
column order passes).
Fix: remove that early-return so both orderings go through the normal path.
It's guarded behind \{{pandas >= 3.0.0}}, so pandas 2 is unaffected. After the
change \{{test_frame_loc_setitem}} passes and the full \{{indexes}} suite is
green with no regressions.
Will open a PR shortly.
> Support pandas 3 in pandas-on-Spark tests
> -----------------------------------------
>
> Key: SPARK-58013
> URL: https://issues.apache.org/jira/browse/SPARK-58013
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
