spark-test commented on code in PR #60092:
URL: https://github.com/apache/spark/pull/60092#discussion_r4500229845
##########
python/pyspark/pandas/series.py:
##########
@@ -447,7 +447,7 @@ def __init__( # type: ignore[no-untyped-def]
if name:
data = data.rename(name)
- if index:
+ if index is not None:
Review Comment:
**Blocking (P1):** With pandas-on-Spark Series data, this guard now also
lets through a pandas-on-Spark `Index`, a pandas `Index` or `Series`, and a
NumPy array with more than one element; all of them raised `ValueError` in the
old truth test. `Series.reindex` handles them with a right outer join on index
values, with no identical-axis shortcut and no duplicate-label check, so each
target label returns every data row that carries it. When the Series' index
repeats a label, rows are silently multiplied:
```python
psser = ps.from_pandas(pd.Series([1, 2, 3], index=[10, 10, 20], name="x"))
ps.Series(psser, index=psser.index)
# 5 rows: (10, 1), (10, 2), (10, 1), (10, 2), (20, 3)
```
`index=psser.index.to_pandas()` and `index=np.array([10, 10, 20])` return
the same five rows. pandas returns the original three rows whenever the target
equals the Series' index, and raises `cannot reindex on an axis with duplicate
labels` for any other target. This constructor documents non-unique index
values as allowed, and the new tests only use unique labels, so this goes
undetected.
See **Shared repair plan 1** in the review body.
##########
python/pyspark/pandas/series.py:
##########
@@ -447,7 +447,7 @@ def __init__( # type: ignore[no-untyped-def]
if name:
data = data.rename(name)
- if index:
+ if index is not None:
Review Comment:
**Non-blocking (P2):** This guard now also accepts a pandas `Index` or
`Series` as `index`; both raised `ValueError` in the old truth test.
`DataFrame._reindex_index` rebuilds any target that is not a pandas-on-Spark
Index as `ps.Index(list(index))` and keeps the source Series' index names, so
the argument's own name and dtype are dropped:
```python
psser = ps.from_pandas(
pd.Series([1, 2, 3], index=pd.Index([10, 20, 30], name="src"), name="x")
)
ps.Series(psser, index=pd.Index([30, 10], dtype="int32", name="k"))
# the index comes back named 'src' with dtype int64; pandas gives 'k' with
int32
```
A named pandas `Series` argument such as `pd.Series([30, 10], name="s")`
loses its name the same way, and an unnamed pandas `Index` over a named source
index keeps the source name, where pandas drops it. Over a unique index the
rows and values are right, but the index metadata no longer matches pandas,
which changes later name-based results such as the columns that `reset_index()`
produces. The new tests only pass pandas-on-Spark Index and list arguments with
an unnamed source index, so this is not covered.
**Verification:**
- **Regression:** Constructing a Series from a pandas-on-Spark Series with a
named source index and a named int32 pandas Index argument yields the pandas
index name, dtype, rows, and values.
- **Regression:** Passing a named pandas Series as the index argument yields
an index named after that Series, matching pandas.
- **Compatibility:** pandas-on-Spark Index, list, and empty-list index
arguments still match pandas after the conversion is added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
