Hyukjin Kwon created SPARK-35683:
------------------------------------
Summary: Fix Index.difference to avoid collect 'other' to driver
side
Key: SPARK-35683
URL: https://issues.apache.org/jira/browse/SPARK-35683
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 3.2.0
Reporter: Hyukjin Kwon
Assignee: Haejoon Lee
See:
https://github.com/databricks/koalas/pull/1325#discussion_r647889901
https://github.com/databricks/koalas/pull/1325#discussion_r647890007
{code}
midx1 = ps.MultiIndex.from_tuples([('a', 'x', 1), ('b', 'z', 2), ('k', 'z', 3)])
midx1.difference(idx1)
{code}
{code}
pyspark.pandas.exceptions.PandasNotImplementedError: The method
`pd.Index.__iter__()` is not implemented. If you want to collect your data as
an NumPy array, use 'to_numpy()' instead.
{code}
In addition, calling {{MultiIndex.from_tuples}} will result in collecting all
into driver side.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
