HyukjinKwon commented on a change in pull request #34114:
URL: https://github.com/apache/spark/pull/34114#discussion_r718157230
##########
File path: python/pyspark/pandas/base.py
##########
@@ -394,7 +394,17 @@ def __abs__(self: IndexOpsLike) -> IndexOpsLike:
# comparison operators
def __eq__(self, other: Any) -> SeriesOrIndex: # type: ignore[override]
- return self._dtype_op.eq(self, other)
+ if isinstance(other, (list, tuple)):
+ if len(self) != len(other):
+ raise ValueError("Lengths must be equal")
+ name = self._internal.spark_column_name_for(self.spark.column)
+ other = ps.Series(other, name=name)
Review comment:
Can we do this via `transform(array_sort(collect_list( ... index and
column ...)), lambda c: c.getItem[0]) == functions.lit(other)`? `other` is in
memory so the size of the list is expected to be small. I think this is fine to
collect all values of the column into single value from `collect_list`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
