ueshin commented on a change in pull request #33497:
URL: https://github.com/apache/spark/pull/33497#discussion_r675816428
##########
File path: python/pyspark/pandas/data_type_ops/categorical_ops.py
##########
@@ -118,7 +121,42 @@ def _compare(
# Check if categoricals have the same dtype, same categories, and same
ordered
if hash(left.dtype) != hash(right.dtype):
raise TypeError("Categoricals can only be compared if 'categories'
are the same.")
- return column_op(f)(left, right)
+ if cast(CategoricalDtype, left.dtype).ordered:
+ return column_op(f)(left, right)
+ else:
+
+ def map_scol_to_cat(index_op, scol):
+ categories = cast(CategoricalDtype, index_op.dtype).categories
+ kvs = chain(
+ *[(SF.lit(code), SF.lit(category)) for code, category in
enumerate(categories)]
+ )
+ map_scol = F.create_map(*kvs)
+ return map_scol.getItem(scol)
Review comment:
As this part seems the same as the preparation of `astype`, could you
extract a method and reuse it?
##########
File path: python/pyspark/pandas/data_type_ops/categorical_ops.py
##########
@@ -118,7 +121,42 @@ def _compare(
# Check if categoricals have the same dtype, same categories, and same
ordered
if hash(left.dtype) != hash(right.dtype):
raise TypeError("Categoricals can only be compared if 'categories'
are the same.")
- return column_op(f)(left, right)
+ if cast(CategoricalDtype, left.dtype).ordered:
+ return column_op(f)(left, right)
+ else:
+
+ def map_scol_to_cat(index_op, scol):
+ categories = cast(CategoricalDtype, index_op.dtype).categories
+ kvs = chain(
+ *[(SF.lit(code), SF.lit(category)) for code, category in
enumerate(categories)]
+ )
+ map_scol = F.create_map(*kvs)
+ return map_scol.getItem(scol)
Review comment:
Then, I guess we can just:
```py
return column_op(f)(_to_cat(left), _to_cat(right))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
