Yikun commented on a change in pull request #32821:
URL: https://github.com/apache/spark/pull/32821#discussion_r654208604
##########
File path: python/pyspark/pandas/data_type_ops/base.py
##########
@@ -300,6 +303,13 @@ def prepare(self, col: pd.Series) -> pd.Series:
"""Prepare column when from_pandas."""
return col.replace({np.nan: None})
+ def isnull(self, index_ops: Union["Index", "Series"]) -> Union["Series",
"Index"]:
+ from pyspark.pandas.indexes import MultiIndex
+
+ if isinstance(index_ops, MultiIndex):
+ raise NotImplementedError("isna is not defined for MultiIndex")
+ return index_ops._with_new_scol(index_ops.spark.column.isNull())
Review comment:
Emm, it fails on
[BooleanExtensionOpsTest.test_isnull](https://github.com/apache/spark/pull/32821/files/bca1d84b288a166e367ff0785b298808c670b17b#diff-6beed41b484721fbc29caef1b12028fb7808d36d7a2ce5d4e938b0d5c3f81f2dR600-R602)
with dtype mismatch:
```
Left(pser):
0 False
1 False
2 True
dtype: bool
bool
Right(psser):
0 False
1 False
2 True
dtype: boolean
boolean
```
But if we don't specify the field, it works. Looks like we should just keep
it with the default field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
