xinrong-databricks edited a comment on pull request #32821: URL: https://github.com/apache/spark/pull/32821#issuecomment-857848384
Certainly. In the `isnull ` method, https://github.com/apache/spark/blob/master/python/pyspark/pandas/base.py#L987-L990 has data type checks. We aim to remove these data type checks, with the help of DataTypeOps. We aim to rewrite `isnull` method as below: ```py def isnull(self) -> Union["Series", "Index"]: return self._dtype_op.isnull(self) ``` So we might define `isnull` in DataTypeOps, and its subclasses if needed. Now DecimalType, DoubleType, and FloatType data share the FractionalOps class, but DecimalType behaves differently from FloatType and DoubleType (as https://github.com/apache/spark/blob/master/python/pyspark/pandas/base.py#L987-L990), so we propose to introduce DecimalOps. The behavior difference here is caused by `DecimalType could not have NaN`. Feel free to adjust the PR description for a better understanding. :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
