xinrong-databricks commented on pull request #32821: URL: https://github.com/apache/spark/pull/32821#issuecomment-857848384
Certainly. :) In the `isnull ` method, https://github.com/apache/spark/blob/master/python/pyspark/pandas/base.py#L987-L990 has data type checks. We aim to remove these data type checks, with the help of DataTypeOps. We aim to rewrite `isnull` method as below: ```py def isnull(self) -> Union["Series", "Index"]: self._dtype_op.isnull(self) ``` So we might define `isnull` in DataTypeOps, and its subclasses if needed. Now DecimalType, DoubleType, and FloatType data share the FractionalOps class, but DecimalType behaves differently from FloatType and DoubleType (as https://github.com/apache/spark/blob/master/python/pyspark/pandas/base.py#L987-L990), so we propose to introduce DecimalOps. The behavior difference is caused by `DecimalType could not have NaN`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
