Spenserrrr commented on PR #58263: URL: https://github.com/apache/spark/pull/58263#issuecomment-5407772875
> Hi @zhengruifeng @Yicong-Huang! This PR refines the comment in numpy signbit mapping. Could you take a look when you have time? Also, when fixing the comment, I found a way to propagate the null for Float64/32, and I've written the finding in the PR description. Do you think we should pursue the fix? Thanks! Correction on my note above: I said I'd found a way to propagate the null for Float64/Float32. That doesn't hold — I've updated the PR description. DataTypeOps.prepare replaces NaN with None per column on from_pandas (data_type_ops/base.py:549), which is a no-op on a nullable column until one <NA> appears, at which point the column is downcast to object and the unmasked NaNs are counted as missing too. So whether a NaN survives depends on whether an unrelated row in the column is missing, and propagating by dtype would give different answers for the same value depending on that. Today's behavior (treat every floating null as a NaN) is the predictable one. So there is nothing to pursue further for this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
