fangchenli commented on PR #58765:
URL: https://github.com/apache/spark/pull/58765#issuecomment-6025519039
> 1. There's a mypy failure - `f"datetime64[{unit}]"` is not guaranteed to
be valid (even though logically it probably is). We can probably ignore here,
or we can check `unit` and assign literals to this argument.
> 2. Is `ArrowDtype` timestamp commonly used now? This is a breaking change
I think. I mean our timestamp localization is a mess for naive timestamps. But
maybe this is not too bad.
> 3. Do we care about `us` vs `ns` thing? It bothers me a bit - the unit is
different but the result is the same - is it intentional?
> 4. We used the str interpolation for the result comparision -
`self.assertEqual(str(arrow_out.iloc[0]), str(numpy_out.iloc[0]))` why is that?
1. Changed to np dtype object.
2. Probably not at this moment, since it's relatively new. But as the Arrow
backend matures, I'd expect it'll gain more popularity. The change introduced
here is a breaking change and also a bug fix. Now the np path and arrow path
align.
3. Intentional. This function only normalizes the timezone; each path keeps
the column's unit, and the conversion to Spark's `timestamp[us]` happens
afterward, when the Series is converted to the requested Arrow type. The test
pairs a `us` ArrowDtype column (the unit Spark sends to Python) with an
explicitly `ns` numpy column to cover the mixed-unit case; they compare equal
because `pd.Timestamp` equality compares the moment in time, not the unit.
Normalizing to `ns` would also reintroduce the 1677-2262 range limit. One
pre-existing caveat: on pyarrow 18, `ArrowExtensionArray.__arrow_array__`
ignores the requested type, so an ArrowDtype column with `ns` unit is not cast
to `us`. I'll address that in a separate PR.(drafted by Claude)
4. Fixed. It was a residual from Claude.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
