hzmsh commented on PR #59740: URL: https://github.com/apache/spark/pull/59740#issuecomment-6356008253
Hi @zhengruifeng @gaogaotiantian, could you take a look when you have a moment? This follows [SPARK-55843](https://issues.apache.org/jira/browse/SPARK-55843) ([#54629](https://github.com/apache/spark/pull/54629)) and [SPARK-56310](https://issues.apache.org/jira/browse/SPARK-56310) ([#55118](https://github.com/apache/spark/pull/55118)), which moved `toPandas()` timestamps to `datetime64[us]` on pandas 3. One question where I wasn't sure what you'd prefer: pandas UDF / `applyInPandas` / `mapInPandas` inputs still coerce timestamps to nanoseconds on pandas 3 (`preprocess_time` and `ArrowToPandasConversion`). I left them out because changing them changes the dtype UDFs receive. fangchenli also noted in [#54143](https://github.com/apache/spark/pull/54143#discussion_r2927397588) that `preprocess_time` overflows for out-of-range timestamps, and left it out of that PR as out of scope. Would you like a follow-up for those, or is that intended for now? (The pandas < 3 question, about a clearer error for out-of-range timestamps, is on the [JIRA](https://issues.apache.org/jira/browse/SPARK-60499?focusedCommentId=18130657&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-18130657).) I'll follow whichever you think is best. Thanks for your time! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
