jorisvandenbossche commented on issue #37291: URL: https://github.com/apache/arrow/issues/37291#issuecomment-1687719702
So this is actually documented in the pandas' source code: In the Timedelta constructor (https://github.com/pandas-dev/pandas/blob/43691a2f5d235b08f0f3aa813d8fdcb7c4ce1e47/pandas/_libs/tslibs/timedeltas.pyx#L953-L964): ``` # For millisecond and second resos, we cannot actually pass int(value) because # many cases would fall outside of the pytimedelta implementation bounds. # We pass 0 instead, and override seconds, microseconds, days. # In principle we could pass 0 for ns and us too. if reso == NPY_FR_ns: td_base = _Timedelta.__new__(cls, microseconds=int(value) // 1000) elif reso == NPY_DATETIMEUNIT.NPY_FR_us: td_base = _Timedelta.__new__(cls, microseconds=int(value)) elif reso == NPY_DATETIMEUNIT.NPY_FR_ms: td_base = _Timedelta.__new__(cls, milliseconds=0) elif reso == NPY_DATETIMEUNIT.NPY_FR_s: td_base = _Timedelta.__new__(cls, seconds=0) ``` and then the relevant note in the seconds attribute (https://github.com/pandas-dev/pandas/blob/43691a2f5d235b08f0f3aa813d8fdcb7c4ce1e47/pandas/_libs/tslibs/timedeltas.pyx#L1103-L1104): ``` # NB: using the python C-API PyDateTime_DELTA_GET_SECONDS will fail # (or be incorrect) ``` So pyarrow is exactly using those C APIs, which are now silently incorrect for second and millisecond resolution Timedeltas. While pyarrow should certainly adapt to properly support pandas.Timedelta (especially for cases outside the range of datetime.timedelta), I think it might be safer for pandas to ensure those C APIs keep working (I think pyarrow will certainly not be the only one using those). It seems possible to me to at least set those for the cases where they are within the range of datetime.timedelta. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
