S-h-u-b-h-1 opened a new pull request, #55504: URL: https://github.com/apache/arrow/pull/55504
### Rationale for this change Fixes #55490. Dictionary-encoded timestamp columns currently lose their timezone when converted to pandas categoricals. Comparisons against timezone-aware timestamps can then silently return incorrect results. ### What changes are included in this PR? - Carry the dictionary timestamp timezone through the C++ categorical conversion metadata. - Reconstruct timezone-aware categories in a shared helper used by Array/ChunkedArray and Table conversion, retaining the timestamp unit and interpreting values as instants. - Leave object-valued and timezone-naive categories unchanged. - Cover units, non-UTC zones, null/empty inputs, independently encoded chunks, zero chunks, ordered categories and conversion options. - Remove the existing fully-loaded DataFrame test's workaround that expected categorical timezone loss. ### Are these changes tested? Built native Arrow C++ 27.0.0-SNAPSHOT and PyArrow from base `976779b2aac587f51b87c3bd2e6ae4c36f75ab6c` plus this patch on Linux, Python 3.12, GCC 14.2 and NumPy 2.5.4. - New regressions: 41 passed with pandas 3.1.1 and 41 passed with pandas 2.3.3. - Full `test_pandas.py`: 426 passed / 11 skipped with pandas 3.1.1; 421 passed / 16 skipped with pandas 2.3.3. - Dictionary/categorical-selected `test_array.py` and `test_table.py`: 28 passed / 1 skipped. - Additional negative-epoch and DST-boundary smoke checks passed. - `git diff --check` passed. The new regressions produced 28 failures and 13 passes against released PyArrow 26.0.1 before the fix. The full repository suite and macOS/Windows builds have not been run. Optional integrations were not built. ### Are there any user-facing changes? Dictionary-encoded timezone-aware timestamps retain timezone-aware pandas categories instead of becoming naive. The reported UTC comparison now returns `[True, True, False]`. No public API signatures change. **This PR contains a "Critical Fix".** It addresses silently incorrect timezone information and comparison results. ### Was AI used for this PR? **PR code and description written by:** - [ ] Human - [x] AI **Reviewed before submission by:** - [ ] Human - [x] AI - [ ] Not reviewed AI assisted with the implementation, regression tests, PR text, review and local validation. This is a draft pending the contributor's own review and understanding of the changes; human review is not claimed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
