sohailshk opened a new pull request, #55678:
URL: https://github.com/apache/arrow/pull/55678

   ### Rationale for this change
   
   `Table.from_pandas` stores the extension dtype of each index level in the 
pandas metadata (for example `numpy_type: "Int64"`), but `to_pandas` only 
applies it to data columns. Index levels come back with the default dtype for 
the Arrow type, and a nullable integer index with missing values becomes a 
`float64` index:
   
   ```python
   df = pd.DataFrame({"a": [1, 2, 3]}, index=pd.Index(pd.array([1, None, 3], 
dtype="Int64")))
   pa.Table.from_pandas(df).to_pandas().index   # Index([1.0, nan, 3.0], 
dtype='float64')
   ```
   
   ### What changes are included in this PR?
   
   `_extract_index_level` in `pandas_compat.py` now converts an index level 
with the extension dtype recorded in the metadata when that dtype implements 
`__from_arrow__`, through the dtype's `__from_arrow__`, so the level is 
converted once. A user-supplied `types_mapper` still takes precedence, as it 
does for data columns, and dictionary (categorical) levels are left to the 
existing path.
   
   ### Are these changes tested?
   
   Yes. `test_roundtrip_index_with_extension_dtype` (single index and as a 
MultiIndex level) covers `Int64`, `Float32`, `boolean`, `string`, `UInt8` and 
`int64[pyarrow]`; it fails on main for all six and passes with the change. 
`test_pandas.py`, `parquet/`, `interchange/` and `test_feather.py` otherwise 
give identical results with and without the change, on pandas 3.1.1 and 2.3.3.
   
   ### Are there any user-facing changes?
   
   Yes: `to_pandas()` (and `pd.read_parquet`) returns an index with the 
original nullable or Arrow-backed dtype instead of the default dtype for the 
Arrow type.
   
   **This PR contains a "Critical Fix".** A nullable integer index with missing 
values was converted to a float index, and integers above 2**53 lose precision 
in that conversion (`2**53 + 1` comes back as `2**53`).
   
   ### Was AI used for this PR?
   
   The implementation and tests were written with AI assistance.
   
   **PR code and description written by:**
   - [ ] Human
   - [ ] AI
   
   **Reviewed before submission by:**
   - [ ] Human
   - [ ] AI
   - [ ] Not reviewed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to