SAMARTHJAISWAL opened a new pull request, #55702:
URL: https://github.com/apache/arrow/pull/55702

   ### Rationale for this change
   
   `Table.from_pandas` records the extension dtype of an index level in the 
pandas metadata (`numpy_type: "Int64"`), and `to_pandas` uses that for data 
columns but not for index levels. An index level is converted with 
`ChunkedArray.to_pandas`, which does not look at the metadata, so it comes back 
with the default dtype for its Arrow type. A nullable integer index with 
missing values becomes a float index, and integers above 2**53 lose precision.
   
   ### What changes are included in this PR?
   
   `_extract_index_level` now looks up the dtype stored for the level and, if 
it is a pandas extension dtype with `__from_arrow__`, builds the level with it. 
The rules follow `_get_extension_dtypes` for data columns: `types_mapper` and 
an Arrow extension type take priority over the metadata.
   
   Two cases keep the conversion without metadata:
   
   * The stored dtype is a `StringDtype` but the column is not a string column. 
This avoids giving the index the problem reported for data columns in #55700.
   * `__from_arrow__` raises for the column, for example after `set_column` 
replaced an `Int64` index column with strings. Index reconstruction is already 
lenient about metadata that no longer matches the table, so this falls back 
instead of raising. I can make it raise, as it does for data columns, if that 
is preferred.
   
   ### Are these changes tested?
   
   Yes. New tests in `test_pandas.py` round-trip an index of `Int64` (also with 
a value above 2**53), `UInt8`, `Float64`, `boolean`, `string` and 
`int64[pyarrow]`, a MultiIndex with `Int64` and `boolean` levels, and a table 
whose index column was replaced by one of another type.
   
   I ran the new tests and `pyarrow.tests.test_pandas` with the change applied 
to pyarrow 26.0.1, on pandas 3.1.1 and pandas 2.3.3: 9 of the 11 new tests fail 
before the change and all pass after it, and no existing test changes result. I 
did not build Arrow from source.
   
   ### Are there any user-facing changes?
   
   Yes. An index level whose pandas extension dtype is stored in the metadata 
now comes back with that dtype from `Table.to_pandas` (and so from 
`pd.read_parquet`) instead of the default dtype for its Arrow type. For example 
an `Int64` index with missing values is no longer returned as `float64`. 
`ignore_metadata=True` gives the previous result.
   
   ### Was AI used for this PR?
   
   **PR code and description written by:**
   
   - [ ] Human
   - [x] AI
   
   **Reviewed before submission by:**
   
   - [x] Human
   - [ ] AI
   - [ ] Not reviewed
   
   The code, tests and description were written with Claude Code. I read the 
change and ran the tests described above.
   
   * GitHub Issue: #55677
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to