sohailshk opened a new pull request, #55701:
URL: https://github.com/apache/arrow/pull/55701

   ### Rationale for this change
   
   The pandas metadata written by `Table.from_pandas` can describe a column as 
a string column (`numpy_type: "str"` on pandas 3, `"string"` for the string 
dtype on pandas 2). `to_pandas` applied the string dtype from the metadata 
without looking at the Arrow type, so a column replaced with `Table.set_column` 
by an integer, floating point or boolean one came back as strings (plain `str` 
columns on pandas 3, any non-string replacement type):
   
   ```python
   table = pa.Table.from_pandas(pd.DataFrame({"a": [1, 2, 3], "b": ["x", "y", 
"z"]}))
   table.set_column(1, "b", pa.array([1, 2, 3])).to_pandas()["b"].tolist()   # 
['1', '2', '3']
   ```
   
   ### What changes are included in this PR?
   
   In `_get_extension_dtypes` (`pandas_compat.py`), the existing check that 
skips the string dtype for dictionary columns is generalised: the string dtype 
from the metadata is only used when the Arrow column is a string, large string, 
string view, binary, large binary, binary view or null type (the string dtype 
decodes binary, so a string column written with a binary schema keeps reading 
back as strings). Dictionary columns stay skipped.
   
   ### Are these changes tested?
   
   Yes. `test_to_pandas_replaced_string_column_with_stale_metadata` (int64, 
double, boolean, timestamp and decimal replacements, for `str` and `string` 
columns) and `test_from_pandas_string_column_with_binary_schema` are new. On 
main the first one fails (10 failed on pandas 3.1.1, 5 on pandas 2.3.3 where 
plain object strings are not affected); with the change everything passes (14 
passed on 3.1.1, 12 passed and 2 skipped on 2.3.3). `test_pandas.py`, 
`parquet/test_pandas.py` and `test_feather.py` pass with the change except 
`test_dictionary_from_pandas_specified_type`, which also fails on main (pandas 
3.1.1: 518 passed, 1 failed; pandas 2.3.3: 511 passed, 1 failed).
   
   ### Are there any user-facing changes?
   
   Yes: `to_pandas()` returns the replaced column with its own dtype instead of 
strings.
   
   **This PR contains a "Critical Fix".** Numeric values were silently 
converted to strings.
   
   ### Was AI used for this PR?
   
   The implementation and tests were written with AI assistance.
   
   **PR code and description written by:**
   - [ ] Human
   - [ ] AI
   
   **Reviewed before submission by:**
   - [ ] Human
   - [ ] AI
   - [ ] Not reviewed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to