wjones127 commented on issue #15032: URL: https://github.com/apache/arrow/issues/15032#issuecomment-1368096718
`pyarrow.parquet` doesn't round trip all types right now. Right now, the only other example I know of is is dictionary types. They always come back with `int32` indices, regardless of the original index type. See also: https://lists.apache.org/thread/rv29cwf4208jh73s0gyrzpw5l87pf7pb `date64` type only exists for compatibility with systems that use milliseconds to represent dates. That representation doesn't exist in the Parquet format. It's also not a sensible representation of a date, because the logical resolution is a day, so the milliseconds information isn't used. But it looks like we handle this for nearly every other type, including `Large*` variants of string, binary, and list, different timestamp resolutions, and unsigned integers. So maybe it's worth fixing these last few types. <details> <summary>Python code to check type roundtripping</summary> ```python import pyarrow as pa import pyarrow.parquet as pq from decimal import Decimal def check_parquet_roundtrip(arr): tab = pa.table({"x": arr}) pq.write_table(tab, "test.parquet") schema = pq.read_schema("test.parquet") assert schema.field(0).type == arr.type # These fail check_parquet_roundtrip( pa.array([1, 2, 3], pa.date64()) ) check_parquet_roundtrip( pa.array(["a", "b"], pa.dictionary(pa.int8(), pa.string())) ) # All these work check_parquet_roundtrip( pa.array([1, 2, 3], pa.timestamp('ms')) ) check_parquet_roundtrip( pa.array([1, 2, 3], pa.timestamp('us')) ) check_parquet_roundtrip( pa.array(["a", "b"], pa.dictionary(pa.int32(), pa.string())) ) check_parquet_roundtrip( pa.array([Decimal("10.000")], pa.decimal128(19, 3)) ) check_parquet_roundtrip( pa.array([Decimal("10.000")], pa.decimal256(19, 3)) ) check_parquet_roundtrip( pa.array(["a", "b"], pa.large_string()) ) check_parquet_roundtrip( pa.array([["a", "b"]], pa.large_list(pa.large_string())) ) check_parquet_roundtrip( pa.array([["a", "b"]], pa.large_list(pa.large_string())) ) check_parquet_roundtrip( pa.array([1, 2, 3], pa.uint32()) ) ``` </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
