emkornfield commented on issue #14229: URL: https://github.com/apache/arrow/issues/14229#issuecomment-1281840773
I commented on the JIRA for longer term fixes. I think if you want to save all of this data as one parquet file, the path that should round trip is if you control the conversion from [Pandas to Arrow](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html#pyarrow.Table.from_pandas) and use the LargeList type instead of List for columns that contain lists. The Arrow Schema is persisted in the Parquet file and is then used to read it back, which will allow for 64-bit instead of 32-bit indices. It would be nice to be able to set Large types as a flag or infer large types are necessary based on Parquet statistics (I noted these in the JIRA). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
