Fokko commented on PR #36846: URL: https://github.com/apache/arrow/pull/36846#issuecomment-1681277533
@lidavidm Thanks, in PyIceberg we do this by taking the field, casting them accordingly, and then turning it back into a table: https://github.com/apache/iceberg/blob/master/python/pyiceberg/io/pyarrow.py#L952-L1024 It is far from optimal, and the concat operation would be perfect. > Though I don't believe dataset has any provision for manual casting, so if you do have (say) an int column that became a float column, this would be the only way to support it. The `pa.concat_tables` is about tables, and not about datasets. There is an option to cast them today: https://arrow.apache.org/docs/python/generated/pyarrow.Table.html#pyarrow.Table.cast I think having the same safe flag on the `pa.concat_tables` would do the trick as well. WDYT @pitrou ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
