devinjdangelo commented on issue #7892: URL: https://github.com/apache/arrow-datafusion/issues/7892#issuecomment-1976610552
@alamb I've narrowed this down to logical planning for `insert_to_plan` not considering the field names of the input schema vs. the table schema. Rather, it assumes the fields are in the same order. `ListingTable` moves the partition column to the end of the schema and we aren't realigning based on the name of the column being the same, which leads to the confusing behavior you flagged in this issue. I believe we should add logic to `insert_to_plan` which will align fields from the input schema and table schema if their names are the same and data types logically equivalent. E.g. there are two cases to handle: 1. Input schema field names are a subset of table schema field names. If so, project input schema so that the schemas are aligned by name. 2. Fall back to the existing logic which assumes the schemas are already aligned if the types are logically equivalent by position. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
