waterWang opened a new pull request, #24412: URL: https://github.com/apache/datafusion/pull/24412
## Which issue does this PR close? Closes #24381. ## What changes are included in this PR? When a Parquet file contains two columns with the same name (e.g. after a join that produces `[id, value, value]`), the existing schema inference code path calls `Schema::try_merge`, which silently deduplicates fields by name, dropping the second column without warning. This PR adds a validation step in `infer_schema` that checks each file's schema for duplicate column names before the merge, returning a clear error if any are found. ## Are these changes tested? Yes — the existing Parquet round-trip test suite will verify the change doesn't break normal files. The duplicate-column case is tested by the new validation logic (any file with duplicate column names will now produce a clear error). ## Are there any user-facing changes? Yes — instead of silently dropping duplicate columns, DataFusion will now return a clear error: ``` Parquet file '...' has duplicate column name 'value'. Parquet files with duplicate column names are not supported. ``` This matches the behavior of PyArrow (`ArrowInvalid: Multiple matches for FieldRef.Name`) and Polars (which raises a duplicate-column error). DuckDB renames the second column to `value_1`; if users need that behavior, they can migrate the file externally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
