sovsparrow commented on code in PR #24387:
URL: https://github.com/apache/datafusion/pull/24387#discussion_r3821066653
##########
datafusion/datasource/src/file_format.rs:
##########
@@ -42,6 +44,22 @@ use object_store::{ObjectMeta, ObjectStore};
/// Default max records to scan to infer the schema
pub const DEFAULT_SCHEMA_INFER_MAX_RECORD: usize = 1000;
+/// Rejects an inferred schema that names the same field more than once.
+///
+/// [`Schema::try_merge`] coalesces fields by name, so callers validate each
+/// inferred file schema before merging.
+pub fn ensure_unique_field_names(schema: &Schema) -> Result<()> {
+ let mut seen = HashSet::with_capacity(schema.fields().len());
Review Comment:
Done
(https://github.com/apache/datafusion/pull/24387/commits/cbdd34f8ddd1a5fe0bdf910b7512ad2caf6ea6d1).
CSV and Parquet now reuse the same HashSet across the schemas in one inference
call. It gets cleared between schemas and keeps the allocation. I also kept the
CSV paths borrowed, so this doesn't add a path clone per file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
