alamb commented on code in PR #24387:
URL: https://github.com/apache/datafusion/pull/24387#discussion_r3817842014
##########
datafusion/datasource/src/file_format.rs:
##########
@@ -42,6 +44,22 @@ use object_store::{ObjectMeta, ObjectStore};
/// Default max records to scan to infer the schema
pub const DEFAULT_SCHEMA_INFER_MAX_RECORD: usize = 1000;
+/// Rejects an inferred schema that names the same field more than once.
+///
+/// [`Schema::try_merge`] coalesces fields by name, so callers validate each
+/// inferred file schema before merging.
+pub fn ensure_unique_field_names(schema: &Schema) -> Result<()> {
+ let mut seen = HashSet::with_capacity(schema.fields().len());
Review Comment:
this is going to make a hashset for schema, -- maybe can we instead reuse
the same hash table?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
