alamb commented on code in PR #17796:
URL: https://github.com/apache/datafusion/pull/17796#discussion_r2383402440
##########
datafusion/datasource-csv/src/file_format.rs:
##########
@@ -593,15 +593,23 @@ fn build_schema_helper(names: Vec<String>, types:
&[HashSet<DataType>]) -> Schem
.zip(types)
.map(|(field_name, data_type_possibilities)| {
// ripped from
arrow::csv::reader::infer_reader_schema_with_csv_options
- // determine data type based on possible types
- // if there are incompatible types, use DataType::Utf8
- match data_type_possibilities.len() {
- 1 => Field::new(
+ // determine data type based on possible types, ignoring
DataType::Null,
Review Comment:
Though when I did this I found some test failures
```
---- datasource::file_format::csv::tests::infer_schema stdout ----
thread 'datasource::file_format::csv::tests::infer_schema' panicked at
datafusion/core/src/datasource/file_format/csv.rs:273:9:
assertion `left == right` failed
left: ["c1: Utf8", "c2: Int64", "c3: Int64", "c4: Int64", "c5: Int64",
"c6: Int64", "c7: Int64", "c8: Int64", "c9: Int64", "c10: Utf8", "c11:
Float64", "c12: Float64", "c13: Utf8", "c14: Null", "c15: Utf8"]
right: ["c1: Utf8", "c2: Int64", "c3: Int64", "c4: Int64", "c5: Int64",
"c6: Int64", "c7: Int64", "c8: Int64", "c9: Int64", "c10: Utf8", "c11:
Float64", "c12: Float64", "c13: Utf8", "c14: Utf8", "c15: Utf8"]
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---- datasource::file_format::csv::tests::infer_schema_with_null_regex
stdout ----
thread 'datasource::file_format::csv::tests::infer_schema_with_null_regex'
panicked at datafusion/core/src/datasource/file_format/csv.rs:324:9:
assertion `left == right` failed
left: ["c1: Utf8", "c2: Int64", "c3: Int64", "c4: Int64", "c5: Int64",
"c6: Int64", "c7: Int64", "c8: Int64", "c9: Int64", "c10: Utf8", "c11:
Float64", "c12: Float64", "c13: Utf8", "c14: Null", "c15: Null"]
right: ["c1: Utf8", "c2: Int64", "c3: Int64", "c4: Int64", "c5: Int64",
"c6: Int64", "c7: Int64", "c8: Int64", "c9: Int64", "c10: Utf8", "c11:
Float64", "c12: Float64", "c13: Utf8", "c14: Utf8", "c15: Utf8"]
failures:
datasource::file_format::csv::tests::infer_schema
datasource::file_format::csv::tests::infer_schema_with_null_regex
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
