alamb commented on code in PR #17796:
URL: https://github.com/apache/datafusion/pull/17796#discussion_r2383346810
##########
datafusion/datasource-csv/src/file_format.rs:
##########
@@ -593,15 +593,23 @@ fn build_schema_helper(names: Vec<String>, types:
&[HashSet<DataType>]) -> Schem
.zip(types)
.map(|(field_name, data_type_possibilities)| {
// ripped from
arrow::csv::reader::infer_reader_schema_with_csv_options
- // determine data type based on possible types
- // if there are incompatible types, use DataType::Utf8
- match data_type_possibilities.len() {
- 1 => Field::new(
+ // determine data type based on possible types, ignoring
DataType::Null,
Review Comment:
Thank you @dqkqd -- this looks like it would work well. I played around with
it and I think we might be able to make this simpler and more efficient by
passing in the HashTable and removing the the null. Something like this seemed
to work locally
```rust
// changed signature to take Vec<HashSet<DataType>>. ----v
fn build_schema_helper(names: Vec<String>, types: Vec<HashSet<DataType>>) ->
Schema {
let fields = names
.into_iter()
.zip(types)
.map(|(field_name, mut data_type_possibilities)| {
// ripped from
arrow::csv::reader::infer_reader_schema_with_csv_options
// determine data type based on possible types
// Remove Null (missing column) from possibilities
data_type_possibilities.remove(&DataType::Null); <--- changed
this
// if there are incompatible types, use DataType::Utf8
match data_type_possibilities.len() {
...
```
##########
datafusion/core/src/datasource/file_format/csv.rs:
##########
@@ -470,6 +471,47 @@ mod tests {
Ok(())
}
+ #[tokio::test]
+ async fn test_infer_schema_stream_separated_chunks_with_nulls() ->
Result<()> {
Review Comment:
I don't understand how this test coves the new code.
However, I did verify that without this code change, this PR fails like this:
```
thread
'datasource::file_format::csv::tests::test_infer_schema_stream_separated_chunks_with_nulls'
panicked at datafusion/core/src/datasource/file_format/csv.rs:511:9:
assertion `left == right` failed
left: ["c1: Int64", "c2: Float64"]
right: ["c1: Utf8", "c2: Utf8"]
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
