itsjunetime commented on issue #12560:
URL: https://github.com/apache/datafusion/issues/12560#issuecomment-2368726110
I'm running into this behavior after #11989, specifically seeing schema
mismatches where the only thing that is different is that a field's metadata
disappears at some point (so the schemas are the same except for a field's
metadata). E.g.:
```rust
&physical_input_schema = Schema {
fields: [
Field {
name: "alias1",
data_type: Utf8,
nullable: true,
dict_id: 0,
dict_is_ordered: false,
metadata: {},
},
],
metadata: {},
}
&physical_input_schema_from_logical = Schema {
fields: [
Field {
name: "alias1",
data_type: Utf8,
nullable: true,
dict_id: 0,
dict_is_ordered: false,
metadata: {
"some_key": "some_value"
},
},
],
metadata: {},
}
```
I've yet to figure out exactly where the metadata is being dropped and I
haven't figured out a reproducer either. I suggested comparing only the fields'
non-metadata fields
[here](https://github.com/apache/datafusion/pull/11989#issuecomment-2361812330),
but @jayzhan211 pointed out that that's more of a workaround than an actual
fix, as it's still a problem if the metadata is disappearing.
The issue that I'm running into, though, seems to be somewhat different than
the issue that others (like @phillipleblanc) are running into, where some
fields completely disappear from the schema (see
[here](https://github.com/apache/datafusion/pull/11989#issuecomment-2360822666)).
I don't think these are the same issue, exactly (since they manifest
differently), but they may have the same root cause/solution, so I think it's
fair to keep them all under this issue unless needed otherwise.
I'll work on getting a fix or reproducer today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
