VedantSawant616 commented on issue #18337: URL: https://github.com/apache/datafusion/issues/18337#issuecomment-3457648053
hi @alamb, the root cause seems to be that when `datafusion.execution.parquet.skip_metadata = false`, the physical plan reads and keeps the field metadata, but the logical plan drops it somewhere during schema reconstruction (likely in `Projection `or `WindowExec`). this creates a mismatch that triggers the internal consistency error. the blank `Differences: .` output happens because the schema diff code doesn’t know how to display metadata only differences ,a minor formatting bug. workaround: leave `skip_metadata `at its default (`true`). permanent fix: make DataFusion’s schema handling fully metadata-aware, either by consistently propagating field metadata across all planning stages or normalizing it before comparison. that would eliminate these logical–physical schema mismatches once and for all. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
