adriangb opened a new issue, #27485: URL: https://github.com/apache/datafusion/issues/27485
### Is your feature request related to a problem or challenge? The SQL unparser (`datafusion/sql/src/unparser`) is tested by hand-written cases in `unparser/expr.rs` and `datafusion/sql/tests/cases/plan_to_sql.rs`. Those cases cover the shapes their authors thought of. Gaps show up later, one at a time, when a downstream user unparses a real plan and the SQL does not plan again. For example, #27477 reports that a non-finite float literal unparses as a bare `inf` or `NaN` token. The sqllogictest suite already has a few thousand queries with checked results. Today it runs them through the unparser only by accident. Nothing in `datafusion/sqllogictest` calls `Unparser` or `plan_to_sql`. ### Describe the solution you'd like Add a round-trip engine, like the existing `datafusion_substrait_roundtrip_engine`, behind a new runner flag (for example `--unparser-round-trip`). For each query, the engine: 1. Plans the SQL (optimized or unoptimized, like `--substrait-optimize`). 2. Unparses the logical plan with `plan_to_sql` and the default dialect. 3. Plans the unparsed SQL again in the same `SessionContext`. 4. Runs the re-planned query and returns its rows for the normal comparison. Like the Substrait engine, it passes through plans that SQL cannot express (`Ddl`, `Dml`, `Copy`, `Explain`, `Show`, `Statement`) unchanged. A failure in step 2 or 3 is a bug in the unparser or the planner. A row difference in step 4 is a semantic bug in the unparser, which is harder to find by hand. ### Describe alternatives you've considered - Keep adding cases to `plan_to_sql.rs` per bug report. That works, but it finds each gap only after a user hits it. - Compare the re-planned logical plan with the original plan instead of the rows. That is stricter, but many correct unparses do not give an identical plan (aliases, cast placement). Comparing the rows matches what the Substrait mode does. ### Additional context - Many files will fail at first. So a first version can be opt-in only, like `--substrait-round-trip`, which I do not see in CI today. A later step can run it in CI with a list of known-failing files that shrinks over time. - The known-failing list doubles as an inventory of unparser gaps. - I can work on this if maintainers agree with the approach. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
