namanjain24-sudo commented on issue #25100: URL: https://github.com/apache/datafusion/issues/25100#issuecomment-5683735063
@alexandrefimov @alamb yes, I'll add it to #25146. Before writing it, I checked that such a test really tells the fix apart from `main`, using a plan DataFusion itself generates. The query is `SELECT count(i), sum(i), avg(i) FROM t`, run through substrait-java 0.103.0 and substrait-spark on Spark 3.5.4 over `t(i) = 1, 2, 3`: | producer | workarounds applied | result | | --- | --- | --- | | #25146 | none | substrait-java: `Function 'avg' references URN anchor -1` (#11545) | | #25146 | #11545 | substrait-java: `Type is not set` (#25049) | | #25146 | #11545 + `output_type` | Spark `Complete`, `[3, 6, 2.0]` | | `main` (f8cc678, the base of #25146) | #11545 + `output_type` | Spark `Final`, analysis fails with `MISSING_ATTRIBUTES` | | #25146 with #25090 applied | #11545 | Spark `Complete`, `[3, 6, 2.0]` | So on #25146 alone the test needs a second workaround: substrait-java stops at the missing `output_type` before it reads `phase`. I'll include both and mark the `output_type` one for removal once #25090 lands. If #25090 is merged first, only the #11545 workaround is needed. What I plan to push to #25146: - a Rust test in `datafusion/substrait` that writes the plan, - a small Java program (substrait-java and substrait-spark 0.103.0, Spark 3.5.4 in local mode) that applies the workarounds, runs the plan and checks the rows, - a CI job with JDK 17 that runs both only when `datafusion/substrait/**` changes. The Spark classpath is about 260 MB, so the job would cache it. This would be the first Java code and JVM job in the repo, so please say if you'd rather it lived in a separate workflow or somewhere else. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
