Blizzara opened a new pull request, #11547: URL: https://github.com/apache/datafusion/pull/11547
## Which issue does this PR close? Related to Map epic https://github.com/apache/datafusion/issues/11434 ## Rationale for this change Substrait didn't support Map literals, since we previously didn't have Map ScalarValues. https://github.com/apache/datafusion/pull/11224 implemented ScalarValues (thanks!), so now we can add them into Substrait as well. There were also couple gaps left by the implementation that I encountered while testing this, so I fixed those as well. Also, there was a bug in the from_substrait_literal for lists containing structs, which I realized while implementing the map support. ## What changes are included in this PR? - Implement Substrait roundtrip support for Map literals, incl. null and empty maps. - Fix field name handling for Substrait lists containing structs - Add Map support to `ScalarValue::iter_to_array` and `create_hashes` ## Are these changes tested? Tested with new roundtrip tests for the Map literals, and unit tests for hashing. Also an existing UT for list literals is extended to cover the multiple structs values case. I'd have added a sql roundtrip test for Map as well, but it seems that the `MAP` command turns into a ScalarFunction rather than ScalarValue. Maybe we'd need to run some round of optimizer to fold it, but I wonder why that is different from the `STRUCT` command which does turn into a struct literal? I.e. doing `roundtrip("VALUES (MAP(['k1', 'k2'], [true, CAST(NULL AS BOOLEAN)]))").await?;` results in `Error: Substrait("Only literal types can be aliased in Virtual Tables, got: ScalarFunction")`, while similar code for STRUCT works fine. ## Are there any user-facing changes? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
