mxtymoshyk opened a new pull request, #40841:
URL: https://github.com/apache/beam/pull/40841
Fixes #26911
`CROSS JOIN UNNEST` over an `ARRAY<ROW<...>>` whose element has a nested ROW
field fails when the query runs through the JDBC driver or the SQL shell:
```
java.lang.IllegalArgumentException: Row expected 4 fields (Field{name=ref,
...}, Field{name=commits, ...}, Field{name=id, ...}, Field{name=author,
type=ROW<name STRING, email STRING>, ...}). initialized with 5 fields.
at org.apache.beam.sdk.values.Row$Builder.build(Row.java:861)
at
org.apache.beam.sdk.extensions.sql.impl.rel.BeamUnnestRel$UnnestFn.process(BeamUnnestRel.java:195)
```
Beam plans SQL with two different Calcite planners, and they give different
unnest output for the same query. The planner behind `SqlTransform` flattens
the element's nested row into leaf columns (`id, name, email`). The planner
behind the JDBC driver keeps it as one ROW column (`id, author`). `UnnestFn`
always flattened, so it only matched `SqlTransform`.
`UnnestFn` now checks each nested row field of the element against the
output column at the same position. If that column is a ROW, the row goes in as
is. Otherwise the row's values go in flattened.
The same change fixes a null nested row on the flattened path. The old code
added no values for it, so the row came out short and hit the same field count
error. It now fills one null per flattened column.
Tests:
- `JdbcDriverTest#testUnnestArrayOfRowsWithNestedRow` covers the JDBC path.
- `BeamSqlDslUnnestRowsTest#testUnnestArrayWithNullNestedRow` covers a null
nested row on the `SqlTransform` path.
- Both fail without the change to `BeamUnnestRel`. The existing
`testUnnestArrayWithNestedRows` still passes and covers the flattened path.
- `./gradlew :sdks:java:extensions:sql:check` passes locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
