viirya commented on PR #5754:
URL:
https://github.com/apache/datafusion-comet/pull/5754#issuecomment-5578055761
Correction to my previous comment: the `array<struct<..>>` case **does**
reproduce, and there is now an end-to-end test for it.
What I had missed is that it depends on how the struct reaches the array.
Using it as the element directly, `array(c)`, does not reproduce — a null
element is rebuilt on the way in and the hidden child values go with it, which
is why my first attempt was green either way. Wrapping it does:
`array(named_struct('tag', 1, 'b', c))` yields an element that is itself valid,
so it is copied rather than rebuilt, and the null `c` inside keeps the values
Parquet wrote under it.
```
hash(array(named_struct('tag', 1, 'b', c)))
```
On unfixed `main` that gives `245521047, 292098604, 1066699649, 1066699649`
where Spark gives `-559580957, -559580957, 245521047, 1066699649`; with the
change they agree. Covered with one element and with two, so the chaining
between elements is exercised as well, and the plan is native in both. So the
per-element path #5567 made reachable now has direct end-to-end coverage rather
than only the unit test I mentioned before. I had concluded too early that the
shape was unreachable through a query.
One number in that comment has also moved: `array<struct<..>>` in #5765 now
measures 12609 µs rather than 9660 µs, because I added a skewed shape to that
benchmark and remeasured. The 141 µs for `array<int32>` is unchanged, so the
gap is wider than I quoted, not narrower.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
