sunchao commented on code in PR #24767:
URL: https://github.com/apache/datafusion/pull/24767#discussion_r4027954075
##########
datafusion/common/src/utils/mod.rs:
##########
@@ -603,9 +601,31 @@ impl SingleRowListArrayBuilder {
/// Build a single element [`LargeListArray`]
pub fn build_large_list_array(self) -> LargeListArray {
+ self.build_generic_list_array()
+ }
+
+ fn build_generic_list_array<OffsetSize: OffsetSizeTrait>(
+ self,
+ ) -> GenericListArray<OffsetSize> {
let (field, arr) = self.into_field_and_arr();
let offsets = OffsetBuffer::from_lengths([arr.len()]);
- LargeListArray::new(field, offsets, arr, None)
+
+ // `is_nullable` is conservative for encoded arrays and can be true
+ // even when the array contains no logical nulls. In that case the
+ // generic constructor rejects a valid non-nullable list child.
+ if !field.is_nullable() && arr.is_nullable() &&
arr.logical_null_count() == 0 {
+ let data = ArrayData::builder(
+ GenericListArray::<OffsetSize>::DATA_TYPE_CONSTRUCTOR(field),
+ )
+ .len(1)
+ .add_buffer(offsets.into_inner().into_inner())
+ .add_child_data(arr.to_data())
+ .build()
+ .expect("single-row list array should contain valid data");
+ return GenericListArray::from(data);
Review Comment:
Rechecked at `4f792b350a6a77dfd2830dbb3833a91901862b16` with local execution
and independent base/head build directories. This confirms my earlier
source-traced finding: `evaluate()?.compacted()` panics in Arrow's list
constructor for dictionary `collect_list` and sparse-union
`collect_list`/`collect_set`. The identical three probes pass on merge base
`c4910e0764f3b02cc27d347bfc117f85a4edb46d`.
I also reproduced the existing grouped/window issue using Dictionary<Int8,
Utf8> keys `[0,0,1,1]`, dictionary values `[Some("a"),Some("b"),None]`, groups
`[0,0,1,1]`, and IDs `[0,1,2,3]`:
```sql
SELECT g, collect_list(x) FROM t GROUP BY g ORDER BY g;
SELECT id, collect_list(x) OVER (
ORDER BY id ROWS BETWEEN 1 PRECEDING AND CURRENT ROW
) FROM t ORDER BY id;
```
Both queries pass on the base and fail on the head with `Non-nullable field
of ListArray "item" cannot contain nulls`, at target partitions 1 and 4. An
ordered outer `array_agg` over the grouped result also fails, although that
query can fail while assembling the inner grouped result; the direct compaction
probes independently isolate the panic.
Dictionary `collect_set`, plain Utf8 SQL, and sliced run-end compaction
controls pass on both revisions. The encoded-child issue therefore remains
unresolved after the normalization/retraction follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
