github-actions[bot] commented on code in PR #66819:
URL: https://github.com/apache/doris/pull/66819#discussion_r3793589678
##########
be/src/format/arrow/arrow_array_normalizer.cpp:
##########
@@ -89,6 +92,31 @@ Status normalize_arrow_array(const
std::shared_ptr<arrow::Array>& arr,
return Status::OK();
}
+ // List views may share or reorder value ranges, so rebuild canonical
offsets instead of
+ // exposing their buffers to a serde that requires contiguous list
values.
+ if (type.id() == arrow::Type::LIST_VIEW) {
+ auto converted = arrow::ListArray::FromListView(
+ static_cast<const arrow::ListViewArray&>(*current),
+ arrow::default_memory_pool());
Review Comment:
[P1] Route the canonicalization allocation through Doris tracking
`FromListView` computes `SumOfLogicalListSizes`, reserves that full child
capacity, and copies each logical range from this pool. With N rows sharing an
M-element range, a small M-element physical child creates an N*M intermediate,
which remains live while the Doris array SerDe copies N*M values again. Arrow's
default system pool is not Doris's `ArrowMemoryPool`, and no repository
override connects it to the thread/query MemTracker, so a large batch bypasses
query limits and can exhaust process memory. Please pass an explicit
caller-owned tracked pool (for example, the ADBC runtime's ExecEnv pool, whose
lifetime encloses `out`) for both widths; direct SerDe materialization could
also remove the extra peak.
##########
be/src/format/arrow/arrow_array_normalizer.cpp:
##########
@@ -89,6 +92,31 @@ Status normalize_arrow_array(const
std::shared_ptr<arrow::Array>& arr,
return Status::OK();
}
+ // List views may share or reorder value ranges, so rebuild canonical
offsets instead of
+ // exposing their buffers to a serde that requires contiguous list
values.
+ if (type.id() == arrow::Type::LIST_VIEW) {
+ auto converted = arrow::ListArray::FromListView(
Review Comment:
[P1] Avoid this unsafe Arrow 24 conversion path
This API is unsafe for two inputs ADBC can deliver. On a nullable slice,
Arrow 24 calls `GetValues<uint8_t>(0)` (which already advances by
`ArrayData::offset` bytes) and then `GetBit(..., offset + i)`, so a nonzero
offset reads the wrong validity bit and can silently flip null rows. On a
malformed range, `ListFromListViewImpl` calls `AppendArraySlice(values, offset,
size)` before Doris's default-on SerDe validation, and the numeric builder
copies without checking `offset + size <= values.length()`, allowing an
out-of-bounds read. C-stream import preserves child offsets and does not
`ValidateFull`. Please validate/rebase the source or use a corrected
conversion, and add nullable-slice and out-of-range tests for both widths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
