github-actions[bot] commented on code in PR #65715:
URL: https://github.com/apache/doris/pull/65715#discussion_r3611950886
##########
be/src/core/data_type_serde/data_type_varbinary_serde.cpp:
##########
@@ -115,6 +116,98 @@ Status DataTypeVarbinarySerDe::write_column_to_arrow(const
IColumn& column, cons
return Status::OK();
}
+Status DataTypeVarbinarySerDe::read_column_from_arrow(IColumn& column,
+ const arrow::Array*
arrow_array,
+ int64_t start, int64_t
end,
+ const cctz::time_zone&
ctz) const {
+ auto& varbinary_column = assert_cast<ColumnVarbinary&>(column);
+ const auto read_binary = [&](const auto* concrete_array, const auto&
read_offset) -> Status {
+ const auto& buffer = concrete_array->value_data();
+ const size_t buffer_size = buffer ?
static_cast<size_t>(buffer->size()) : 0;
+ for (int64_t offset_i = start; offset_i < end; ++offset_i) {
+ if (concrete_array->IsNull(offset_i)) {
+ varbinary_column.insert_default();
+ continue;
+ }
+
+ const int64_t start_offset = read_offset(offset_i);
+ const int64_t end_offset = read_offset(offset_i + 1);
+ const int64_t length =
+ config::enable_arrow_input_validation
+ ? check_arrow_value_offsets(*concrete_array,
start_offset, end_offset,
+ buffer_size)
+ : end_offset - start_offset;
+ // Arrow may omit an empty values buffer. Do not form a pointer
from it for an empty
+ // value, but require a buffer before reading any non-empty value.
+ if (length == 0) {
+ varbinary_column.insert_data("", 0);
+ } else if (!buffer) {
+ return Status::InvalidArgument("Arrow {} values buffer is
missing",
+ concrete_array->type()->name());
+ } else {
+ varbinary_column.insert_data(
Review Comment:
[P1] Reject unrepresentable LargeBinary values before copying
`LargeBinary` can legally produce an in-bounds `length` of `UINT32_MAX + 1`,
but each `ColumnVarbinary` value is stored in a `StringView` with a `uint32_t`
size. This call reaches `ColumnVarbinary::insert_to_buffer`, which allocates
and memcpy's the entire >4 GiB value before `cast_set<uint32_t>(length)`
throws. A valid Arrow value can therefore double its multi-GiB footprint and
hit memory limits even though Doris can never represent it. Please reject
lengths above `uint32_t` max before `insert_data`, and cover the boundary with
a test that proves no data pointer or allocation is touched.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
