Mr-Nobody-Anonymous commented on PR #55671:
URL: https://github.com/apache/arrow/pull/55671#issuecomment-7010324553
Hi @Awesome06,
Thank you for this valuable hardening work on the IPC decoder and message
boundaries.
I noticed the CI build failed across the C++ matrix on
`read_write_test.cc:3579` with the following compilation error:
`
error: unable to deduce 'std::initializer_list<auto>&&' from
'{(&options.arrow::ipc::IpcReadOptions::max_metadata_length),
(&options.arrow::ipc::IpcReadOptions::max_body_length),
(&options.arrow::ipc::IpcReadOptions::max_decompression_ratio)}'
note: deduced conflicting types for parameter 'auto' ('long int*' and
'int32_t*' {aka 'int*'})
`
The root cause is that `max_metadata_length` and `max_body_length` are
`int64_t*`, whereas `max_decompression_ratio` is `int32_t*`. Because the braced
initializer list contains mixed pointer types, template type deduction fails.
I investigated this and prepared a patch that addresses both this build
failure and the active review findings:
1. Build Failure in Test:
- Separated the `int64_t*` fields (`max_metadata_length`, `max_body_length`)
from `int32_t*` (`max_decompression_ratio`) in `TEST(TestIpcReadOptions,
NegativeLimitsInvalid)`.
2. Negative Body and Metadata Length Rejection:
- Added validation in `IpcReadOptions::CheckMetadataLength` and
`IpcReadOptions::CheckBodyLength` (as well as `CheckBodyLength` in
`message.cc`) to reject negative lengths directly before checking limits or
performing arithmetic.
3. Limit Enforcement in Direct RecordBatch Entry Points:
- Moved `options.Validate()`, `CheckMetadataLength()`, and
`CheckBodyLength()` into the shared `ReadRecordBatchInternal` function in
`cpp/src/arrow/ipc/reader.cc` so that direct calls like
`ReadRecordBatch(metadata, ...)` and `ReadRecordBatch(message, ...)` respect
the configured limits.
4. Unconditional Metadata Allocation:
- Removed `metadata->reserve(fb_metadata->size())` in `GetKeyValueMetadata`
in `cpp/src/arrow/ipc/metadata_internal.cc` to prevent untrusted FlatBuffers
custom metadata from triggering premature large vector allocations.
5. Added Unit Tests:
- Added tests verifying that negative lengths are rejected and that direct
`ReadRecordBatch` calls enforce metadata and body length limits.
I pushed the complete fix branch here for your convenience if you would like
to cherry-pick or merge it:
https://github.com/Mr-Nobody-Anonymous/arrow/tree/fix-pr-55671-ipc-bounds
Please feel free to use any or all of these changes if they are helpful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]