rustyconover opened a new pull request, #982: URL: https://github.com/apache/arrow-nanoarrow/pull/982
Depends on #981 (`feat/validate-utf8-full`), which adds UTF-8 validation for string and large_string at FULL. This PR is stacked on it; only the last commit is new. `NANOARROW_VALIDATION_LEVEL_FULL` never checked that a binary view or string view refers to bytes that exist. An out-of-line view from untrusted input could name a variadic buffer that doesn't exist, or a range outside its buffer, and FULL accepted it. Arrow C++'s `ValidateFull()` rejects both. For every non-null view of a `binary_view` or `string_view` array, FULL validation now checks: - the size is non-negative; - for an out-of-line value (more than 12 bytes), the buffer index is in range and the byte range is within that variadic buffer; - for `string_view`, that the value is valid UTF-8, as the UTF-8 PR does for string and large_string. As in Arrow C++ (checked against pyarrow 25.0.1), null views are not checked. Arrow C++ also checks that an out-of-line view's 4-byte prefix matches its data and that an inline view's padding is zero. Neither affects memory safety, so they are left out of this PR. Errors name the view, e.g. `[1] Expected view buffer index >= 0 and < 1 but found 1` or `[1] Expected view range 1-21 to be within variadic buffer 0 of size 20`. `ArrowArrayFinishBuilding()` validates through an internal `ArrowArrayView` (`ArrowArrayViewInitFromArray()`) that did not point at the variadic buffers or their sizes. Nothing read them at any level before, so this was invisible. With these checks, though, every built view array with an out-of-line value would fail FULL with `Expected view buffer index >= 0 and < 0 but found 0`. It now points at them, as `ArrowArrayViewSetArray()` does. `ArrayTestFinishBuildingFullViewArray` covers this: it builds binary_view and string_view arrays that span several variadic buffers and finishes them at FULL. That test fails without the fix. Like the UTF-8 PR, **FULL now rejects data it used to accept**: views that point outside their buffers, and string views that aren't valid UTF-8. Cost of FULL validation over 10 MB, compared with `main` and with Arrow C++ `ValidateFull()` (pyarrow 25.0.1) on the same machine (Linux aarch64, Neoverse-N1, Release build). `main` checks nothing per view. | Array | `main` | This PR | Arrow C++ | |---|---|---|---| | binary_view, 1M × 10 B | <0.01 ms | 2.0 ms | — | | binary_view, 10k × 1000 B | <0.01 ms | 0.04 ms | — | | string_view, 1M × 10 B ASCII | <0.01 ms | 10.4 ms | 13.2 ms | | string_view, 10k × 1000 B ASCII | <0.01 ms | 1.2 ms | 1.4 ms | Tests: `ArrayViewTestValidateUtf8` now also covers string_view, with values both inlined and out-of-line. `ArrayViewTestValidateBinaryViewBounds` covers an out-of-range buffer index (too large and negative), a negative offset, a range past the end of the buffer, a negative size, and a null view with a bad index, which is not checked. `ArrayTestFinishBuildingFullViewArray` is described above. `ctest` passes (Linux aarch64, Debug, 872 tests), including under ASan/UBSan with clang (397 tests). The changed files are formatted with clang-format 16.0.6, the pre-commit version. This PR's code and description were written by AI (Claude Code) and reviewed by a human maintainer. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
