jbonofre opened a new pull request, #1467: URL: https://github.com/apache/arrow-java/pull/1467
## What's Changed Found while reviewing #1422. This change is independent of it and based on `main`. **New option: `ArrowReader.setValidateBatches(boolean)`**, off by default. When it is on, every record batch and dictionary batch is checked with `ValueVectorUtility.validateFull` once it is loaded. A batch that fails the check is cleared, and the load throws `ValidateUtil.ValidateException`. A delta dictionary is checked before it is appended, so an invalid delta leaves the dictionary unchanged. The option is meant for reading untrusted data: without it, a malformed batch, such as a `Utf8` column with an offset past the end of its data buffer, loads fine and is then read out of bounds (without bounds checks when `arrow.enable_unsafe_memory_access` is set). The option is off by default because the check reads every value. **`validateFull` now works on every type a batch can hold.** Turning the option on in `TestRoundTrip` showed that it did not: - `ListViewVector` / `LargeListViewVector`: added to the type, buffer and data validators (they threw `UnsupportedOperationException`). The data check rejects a non-empty list view outside the inner vector. - `RunEndEncodedVector`: added to the data validator. Run ends must be strictly increasing, and each run must have a value. - `FixedSizeListVector`: the data validator no longer reads an offset buffer the vector does not have (`UnsupportedOperationException` for any non-empty vector). - `MapVector`: the type validator accepted only `ArrowType.List`, so it rejected every map. - View vectors: each non-inline view is checked against the data buffers before `validateScalars` reads it. `TestVariableWidthViewVector.testValidateInvalidOffsets` now expects a `ValidateException` instead of the `IndexOutOfBoundsException` the out-of-bounds read used to throw. - Dense unions: negative offsets and type ids without a sub-vector are rejected. **Tests** - `TestArrowReaderValidation`: a valid batch with string views, list views, large list views and run-end encoding passes. A batch with an invalid interior offset, and a dictionary batch with one, are rejected only when the option is on. - `TestValidateVectorFull`: a case for each validator change. - `TestRoundTrip` reads every batch with the option on, which covers struct, union, dictionary, fixed-size list, map and the other round-tripped types. - `ValueVectorDataPopulator.setVector(LargeListViewVector, ...)` wrote 4-byte sizes at the wrong stride. It now writes 64-bit offsets and sizes (it was not used by any other test). Without the validator changes, the new tests and the fixed-size list, map, list-as-map and union V5 round trips fail. The full `vector` test suite passes. Not covered: C Data imports carry no buffer sizes (the importer sizes buffers from the producer's length), and Flight's `FlightStream` loads batches with its own `VectorLoader`. Closes #1466. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
