lidavidm commented on code in PR #1467:
URL: https://github.com/apache/arrow-java/pull/1467#discussion_r4557942118
##########
vector/src/main/java/org/apache/arrow/vector/ipc/ArrowReader.java:
##########
@@ -57,6 +60,34 @@ protected ArrowReader(BufferAllocator allocator,
CompressionCodec.Factory compre
this.compressionFactory = compressionFactory;
}
+ /**
+ * Sets whether every record batch and dictionary batch is checked with
{@link
+ * ValueVectorUtility#validateFull} once it is loaded. This is off by
default, because the check
+ * reads every value of the batch.
+ *
+ * <p>Turn it on when the data comes from an untrusted source. The IPC
format does not guarantee
+ * that offsets, views or run ends stay within the buffers they point into,
and reading such a
+ * batch can read past the end of a buffer, without bounds checks when {@code
+ * arrow.enable_unsafe_memory_access} is set. With this option, a batch that
fails the check is
+ * cleared and the load throws {@link
+ * org.apache.arrow.vector.validate.ValidateUtil.ValidateException}.
+ *
+ * @param validateBatches whether to check each loaded batch
+ */
+ public void setValidateBatches(boolean validateBatches) {
Review Comment:
Ideally we'd have some way to specify validate vs validateFull
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
