andygrove commented on code in PR #1422:
URL: https://github.com/apache/arrow-java/pull/1422#discussion_r4436131519
##########
vector/src/main/java/org/apache/arrow/vector/ipc/message/ArrowFieldNode.java:
##########
@@ -35,6 +36,13 @@ public class ArrowFieldNode implements FBSerializable {
*/
public ArrowFieldNode(long length, long nullCount) {
super();
+ // A negative length is never valid and, if accepted, flows unchecked into
buffer size
+ // computations (BitVectorHelper.getValidityBufferSizeFromCount performs
an unsigned shift)
+ // and into ArrowBuf.setOne/setZero, which can corrupt memory when bounds
checking is
+ // disabled. Reject it at this single chokepoint shared by every
deserialization path
+ // (IPC, JSON, C Data). nullCount is intentionally not checked here:
VectorUnloader uses -1
+ // as a sentinel meaning "null count not included".
Review Comment:
Removed the comment. The `-1` sentinel is now documented on the `nullCount`
param, and the constructor rejects `nullCount < -1` (e593bb3d8).
##########
vector/src/test/java/org/apache/arrow/vector/ipc/message/TestArrowFieldNode.java:
##########
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.arrow.vector.ipc.message;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+import org.junit.jupiter.api.Test;
+
+public class TestArrowFieldNode {
+
+ @Test
+ public void testNegativeLengthRejected() {
+ // A negative length that still fits in an int must be rejected. Left
unchecked it flows into
+ // BitVectorHelper.getValidityBufferSizeFromCount (unsigned shift ->
~512MB) and into
+ // ArrowBuf.setOne(0, length / 8) with a negative length, corrupting
memory when bounds
+ // checking is disabled. See the deserialization paths in
MessageSerializer and JsonFileReader.
Review Comment:
Removed the test comments too.
##########
vector/src/main/java/org/apache/arrow/vector/ipc/message/ArrowFieldNode.java:
##########
@@ -35,6 +36,13 @@ public class ArrowFieldNode implements FBSerializable {
*/
public ArrowFieldNode(long length, long nullCount) {
super();
+ // A negative length is never valid and, if accepted, flows unchecked into
buffer size
+ // computations (BitVectorHelper.getValidityBufferSizeFromCount performs
an unsigned shift)
+ // and into ArrowBuf.setOne/setZero, which can corrupt memory when bounds
checking is
+ // disabled. Reject it at this single chokepoint shared by every
deserialization path
+ // (IPC, JSON, C Data). nullCount is intentionally not checked here:
VectorUnloader uses -1
+ // as a sentinel meaning "null count not included".
+ Preconditions.checkArgument(length >= 0, "Expected length >= 0, got %s",
length);
this.length = checkedCastToInt(length);
this.nullCount = checkedCastToInt(nullCount);
Review Comment:
Applied in e593bb3d8: both values are narrowed with `Math.toIntExact`, and
the `checkedCastToInt` import is gone. Added `testLengthAboveIntRangeRejected`
and `testNullCountAboveIntRangeRejected`. As you said, they pass on the old
code in the default run; with `ARROW_ENABLE_UNSAFE_MEMORY_ACCESS=true` they
fail on the old code and pass now.
##########
vector/src/main/java/org/apache/arrow/vector/ipc/message/ArrowFieldNode.java:
##########
@@ -35,6 +36,13 @@ public class ArrowFieldNode implements FBSerializable {
*/
public ArrowFieldNode(long length, long nullCount) {
super();
+ // A negative length is never valid and, if accepted, flows unchecked into
buffer size
+ // computations (BitVectorHelper.getValidityBufferSizeFromCount performs
an unsigned shift)
+ // and into ArrowBuf.setOne/setZero, which can corrupt memory when bounds
checking is
+ // disabled. Reject it at this single chokepoint shared by every
deserialization path
+ // (IPC, JSON, C Data). nullCount is intentionally not checked here:
VectorUnloader uses -1
Review Comment:
Added a `readCount()` helper in e593bb3d8 that rejects a negative count as
soon as it's read, for the column count and both batch counts (`read()` and
`read(VectorSchemaRoot)`, which dictionary batches also use). The comment
claiming JSON coverage is gone. `TestJSONFile` has a test per site; the
column-count one reads through a 1 MB allocator and failed on the old code with
`OutOfMemoryException` on the 536,870,900-byte validity buffer.
##########
vector/src/main/java/org/apache/arrow/vector/ipc/message/ArrowFieldNode.java:
##########
@@ -35,6 +36,13 @@ public class ArrowFieldNode implements FBSerializable {
*/
public ArrowFieldNode(long length, long nullCount) {
super();
+ // A negative length is never valid and, if accepted, flows unchecked into
buffer size
+ // computations (BitVectorHelper.getValidityBufferSizeFromCount performs
an unsigned shift)
+ // and into ArrowBuf.setOne/setZero, which can corrupt memory when bounds
checking is
+ // disabled. Reject it at this single chokepoint shared by every
deserialization path
+ // (IPC, JSON, C Data). nullCount is intentionally not checked here:
VectorUnloader uses -1
+ // as a sentinel meaning "null count not included".
+ Preconditions.checkArgument(length >= 0, "Expected length >= 0, got %s",
length);
Review Comment:
Same as the thread above
(https://github.com/apache/arrow-java/pull/1422#discussion_r4434464977);
addressed there.
##########
vector/src/main/java/org/apache/arrow/vector/ipc/message/ArrowFieldNode.java:
##########
@@ -35,6 +36,13 @@ public class ArrowFieldNode implements FBSerializable {
*/
public ArrowFieldNode(long length, long nullCount) {
super();
+ // A negative length is never valid and, if accepted, flows unchecked into
buffer size
+ // computations (BitVectorHelper.getValidityBufferSizeFromCount performs
an unsigned shift)
+ // and into ArrowBuf.setOne/setZero, which can corrupt memory when bounds
checking is
+ // disabled. Reject it at this single chokepoint shared by every
deserialization path
+ // (IPC, JSON, C Data). nullCount is intentionally not checked here:
VectorUnloader uses -1
Review Comment:
Same as the thread above
(https://github.com/apache/arrow-java/pull/1422#discussion_r4434464993);
addressed there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
