nssalian commented on code in PR #16568:
URL: https://github.com/apache/iceberg/pull/16568#discussion_r3555508917
##########
api/src/main/java/org/apache/iceberg/variants/VariantUtil.java:
##########
@@ -30,8 +32,41 @@ class VariantUtil {
private static final int BASIC_TYPE_OBJECT = 2;
private static final int BASIC_TYPE_ARRAY = 3;
+ /**
+ * Maximum permitted nesting depth of a Variant value. The top-level value
is depth 0, so a
+ * Variant may contain up to {@code MAX_VARIANT_DEPTH + 1} nested levels.
+ */
+ static final int MAX_VARIANT_DEPTH = 1000;
Review Comment:
Aligned `MAX_VARIANT_DEPTH` to 1000 following Steve's parquet dev-list
thread, and flipped the guard to `depth <= MAX_VARIANT_DEPTH` so the accepted
boundary matches parquet-java 3562 exactly (both allow depths 0..1000). Javadoc
now marks it a safety limit, not a spec bound, with a cross-reference to 3562.
##########
api/src/main/java/org/apache/iceberg/variants/SerializedArray.java:
##########
@@ -35,30 +35,47 @@ static SerializedArray from(VariantMetadata metadata,
byte[] bytes) {
return from(metadata,
ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN), bytes[0]);
}
+ @VisibleForTesting
static SerializedArray from(VariantMetadata metadata, ByteBuffer value, int
header) {
+ return from(metadata, value, header, 0);
+ }
+
+ static SerializedArray from(VariantMetadata metadata, ByteBuffer value, int
header, int depth) {
Preconditions.checkArgument(
value.order() == ByteOrder.LITTLE_ENDIAN, "Unsupported byte order: big
endian");
BasicType basicType = VariantUtil.basicType(header);
Preconditions.checkArgument(
basicType == BasicType.ARRAY, "Invalid array, basic type: " +
basicType);
- return new SerializedArray(metadata, value, header);
+ return new SerializedArray(metadata, value, header, depth);
}
private final VariantMetadata metadata;
private final ByteBuffer value;
private final int offsetSize;
private final int offsetListOffset;
private final int dataOffset;
+ private final int depth;
private final VariantValue[] array;
- private SerializedArray(VariantMetadata metadata, ByteBuffer value, int
header) {
+ private SerializedArray(VariantMetadata metadata, ByteBuffer value, int
header, int depth) {
this.metadata = metadata;
this.value = value;
+ this.depth = depth;
this.offsetSize = 1 + ((header & OFFSET_SIZE_MASK) >> OFFSET_SIZE_SHIFT);
int numElementsSize = ((header & IS_LARGE) == IS_LARGE) ? 4 : 1;
+ Preconditions.checkArgument(
+ value.remaining() >= HEADER_SIZE + numElementsSize,
+ "Invalid variant array: buffer too small for element count field");
int numElements = ByteBuffers.readLittleEndianUnsigned(value, HEADER_SIZE,
numElementsSize);
+ Preconditions.checkArgument(
+ numElements >= 0, "Invalid variant array: negative element count %s",
numElements);
this.offsetListOffset = HEADER_SIZE + numElementsSize;
- this.dataOffset = offsetListOffset + ((1 + numElements) * offsetSize);
+ long offsetTableEnd = (long) offsetListOffset + ((long) numElements + 1L)
* offsetSize;
+ Preconditions.checkArgument(
+ offsetTableEnd <= value.remaining(),
Review Comment:
Fixed this too. I added an absolute `MAX_ELEMENTS` cap (16,777,216) checked
before the array/object/metadata allocations. parquet-java 3562 has no
equivalent (it bounds by buffer size only), so this one is iceberg-specific and
I've noted that in the javadoc. Added `testExcessiveArrayNumElementsRejected`
plus object and metadata variants.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
