This is an automated email from the ASF dual-hosted git repository.
uros-b pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 6a021f1ef425 [SPARK-60711][SQL] `is_valid_variant` does not validate
the final offset of objects and arrays
6a021f1ef425 is described below
commit 6a021f1ef425d892ce6e70135f9cb205cc5c8eb8
Author: bajka <[email protected]>
AuthorDate: Thu Nov 5 14:55:21 2026 +0000
[SPARK-60711][SQL] `is_valid_variant` does not validate the final offset of
objects and arrays
### What changes were proposed in this pull request?
`is_valid_variant` now checks the final offset of every object and array.
It returns false when:
1. The final offset is missing (an empty array `[3, 0]` or an empty
object `[2, 0]`).
2. The final offset points past the end of the value (`[3, 0, 5]`).
3. A child does not end within its container's data.
Unused bytes inside a containers data are still allowed.
### Why are the changes needed?
`is_valid_variant` returned true for malformed variants. A container with n
children needs n + 1 offsets, but the final offset was never read, so an empty
array or object without it was accepted and its value was not checked either,
so a container or child extending past its bounds was also accepted. Such
values can later fail with `MALFORMED_VARIANT` or be silently truncated when
subvalues are extracted.
### Does this PR introduce _any_ user-facing change?
Yes. `is_valid_variant` now returns false for the malformed variants
described above, where it previously returned true. Valid variants are not
affected.
### How was this patch tested?
Added cases to `VariantExpressionSuite` "is_valid_variant":
- Final offset missing: empty array, empty object, truncated 4-byte
offset, 4-byte size header, nested empty array.
- Final offset past the end of the value: empty array and empty object
with final offset 5.
- Child not ending within its container: array element, object field,
element starting at the final offset, nested array, long string.
- Still valid: unused bytes in container data, unused bytes between
children, object fields not in offset order, trailing bytes after the top-level
value, complete empty containers with 4-byte offset and 4-byte size headers.
- Variants built by `parse_json` (nested, empty, long strings, duplicate
keys) are valid.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
Closes #59927 from SavicStefan/effort/isvalidvariant.
Lead-authored-by: bajka <[email protected]>
Co-authored-by: Stefan Savić <[email protected]>
Signed-off-by: Uros Bojanic <[email protected]>
(cherry picked from commit 52357949c6d43209a16ff991359e31fa4d2118e5)
Signed-off-by: Uros Bojanic <[email protected]>
---
.../apache/spark/types/variant/VariantUtil.java | 24 ++++-
.../variant/VariantExpressionSuite.scala | 109 +++++++++++++++++++++
2 files changed, 129 insertions(+), 4 deletions(-)
diff --git
a/common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java
b/common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java
index 184a7b29a978..64044ad49eb7 100644
---
a/common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java
+++
b/common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java
@@ -605,10 +605,12 @@ public class VariantUtil {
// Validate whether a variant is well-formed. Returns true if the variant
binary is structurally
// well-formed (all bounds and type-info checks pass), false if it is
malformed.
//
- // This is close to, but not strictly equivalent to, "`toJson` does not
throw": this function
- // does not enforce the `SIZE_LIMIT` check that the `Variant` constructor
applies (which throws
- // `VARIANT_CONSTRUCTOR_SIZE_LIMIT`). The implementation otherwise has the
same structure as
- // `toJson` (see `Variant.toJsonImpl`).
+ // This is close to, but not strictly equivalent to, "`toJson` does not
throw". It is more
+ // lenient in that it does not enforce the `SIZE_LIMIT` check that the
`Variant` constructor
+ // applies (which throws `VARIANT_CONSTRUCTOR_SIZE_LIMIT`). It is stricter
in that it requires
+ // every object and array to have its final offset, and its data and
children to lie within
+ // their bounds. The implementation otherwise has the same structure as
`toJson` (see
+ // `Variant.toJsonImpl`).
//
// Implementation note: this `try { ... } catch (SparkRuntimeException e)`
is sound only because
// every helper invoked by `validateImpl` throws `MALFORMED_VARIANT` /
@@ -630,22 +632,36 @@ public class VariantUtil {
switch (getType(value, pos)) {
case OBJECT:
handleObject(value, pos, (size, idSize, offsetSize, idStart,
offsetStart, dataStart) -> {
+ // The offset list has `size + 1` entries. The last one is the size
of the field data,
+ // which must lie within `value`.
+ int dataSize = readUnsigned(value, offsetStart + offsetSize * size,
offsetSize);
+ if ((long) dataStart + dataSize > value.length) throw
malformedVariant();
for (int i = 0; i < size; ++i) {
int id = readUnsigned(value, idStart + idSize * i, idSize);
int offset = readUnsigned(value, offsetStart + offsetSize * i,
offsetSize);
int elementPos = dataStart + offset;
getMetadataKey(metadata, id);
validateImpl(value, metadata, elementPos);
+ // Sub-values are sliced by `valueSize` (e.g. `Variant.getValue`),
so a field that
+ // extends past the field data of the object would be truncated.
+ if ((long) offset + valueSize(value, elementPos) > dataSize) throw
malformedVariant();
}
return null;
});
break;
case ARRAY:
handleArray(value, pos, (size, offsetSize, offsetStart, dataStart) -> {
+ // The offset list has `size + 1` entries. The last one is the size
of the element data,
+ // which must lie within `value`.
+ int dataSize = readUnsigned(value, offsetStart + offsetSize * size,
offsetSize);
+ if ((long) dataStart + dataSize > value.length) throw
malformedVariant();
for (int i = 0; i < size; ++i) {
int offset = readUnsigned(value, offsetStart + offsetSize * i,
offsetSize);
int elementPos = dataStart + offset;
validateImpl(value, metadata, elementPos);
+ // Sub-values are sliced by `valueSize` (e.g. `Variant.getValue`),
so an element that
+ // extends past the element data of the array would be truncated.
+ if ((long) offset + valueSize(value, elementPos) > dataSize) throw
malformedVariant();
}
return null;
});
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionSuite.scala
index c298278106d8..e23857541b14 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionSuite.scala
@@ -1468,6 +1468,115 @@ class VariantExpressionSuite extends SparkFunSuite with
ExpressionEvalHelper {
/* offset list */ 0, 2,
/* field data: INT8 with only 1 byte */ primitiveHeader(INT8), 0),
metadata)
+ // Malformed containers: the final offset is missing. A container with n
children requires
+ // n + 1 offsets, so even an empty container requires one.
+ invalid(Array(arrayHeader(false, 1),
+ /* size */ 0))
+ invalid(Array(objectHeader(false, 1, 1),
+ /* size */ 0))
+ invalid(Array(arrayHeader(false, 4),
+ /* size */ 0,
+ /* truncated offset list */ 0, 0))
+ valid(Array(arrayHeader(false, 4),
+ /* size */ 0,
+ /* offset list */ 0, 0, 0, 0))
+ invalid(Array(arrayHeader(true, 1),
+ /* size */ 0, 0, 0, 0))
+ valid(Array(arrayHeader(true, 1),
+ /* size */ 0, 0, 0, 0,
+ /* offset list */ 0))
+ invalid(Array(arrayHeader(false, 1),
+ /* size */ 1,
+ /* offset list */ 0, 2,
+ /* element data: empty array without its offset list */
arrayHeader(false, 1), 0))
+
+ // Malformed containers: the final offset points past the end of the value.
+ invalid(Array(arrayHeader(false, 1),
+ /* size */ 0,
+ /* offset list */ 5))
+ invalid(Array(objectHeader(false, 1, 1),
+ /* size */ 0,
+ /* offset list */ 5))
+
+ // Malformed containers: dataStart + final offset overflows an int.
+ invalid(Array(arrayHeader(false, 4),
+ /* size */ 0,
+ /* offset list */ -1, -1, -1, 0x7F))
+ invalid(Array(objectHeader(false, 1, 4),
+ /* size */ 0,
+ /* offset list */ -1, -1, -1, 0x7F))
+ invalid(Array(arrayHeader(false, 1),
+ /* size */ 1,
+ /* offset list */ 0, 6,
+ /* element data */ arrayHeader(false, 4), 0, -1, -1, -1, 0x7F))
+
+ // Valid container: the final offset may cover unused bytes within the
value.
+ valid(Array(arrayHeader(false, 1),
+ /* size */ 0,
+ /* offset list */ 2,
+ /* unused data */ 0, 0))
+
+ // Malformed containers: a child does not end within the data of its
container.
+ invalid(Array(arrayHeader(false, 1),
+ /* size */ 1,
+ /* offset list */ 0, 1,
+ /* element data: 3-byte INT2 */ primitiveHeader(INT2), 1, 0))
+ invalid(Array(objectHeader(false, 1, 1),
+ /* size */ 1,
+ /* id list */ 0,
+ /* offset list */ 0, 1,
+ /* field data: 2-byte INT1 */ primitiveHeader(INT1), 1), metadata)
+ invalid(Array(arrayHeader(false, 1),
+ /* size */ 1,
+ /* offset list: the element starts at the final offset */ 1, 1,
+ /* element data */ primitiveHeader(TRUE), primitiveHeader(TRUE)))
+ invalid(Array(arrayHeader(false, 1),
+ /* size */ 1,
+ /* offset list */ 0, 1,
+ /* element data: 5-byte nested array */
+ arrayHeader(false, 1), 1, 0, 1, primitiveHeader(TRUE)))
+ invalid(Array(arrayHeader(false, 1),
+ /* size */ 1,
+ /* offset list */ 0, 2,
+ /* element data: 7-byte LONG_STR */ primitiveHeader(LONG_STR), 2, 0, 0,
0, 'a', 'b'))
+
+ // Valid container: unused bytes between children.
+ valid(Array(arrayHeader(false, 1),
+ /* size */ 2,
+ /* offset list */ 0, 2, 3,
+ /* element data */ primitiveHeader(TRUE),
+ /* unused */ 0,
+ primitiveHeader(FALSE)))
+
+ // Valid object: field data is not required to be in the order of the
offset list.
+ valid(Array(objectHeader(false, 1, 1),
+ /* size */ 2,
+ /* id list */ 0, 1,
+ /* offset list */ 2, 0, 4,
+ /* field data */ primitiveHeader(INT1), 2, primitiveHeader(INT1), 1),
metadata)
+
+ // Valid value: trailing bytes after the top-level value are ignored.
+ valid(Array(arrayHeader(false, 1),
+ /* size */ 0,
+ /* offset list */ 0,
+ /* trailing bytes */ 99))
+
+ // Variants built by `parse_json` are valid.
+ Seq(
+ "{}",
+ "[]",
+ """{"a": {}, "b": []}""",
+ """[[], {}, [1, [2, {"c": "d"}]]]""",
+ """{"a": 1, "b": {"c": [true, null, 1.5]}}""",
+ s"""["${"x" * 100}", {"k": "${"y" * 100}"}]"""
+ ).foreach { json =>
+ val v = parseJson(json)
+ valid(v.getValue, v.getMetadata)
+ }
+ val dedup = VariantExpressionEvalUtils.parseJson(
+ UTF8String.fromString("""{"a": 1, "a": {"b": [1, 2]}, "c": 3}"""),
allowDuplicateKeys = true)
+ valid(dedup.getValue, dedup.getMetadata)
+
// Unknown primitive type (type info 17 is not defined).
invalid(Array(primitiveHeader(17)))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]