This is an automated email from the ASF dual-hosted git repository.

uros-b pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 6a021f1ef425 [SPARK-60711][SQL] `is_valid_variant` does not validate 
the final offset of objects and arrays
6a021f1ef425 is described below

commit 6a021f1ef425d892ce6e70135f9cb205cc5c8eb8
Author: bajka <[email protected]>
AuthorDate: Thu Nov 5 14:55:21 2026 +0000

    [SPARK-60711][SQL] `is_valid_variant` does not validate the final offset of 
objects and arrays
    
    ### What changes were proposed in this pull request?
    
    `is_valid_variant` now checks the final offset of every object and array. 
It returns false when:
      1. The final offset is missing (an empty array `[3, 0]` or an empty 
object `[2, 0]`).
      2. The final offset points past the end of the value (`[3, 0, 5]`).
      3. A child does not end within its container's data.
    
    Unused bytes inside a containers data are still allowed.
    
    ### Why are the changes needed?
    
    `is_valid_variant` returned true for malformed variants. A container with n 
children needs n + 1 offsets, but the final offset was never read, so an empty 
array or object without it was accepted and its value was not checked either, 
so a container or child extending past its bounds was also accepted. Such 
values can later fail with `MALFORMED_VARIANT` or be silently truncated when 
subvalues are extracted.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `is_valid_variant` now returns false for the malformed variants 
described above, where it previously returned true. Valid variants are not 
affected.
    
    ### How was this patch tested?
    
    Added cases to `VariantExpressionSuite` "is_valid_variant":
      - Final offset missing: empty array, empty object, truncated 4-byte 
offset, 4-byte size header, nested empty array.
      - Final offset past the end of the value: empty array and empty object 
with final offset 5.
      - Child not ending within its container: array element, object field, 
element starting at the final offset, nested array, long string.
      - Still valid: unused bytes in container data, unused bytes between 
children, object fields not in offset order, trailing bytes after the top-level 
value, complete empty containers with 4-byte offset and 4-byte size headers.
      - Variants built by `parse_json` (nested, empty, long strings, duplicate 
keys) are valid.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Claude Opus 5.5)
    
    Closes #59927 from SavicStefan/effort/isvalidvariant.
    
    Lead-authored-by: bajka <[email protected]>
    Co-authored-by: Stefan Savić <[email protected]>
    Signed-off-by: Uros Bojanic <[email protected]>
    (cherry picked from commit 52357949c6d43209a16ff991359e31fa4d2118e5)
    Signed-off-by: Uros Bojanic <[email protected]>
---
 .../apache/spark/types/variant/VariantUtil.java    |  24 ++++-
 .../variant/VariantExpressionSuite.scala           | 109 +++++++++++++++++++++
 2 files changed, 129 insertions(+), 4 deletions(-)

diff --git 
a/common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java 
b/common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java
index 184a7b29a978..64044ad49eb7 100644
--- 
a/common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java
+++ 
b/common/variant/src/main/java/org/apache/spark/types/variant/VariantUtil.java
@@ -605,10 +605,12 @@ public class VariantUtil {
   // Validate whether a variant is well-formed. Returns true if the variant 
binary is structurally
   // well-formed (all bounds and type-info checks pass), false if it is 
malformed.
   //
-  // This is close to, but not strictly equivalent to, "`toJson` does not 
throw": this function
-  // does not enforce the `SIZE_LIMIT` check that the `Variant` constructor 
applies (which throws
-  // `VARIANT_CONSTRUCTOR_SIZE_LIMIT`). The implementation otherwise has the 
same structure as
-  // `toJson` (see `Variant.toJsonImpl`).
+  // This is close to, but not strictly equivalent to, "`toJson` does not 
throw". It is more
+  // lenient in that it does not enforce the `SIZE_LIMIT` check that the 
`Variant` constructor
+  // applies (which throws `VARIANT_CONSTRUCTOR_SIZE_LIMIT`). It is stricter 
in that it requires
+  // every object and array to have its final offset, and its data and 
children to lie within
+  // their bounds. The implementation otherwise has the same structure as 
`toJson` (see
+  // `Variant.toJsonImpl`).
   //
   // Implementation note: this `try { ... } catch (SparkRuntimeException e)` 
is sound only because
   // every helper invoked by `validateImpl` throws `MALFORMED_VARIANT` /
@@ -630,22 +632,36 @@ public class VariantUtil {
     switch (getType(value, pos)) {
       case OBJECT:
         handleObject(value, pos, (size, idSize, offsetSize, idStart, 
offsetStart, dataStart) -> {
+          // The offset list has `size + 1` entries. The last one is the size 
of the field data,
+          // which must lie within `value`.
+          int dataSize = readUnsigned(value, offsetStart + offsetSize * size, 
offsetSize);
+          if ((long) dataStart + dataSize > value.length) throw 
malformedVariant();
           for (int i = 0; i < size; ++i) {
             int id = readUnsigned(value, idStart + idSize * i, idSize);
             int offset = readUnsigned(value, offsetStart + offsetSize * i, 
offsetSize);
             int elementPos = dataStart + offset;
             getMetadataKey(metadata, id);
             validateImpl(value, metadata, elementPos);
+            // Sub-values are sliced by `valueSize` (e.g. `Variant.getValue`), 
so a field that
+            // extends past the field data of the object would be truncated.
+            if ((long) offset + valueSize(value, elementPos) > dataSize) throw 
malformedVariant();
           }
           return null;
         });
         break;
       case ARRAY:
         handleArray(value, pos, (size, offsetSize, offsetStart, dataStart) -> {
+          // The offset list has `size + 1` entries. The last one is the size 
of the element data,
+          // which must lie within `value`.
+          int dataSize = readUnsigned(value, offsetStart + offsetSize * size, 
offsetSize);
+          if ((long) dataStart + dataSize > value.length) throw 
malformedVariant();
           for (int i = 0; i < size; ++i) {
             int offset = readUnsigned(value, offsetStart + offsetSize * i, 
offsetSize);
             int elementPos = dataStart + offset;
             validateImpl(value, metadata, elementPos);
+            // Sub-values are sliced by `valueSize` (e.g. `Variant.getValue`), 
so an element that
+            // extends past the element data of the array would be truncated.
+            if ((long) offset + valueSize(value, elementPos) > dataSize) throw 
malformedVariant();
           }
           return null;
         });
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionSuite.scala
index c298278106d8..e23857541b14 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionSuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionSuite.scala
@@ -1468,6 +1468,115 @@ class VariantExpressionSuite extends SparkFunSuite with 
ExpressionEvalHelper {
       /* offset list */ 0, 2,
       /* field data: INT8 with only 1 byte */ primitiveHeader(INT8), 0), 
metadata)
 
+    // Malformed containers: the final offset is missing. A container with n 
children requires
+    // n + 1 offsets, so even an empty container requires one.
+    invalid(Array(arrayHeader(false, 1),
+      /* size */ 0))
+    invalid(Array(objectHeader(false, 1, 1),
+      /* size */ 0))
+    invalid(Array(arrayHeader(false, 4),
+      /* size */ 0,
+      /* truncated offset list */ 0, 0))
+    valid(Array(arrayHeader(false, 4),
+      /* size */ 0,
+      /* offset list */ 0, 0, 0, 0))
+    invalid(Array(arrayHeader(true, 1),
+      /* size */ 0, 0, 0, 0))
+    valid(Array(arrayHeader(true, 1),
+      /* size */ 0, 0, 0, 0,
+      /* offset list */ 0))
+    invalid(Array(arrayHeader(false, 1),
+      /* size */ 1,
+      /* offset list */ 0, 2,
+      /* element data: empty array without its offset list */ 
arrayHeader(false, 1), 0))
+
+    // Malformed containers: the final offset points past the end of the value.
+    invalid(Array(arrayHeader(false, 1),
+      /* size */ 0,
+      /* offset list */ 5))
+    invalid(Array(objectHeader(false, 1, 1),
+      /* size */ 0,
+      /* offset list */ 5))
+
+    // Malformed containers: dataStart + final offset overflows an int.
+    invalid(Array(arrayHeader(false, 4),
+      /* size */ 0,
+      /* offset list */ -1, -1, -1, 0x7F))
+    invalid(Array(objectHeader(false, 1, 4),
+      /* size */ 0,
+      /* offset list */ -1, -1, -1, 0x7F))
+    invalid(Array(arrayHeader(false, 1),
+      /* size */ 1,
+      /* offset list */ 0, 6,
+      /* element data */ arrayHeader(false, 4), 0, -1, -1, -1, 0x7F))
+
+    // Valid container: the final offset may cover unused bytes within the 
value.
+    valid(Array(arrayHeader(false, 1),
+      /* size */ 0,
+      /* offset list */ 2,
+      /* unused data */ 0, 0))
+
+    // Malformed containers: a child does not end within the data of its 
container.
+    invalid(Array(arrayHeader(false, 1),
+      /* size */ 1,
+      /* offset list */ 0, 1,
+      /* element data: 3-byte INT2 */ primitiveHeader(INT2), 1, 0))
+    invalid(Array(objectHeader(false, 1, 1),
+      /* size */ 1,
+      /* id list */ 0,
+      /* offset list */ 0, 1,
+      /* field data: 2-byte INT1 */ primitiveHeader(INT1), 1), metadata)
+    invalid(Array(arrayHeader(false, 1),
+      /* size */ 1,
+      /* offset list: the element starts at the final offset */ 1, 1,
+      /* element data */ primitiveHeader(TRUE), primitiveHeader(TRUE)))
+    invalid(Array(arrayHeader(false, 1),
+      /* size */ 1,
+      /* offset list */ 0, 1,
+      /* element data: 5-byte nested array */
+      arrayHeader(false, 1), 1, 0, 1, primitiveHeader(TRUE)))
+    invalid(Array(arrayHeader(false, 1),
+      /* size */ 1,
+      /* offset list */ 0, 2,
+      /* element data: 7-byte LONG_STR */ primitiveHeader(LONG_STR), 2, 0, 0, 
0, 'a', 'b'))
+
+    // Valid container: unused bytes between children.
+    valid(Array(arrayHeader(false, 1),
+      /* size */ 2,
+      /* offset list */ 0, 2, 3,
+      /* element data */ primitiveHeader(TRUE),
+      /* unused */ 0,
+      primitiveHeader(FALSE)))
+
+    // Valid object: field data is not required to be in the order of the 
offset list.
+    valid(Array(objectHeader(false, 1, 1),
+      /* size */ 2,
+      /* id list */ 0, 1,
+      /* offset list */ 2, 0, 4,
+      /* field data */ primitiveHeader(INT1), 2, primitiveHeader(INT1), 1), 
metadata)
+
+    // Valid value: trailing bytes after the top-level value are ignored.
+    valid(Array(arrayHeader(false, 1),
+      /* size */ 0,
+      /* offset list */ 0,
+      /* trailing bytes */ 99))
+
+    // Variants built by `parse_json` are valid.
+    Seq(
+      "{}",
+      "[]",
+      """{"a": {}, "b": []}""",
+      """[[], {}, [1, [2, {"c": "d"}]]]""",
+      """{"a": 1, "b": {"c": [true, null, 1.5]}}""",
+      s"""["${"x" * 100}", {"k": "${"y" * 100}"}]"""
+    ).foreach { json =>
+      val v = parseJson(json)
+      valid(v.getValue, v.getMetadata)
+    }
+    val dedup = VariantExpressionEvalUtils.parseJson(
+      UTF8String.fromString("""{"a": 1, "a": {"b": [1, 2]}, "c": 3}"""), 
allowDuplicateKeys = true)
+    valid(dedup.getValue, dedup.getMetadata)
+
     // Unknown primitive type (type info 17 is not defined).
     invalid(Array(primitiveHeader(17)))
 


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to