[
https://issues.apache.org/jira/browse/SPARK-60711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Savic updated SPARK-60711:
---------------------------------
Summary: is_valid_variant does not validate the final offset of objects and
arrays (was: is_valid_variant returns true when an object or array has a zero
child count but is missing its required final offset)
> is_valid_variant does not validate the final offset of objects and arrays
> -------------------------------------------------------------------------
>
> Key: SPARK-60711
> URL: https://issues.apache.org/jira/browse/SPARK-60711
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Stefan Savic
> Priority: Major
>
> Using valid empty metadata [1, 0, 0], with bytes shown in decimal:
> - Object value [2, 0]: currently returns true; expected false.
> - Array value [3, 0]: currently returns true; expected false.
> - Complete empty object [2, 0, 0]: correctly returns true.
> - Complete empty array [3, 0, 0]: correctly returns true.
> An object or array with n children requires n + 1 offsets. Even an empty
> container requires
> one final offset, equal to zero.
> VariantUtil.validateImpl validates children in a loop without checking the
> final offset.
> When the child count is zero, the loop is skipped and the incomplete encoding
> is accepted.
> More generally, validateImpl never reads the final offset, so it also does
> not check its value:
> - Array value [3, 0, 5]: the final offset claims 5 bytes of data that are
> not in the buffer;
> currently returns true; expected false.
> - A child that starts or ends past its container's final offset (but still
> inside the buffer)
> is currently accepted. Code that slices sub-values by their size
> (Variant.getValue,
> VariantBuilder.shallowAppendVariant) would truncate such a child, so
> values reported as
> valid can later fail with MALFORMED_VARIANT or be silently truncated.
> Validation should:
> 1. Reject objects and arrays whose final offset is missing.
> 2. Reject objects and arrays whose data (dataStart + final offset) extends
> past the buffer.
> 3. Reject children that do not end within their container's data.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
