[
https://issues.apache.org/jira/browse/SPARK-60711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Savic updated SPARK-60711:
---------------------------------
Description:
Using valid empty metadata [1, 0, 0], with bytes shown in decimal:
- Object value [2, 0]: currently returns true; expected false.
- Array value [3, 0]: currently returns true; expected false.
- Complete empty object [2, 0, 0]: correctly returns true.
- Complete empty array [3, 0, 0]: correctly returns true.
An object or array with n children requires n + 1 offsets. Even an empty
container requires
one final offset, equal to zero.
VariantUtil.validateImpl validates children in a loop without checking the
final offset.
When the child count is zero, the loop is skipped and the incomplete encoding
is accepted.
More generally, validateImpl never reads the final offset, so it also does not
check its value:
- Array value [3, 0, 5]: the final offset claims 5 bytes of data that are not
in the buffer;
currently returns true; expected false.
- A child that starts or ends past its container's final offset (but still
inside the buffer)
is currently accepted. Code that slices sub-values by their size
(Variant.getValue,
VariantBuilder.shallowAppendVariant) would truncate such a child, so values
reported as
valid can later fail with MALFORMED_VARIANT or be silently truncated.
Validation should:
1. Reject objects and arrays whose final offset is missing.
2. Reject objects and arrays whose data (dataStart + final offset) extends
past the buffer.
3. Reject children that do not end within their container's data.
was:
Using valid empty metadata [1, 0, 0], with bytes shown in decimal:
- Object value [2, 0]: currently returns true; expected false.
- Array value [3, 0]: currently returns true; expected false.
- Complete empty object [2, 0, 0]: correctly returns true.
- Complete empty array [3, 0, 0]: correctly returns true.
An object or array with n children requires n + 1 offsets. Even an empty
container requires one final offset, equal to zero.
VariantUtil.validateImpl validates children in a loop without checking the
final offset. When the child count is zero, the loop is skipped and the
incomplete encoding is accepted.
Validation should reject missing final offsets.
> is_valid_variant returns true when an object or array has a zero child count
> but is missing its required final offset
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60711
> URL: https://issues.apache.org/jira/browse/SPARK-60711
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Stefan Savic
> Priority: Major
>
> Using valid empty metadata [1, 0, 0], with bytes shown in decimal:
> - Object value [2, 0]: currently returns true; expected false.
> - Array value [3, 0]: currently returns true; expected false.
> - Complete empty object [2, 0, 0]: correctly returns true.
> - Complete empty array [3, 0, 0]: correctly returns true.
> An object or array with n children requires n + 1 offsets. Even an empty
> container requires
> one final offset, equal to zero.
> VariantUtil.validateImpl validates children in a loop without checking the
> final offset.
> When the child count is zero, the loop is skipped and the incomplete encoding
> is accepted.
> More generally, validateImpl never reads the final offset, so it also does
> not check its value:
> - Array value [3, 0, 5]: the final offset claims 5 bytes of data that are
> not in the buffer;
> currently returns true; expected false.
> - A child that starts or ends past its container's final offset (but still
> inside the buffer)
> is currently accepted. Code that slices sub-values by their size
> (Variant.getValue,
> VariantBuilder.shallowAppendVariant) would truncate such a child, so
> values reported as
> valid can later fail with MALFORMED_VARIANT or be silently truncated.
> Validation should:
> 1. Reject objects and arrays whose final offset is missing.
> 2. Reject objects and arrays whose data (dataStart + final offset) extends
> past the buffer.
> 3. Reject children that do not end within their container's data.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
