Stefan Savic created SPARK-60713:
------------------------------------
Summary: Report the actual size limit in
VARIANT_CONSTRUCTOR_SIZE_LIMIT
Key: SPARK-60713
URL: https://issues.apache.org/jira/browse/SPARK-60713
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Stefan Savic
VARIANT_CONSTRUCTOR_SIZE_LIMIT hardcodes 16 MiB in its error message:
Cannot construct a Variant larger than 16 MiB. The maximum allowed size of a
Variant value is 16 MiB.
SPARK-52181 increased the normal Variant size limit from 16 MiB to 128 MiB:
https://github.com/apache/spark/pull/50913
In current OSS Spark, VariantUtil.SIZE_LIMIT is 128 MiB during normal execution
and 16 MiB during tests. The Variant constructor checks the value and metadata
arrays against this constant, but the error message does not use it. When the
Variant value or metadata exceeds 128 MiB, the error incorrectly says the limit
is 16 MiB. The lower test limit masks this mismatch.
The message should report the actual enforced limit. Replace the hardcoded size
with a sizeLimit parameter populated from VariantUtil.SIZE_LIMIT and update the
affected error factories and test expectations. This should preserve the
existing enforced limits and error class.
VARIANT_SIZE_LIMIT already accepts a sizeLimit parameter;
this issue concerns VARIANT_CONSTRUCTOR_SIZE_LIMIT.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
