Anil K created SPARK-61812:
------------------------------
Summary: Storage-partitioned join passes a decimal argument of
more precision than its partition transform declares, which the write path's
cast can change
Key: SPARK-61812
URL: https://issues.apache.org/jira/browse/SPARK-61812
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Anil K
A storage-partitioned join evaluates a connector's partition transform itself
in the identity reduce and the one-side shuffle. Since SPARK-60027, it takes an
argument as it is when it is stored the way the function's declared input type
is ({{TransformExpression.storedAlike}}). For a top-level decimal, that holds
for any precision at the declared scale, so a {{DECIMAL(12,2)}} column under a
function declaring {{DECIMAL(10,2)}} is passed as it is.
The write path ({{DistributionAndOrderingUtils}}) lets type coercion cast the
argument to the declared type, and both modes insert that cast
({{TypeCoercion.implicitCast}} takes any numeric to a numeric target;
{{AnsiTypeCoercion}} allows it by store assignment). For a value of more digits
than the declared precision, the two then differ. Measured on {{1234567890.12}}
as a {{DECIMAL(12,2)}} cast to {{DECIMAL(10,2)}}:
* {{spark.sql.ansi.enabled=false}}: the cast gives {{NULL}}, so the write path
places the row by {{f(NULL)}}, while the identity reduce or a one-side shuffle
places the other side's {{1234567890.12}} by {{f(1234567890.12)}}, and the join
can miss the match;
* {{spark.sql.ansi.enabled=true}}: the cast throws
{{SparkArithmeticException}}, while the storage-partitioned join evaluates the
function on the raw value.
Values that fit the declared precision, e.g. {{12.34}}, are unchanged by the
cast, so this only affects values that overflow it.
This is on master, and outside SPARK-60813 and SPARK-61650, which cover
datetime and interval types, but it breaks the rule they state: a function gets
the value the write path's cast gives. The top-level exemption exists for a
cogroup key of more precision against a narrower bucket column (a
{{DECIMAL(12,2)}} key against a {{DECIMAL(10,2)}} column), which a cogroup that
does not coerce its keys can produce. Since
https://github.com/apache/spark/pull/60431, such a key is marked on the
transform rebuilt over it ({{TransformExpression.storedKey}}). So a fix could
keep the exemption for a marked transform only, and require {{f.precision <=
t.precision}} otherwise, so that a join over such a column shuffles both sides.
Raised in the review of https://github.com/apache/spark/pull/60431
(https://github.com/apache/spark/pull/60431#discussion_r4584464865).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]