[
https://issues.apache.org/jira/browse/SPARK-61812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anil K updated SPARK-61812:
---------------------------
Affects Version/s: 4.3.0
(was: 5.0.0)
> Storage-partitioned join passes a decimal argument of more precision than its
> partition transform declares, which the write path's cast can change
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61812
> URL: https://issues.apache.org/jira/browse/SPARK-61812
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Anil K
> Priority: Major
>
> A storage-partitioned join evaluates a connector's partition transform itself
> in the identity reduce and the one-side shuffle. Since SPARK-60027, it takes
> an argument as it is when it is stored the way the function's declared input
> type is ({{TransformExpression.storedAlike}}). For a top-level decimal, that
> holds for any precision at the declared scale, so a {{DECIMAL(12,2)}} column
> under a function declaring {{DECIMAL(10,2)}} is passed as it is.
> The write path ({{DistributionAndOrderingUtils}}) lets type coercion cast the
> argument to the declared type, and both modes insert that cast
> ({{TypeCoercion.implicitCast}} takes any numeric to a numeric target;
> {{AnsiTypeCoercion}} allows it by store assignment). For a value of more
> digits than the declared precision, the two then differ. Measured on
> {{1234567890.12}} as a {{DECIMAL(12,2)}} cast to {{DECIMAL(10,2)}}:
> * {{spark.sql.ansi.enabled=false}}: the cast gives {{NULL}}, so the write
> path places the row by {{f(NULL)}}, while the identity reduce or a one-side
> shuffle places the other side's {{1234567890.12}} by {{f(1234567890.12)}},
> and the join can miss the match;
> * {{spark.sql.ansi.enabled=true}}: the cast throws
> {{SparkArithmeticException}}, while the storage-partitioned join evaluates
> the function on the raw value.
> Values that fit the declared precision, e.g. {{12.34}}, are unchanged by the
> cast, so this only affects values that overflow it.
> This is on master, and outside SPARK-60813 and SPARK-61650, which cover
> datetime and interval types, but it breaks the rule they state: a function
> gets the value the write path's cast gives. The top-level exemption exists
> for a cogroup key of more precision against a narrower bucket column (a
> {{DECIMAL(12,2)}} key against a {{DECIMAL(10,2)}} column), which a cogroup
> that does not coerce its keys can produce. Since
> https://github.com/apache/spark/pull/60431, such a key is marked on the
> transform rebuilt over it ({{TransformExpression.storedKey}}). So a fix could
> keep the exemption for a marked transform only, and require {{f.precision <=
> t.precision}} otherwise, so that a join over such a column shuffles both
> sides.
> Raised in the review of https://github.com/apache/spark/pull/60431
> (https://github.com/apache/spark/pull/60431#discussion_r4584464865).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]