sepuri sai krishna created SPARK-61239:
------------------------------------------
Summary: Map subscript casts an integral key to a narrower map key
type and returns the value of a different key
Key: SPARK-61239
URL: https://issues.apache.org/jira/browse/SPARK-61239
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.0.3, 4.4.0, 5.0.0
Reporter: sepuri sai krishna
{{m[key]}} (also {{col("m")(key)}} and {{getItem}}) casts the key to the key
type of the map. When the map key type is a narrower integral type than the
key, this cast wraps around with ANSI mode off, so the lookup returns the value
of a different key, and fails with ANSI mode on, although the key is simply not
in the map. {{element_at}} widens the map keys instead and returns NULL.
{code:sql}
CREATE TABLE t USING parquet AS
SELECT id, map(CAST(44 AS TINYINT), 'v') AS m, map(44, 'iv') AS mi FROM
range(1);
SET spark.sql.ansi.enabled=false;
SELECT m[300], mi[4294967340L], element_at(m, 300), element_at(mi, 4294967340L)
FROM t;
-- v iv NULL NULL (expected NULL for all four)
{code}
300 is cast to {{TINYINT}} and wraps to 44, and 4294967340 is cast to {{INT}}
and wraps to 44. The same happens with a non-literal key, for example
{{m[CAST(300 AS INT) + id * 0]}}. With {{spark.sql.ansi.enabled=true}} (the
default), {{m[300]}} and {{mi[4294967340L]}} fail:
{noformat}
[CAST_OVERFLOW] The value 300 of the type "INT" cannot be cast to "TINYINT" due
to an overflow. Use `try_cast` to tolerate overflow and return NULL instead.
{noformat}
Checked on 4.0.3 and master.
{{GetMapValue.inputTypes}} is {{Seq(MapType, keyType)}}, so the implicit type
coercion casts the key to the map key type. {{ElementAt.inputTypes}} uses
{{TypeCoercion.findTightestCommonType(keyType, key type)}} and casts the map
and the key to that type.
A key of a fractional type is truncated the same way ({{mi[44.5D]}} returns
{{iv}}, where {{element_at}} returns NULL), which may be worth looking at
separately.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
