Zoltán Borók-Nagy created IMPALA-15611:
------------------------------------------
Summary: variant_get() cannot convert VARIANT decimal values to
DOUBLE, FLOAT or integer types
Key: IMPALA-15611
URL: https://issues.apache.org/jira/browse/IMPALA-15611
Project: IMPALA
Issue Type: Sub-task
Reporter: Zoltán Borók-Nagy
Assignee: Zoltán Borók-Nagy
h3. Problem
The typed form of {{variant_get(v, path, type)}} rejects VARIANT values of the
decimal physical types ({{DECIMAL4}}, {{DECIMAL8}}, {{DECIMAL16}}) for every
numeric target type. For example, a decimal read as {{'double'}} fails with:
{noformat}
UDF ERROR: variant_get: value at path is not convertible to DOUBLE
{noformat}
{{try_variant_get}} returns NULL instead.
Decimals are common in VARIANT data: Spark's {{parse_json}} stores a JSON
number with a fractional part as a decimal when it fits. Engines that follow
Spark's {{variant_get}} semantics (Spark, and DuckDB, Hive, StarRocks and Trino
in the test
below) return the converted value.
h3. Steps to reproduce
Write a VARIANT with a decimal value from Spark (Iceberg v3 table), then read
it with Impala:
{code:sql}
-- Spark 4.1 + Iceberg 1.11
CREATE TABLE t (id INT, v VARIANT) USING iceberg TBLPROPERTIES
('format-version' = '3');
INSERT INTO t VALUES (1, parse_json('{"price": 12.34}')); -- 12.34 is stored
as a VARIANT decimal
-- Impala
SELECT variant_get(v, '$.price', 'double') FROM t; -- ERROR: variant_get:
value at path is not convertible to DOUBLE
SELECT try_variant_get(v, '$.price', 'double') FROM t; -- NULL
{code}
h3. Expected
{{12.34}} for {{'double'}} (and {{'float'}}), as Spark's {{variant_get}} casts
decimal values to the requested numeric type. For the integer targets, the
result should follow the same cast rules Impala applies to other numeric
VARIANT values
(Spark casts decimals to integral types too).
h3. Actual
An error for {{variant_get}} and NULL for {{try_variant_get}}, for every
numeric target type.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
