Aleksandr Chernousov created SPARK-59151:
--------------------------------------------
Summary: Variant aggregation- canonicalisation
Key: SPARK-59151
URL: https://issues.apache.org/jira/browse/SPARK-59151
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 5.0.0
Reporter: Aleksandr Chernousov
Implement canonicalization (+ fastpath aka checking if Variant is already
canonicalized)
This part contains only canonicalization, without wiring to expressions
After canonicalization Variant should have
- All dict keys sorted
- No unused keys
- All numeric types normalized to smallest fitting datatype (e.g.: [10 as
int64] and [10.0 as decimal] to 10 as int8
- float/double -0.0 converted to 0.0
- Have all NaNs in normalised form
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
