Kent Yao created SPARK-55533:
--------------------------------
Summary: [SQL] Support IGNORE NULLS / RESPECT NULLS for collect_set
Key: SPARK-55533
URL: https://issues.apache.org/jira/browse/SPARK-55533
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.2.0
Reporter: Kent Yao
Currently, collect_list/array_agg supports IGNORE NULLS and RESPECT NULLS
syntax (added in SPARK-55256), but collect_set does not.
collect_set always ignores null values (Hive semantics). This ticket adds
explicit IGNORE NULLS / RESPECT NULLS support to collect_set, mirroring the
existing collect_list behavior:
- collect_set(expr) — default, skips nulls (unchanged behavior)
- collect_set(expr) IGNORE NULLS — explicitly skips nulls (same as default)
- collect_set(expr) RESPECT NULLS — includes null in the result set if present
Implementation approach:
- Add ignoreNulls parameter to CollectSet (mirrors CollectList pattern)
- Wire CollectSet into FunctionResolution.applyIgnoreNulls
- Handle null safely in eval() for BinaryType
Note: The SQL standard defines no SET_AGG function. collect_set is
Spark-proprietary; the closest standard equivalent is ARRAY_AGG(DISTINCT ...).
IGNORE NULLS / RESPECT NULLS is not defined for aggregate functions in the
standard (only for window functions), so this is a Spark extension for
consistency across collection aggregates.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
