[
https://issues.apache.org/jira/browse/SPARK-55533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-55533:
-----------------------------------
Labels: pull-request-available (was: )
> [SQL] Support IGNORE NULLS / RESPECT NULLS for collect_set
> ----------------------------------------------------------
>
> Key: SPARK-55533
> URL: https://issues.apache.org/jira/browse/SPARK-55533
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Kent Yao
> Priority: Major
> Labels: pull-request-available
>
> Currently, collect_list/array_agg supports IGNORE NULLS and RESPECT NULLS
> syntax (added in SPARK-55256), but collect_set does not.
> collect_set always ignores null values (Hive semantics). This ticket adds
> explicit IGNORE NULLS / RESPECT NULLS support to collect_set, mirroring the
> existing collect_list behavior:
> - collect_set(expr) — default, skips nulls (unchanged behavior)
> - collect_set(expr) IGNORE NULLS — explicitly skips nulls (same as default)
> - collect_set(expr) RESPECT NULLS — includes null in the result set if present
> Implementation approach:
> - Add ignoreNulls parameter to CollectSet (mirrors CollectList pattern)
> - Wire CollectSet into FunctionResolution.applyIgnoreNulls
> - Handle null safely in eval() for BinaryType
> Note: The SQL standard defines no SET_AGG function. collect_set is
> Spark-proprietary; the closest standard equivalent is ARRAY_AGG(DISTINCT
> ...). IGNORE NULLS / RESPECT NULLS is not defined for aggregate functions in
> the standard (only for window functions), so this is a Spark extension for
> consistency across collection aggregates.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
