ChuckLin2025 commented on PR #57592:
URL: https://github.com/apache/spark/pull/57592#issuecomment-5100521310
Hi @cloud-fan @srielau, could you help me review this PR.
Today for this kind of query:
select key1, key2, array_distinct(flatten(collect_list(array_col1))),
array_distinct(flatten(collect_list(array_col2))) FROM ...
Spark can't run it in an effiecnt way, we have to buffer a huge
array_of_array and perform array distinct on it later. Also it can be rewrote
by collect_set in a graceful way. An additional join cost need to be paid, like
```
with col1 as (
select key1, key2, collect_set(col) as distinct_col1
FROM ... LATERAL VIEW explode(array_col1) e AS col
),
col2 as (
select key1, key2, collect_set(col) as distinct_col2
FROM ... LATERAL VIEW explode(array_col2) e AS col
)
select key1, key2, distinct_col1, distinct_col2
from col1 join col2
using (key1, key2)
```
So I propose to introdue this `collect_union` function to the `collect`
familiy. Also this is already supported in googleSQL by
ARRAY_CONCAT_AGG(distinct ...).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
