Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/8592#issuecomment-140244709
This is not premature optimization. Just run collect_set on a large set and
Spark JVM processes will OOM.
What I'm suggesting is to implement this not as an aggregate function, but
rather an operator that does the function of collect_set (e.g. the operator can
sort all the data by grouping key and set key to compute the set for each
group).
Anyway - maybe it'd make sense also have the aggregate function version of
this, with the caveat written correctly that this would OOM if running on large
amount of data.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
