Github user AnthonyTruchet commented on the issue:
https://github.com/apache/spark/pull/16038
Actually aggregating (and thus sending on the network) on quite dense
SparseVectors with 10s of million elements is not to taken lightly. This would
required serious benchmarking. What I tell is that we can not do that lightly
because we know that aggregating `acc += partial` when `partial` is sparse
and `acc` dense is efficient locally (i.e. within a partition) and that after a
partition has been aggregated the vector is already quite dense (enough so that
the dense and sparse representation have similar size).
Anyway if you consider this is not worth an new API and an optimization in
core, the discusion should probably go on on the MLlib only PR #16037 for which
we already have a specific work around.
I just naively assumed that aggregating over big accumulator was typical
enough to get some support from core, if this is not the case, I'll fall back
on a MLlib only workaround.
Feel free to close this PR is you deem it (ir)relevant.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
