Github user brkyvz commented on the issue:
https://github.com/apache/spark/pull/15730
Hi @WeichenXu123 Thank you for this PR. Sorry for taking so long to get
back to you. Your optimization would be very helpful. I have a couple thoughts
though. Your examples always take into account fully dense matrices, i.e. that
all blocks exist all the time. How would sparsity affect shuffling? Would there
ever be a case where sparsity of blocks and unlucky alignment of blocks could
actually cause a lot more shuffling with your parameter?
Nevertheless, I can see fully dense matrix multiplications benefitting
significantly from your optimization. I guess we will need to work on the APIs
a bit and document it a bit more clearly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
