Github user yanboliang commented on the issue:
https://github.com/apache/spark/pull/18798
@WeichenXu123 @thunterdb
Thanks for this great work, we are always happy to see improvement which
can help us to migrate MLlib workload to Dataset based API.
Here are my two cents:
1, Form the performance test result, there are performance degradation from
3 -> 2 -> 1.
The difference between case 3 and case 2 is the deserialization cost of
each instance, I suspect we can't skip this step, so it maybe more hard to
optimize it. However, the difference between case 2 and case 1 is the aggregate
implementation. The aggregate on Dataset will use user-defined type as
aggregate buffer, so I think we should make some effort to improve
```TypedImperativeAggregate```. I guess it maybe not so hard compared with the
previous bottleneck, but this is just my intuition, letâs check with SQL
guys. cc @liancheng @cloud-fan @hvanhovell
2, @WeichenXu123 In the current experiment, I saw you only use two
partitions. Is there any difference if we run against more partitions or more
dataset?
3, @thunterdb I agree that we can get this in and make improvement
continuously. When we get desirable result, we can start to migrate other MLlib
workload on top of Dataset/catalysts. But we should add comment to let users
know(if we canât get desirable result before
2.3.0) that the performance of Dataset-based multivariate statistic is not
good enough currently. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
