Github user codedeft commented on the pull request:
https://github.com/apache/spark/pull/2868#issuecomment-61170031
Hm, I see. I'll try testing again on the small mnist but my previous test
was on a cluster with 8 executors. However, I realize now that it probably only
utilized 2 our of 8 executors (seems like reducebykey that's used doesn't
really use extra executors?).
In addition to being useful for deep trees and local-training, I do think
that another usefulness of node-Id-cache is if you want to write the model
directly to disk without keeping them in memory.
So even if we may not see any performance benefit for now, I do think that
we'll need them later. So I guess that the decision is really up to you on
whether to include this now or later. It's just that you probably don't want to
have it hanging for too long. As long as it gets in soon after release, I'm ok.
I'm currently running deep tree test, albeit with only 10 trees on mnist8m.
I'll see if I can see some benefit here at least.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
