Github user jdesmet commented on the pull request:
https://github.com/apache/spark/pull/9095#issuecomment-216016788
Humbly, I think I understood what this PR was about. I probably (still) do
not understand some of the reasoning as to why we can't report the correct
vCores even if the default resource calculator does not support it, and vCores
is not used. The thread seemed to suggest it is possible, and was actually
attempted in some modifications that were undone. Don't take it as I am saying
it's wrong, it is probably just that you have a better understanding of it.
However nothing against documenting it further?
Also as to confirm - making sure I am not misunderstanding anything, as per
the threads and documentation, to get it to work based on vCore resource
allocation, following steps need to be accomplished:
1. Use the CapacityScheduler: in `conf/yarn-site.xml`, set
`yarn.resourcemanager.scheduler.class` to
`org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler`.
2. Modify the resource-calculator, to one that supports using vCores: set
`yarn.scheduler.capacity.resource-calculator` to
`org.apache.hadoop.yarn.util.resource.DominantResourceCalculator`.
Probably we need to file a bug to get the hadoop documentation fixed from
`DefaultResourseCalculator` to `DefaultResourceCalculator`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
