Daniel Tomes created YARN-6449:
----------------------------------
Summary: Enable YARN to accept jobs with < 1 core allocations
Key: YARN-6449
URL: https://issues.apache.org/jira/browse/YARN-6449
Project: Hadoop YARN
Issue Type: Bug
Components: yarn
Reporter: Daniel Tomes
Product Enhancement Request
In Spark/HIVE/etc. I often need to complete work for which an entire core is
overkill such as managing a JDBC connection or doing a simple map/transform;
however, when I do this on large datasets, 1 core X 500 partitions/mappers
winds up with quite the cluster level footprint even though most of those
processor cycles are idle.
I propose that we enable YARN to allow a user to submit jobs that "allocate < 1
core". Under the covers, the JVM will still receive one core but YARN/ZK could
keep track of the fractions of cores being used and allow other jobs to consume
the same core twice provided that both jobs were submitted with <= .5 cores.
Now, YARN can more effectively utilize multi-threading and decrease CPU idle
for the power users.
Obviously this can ultimately result in very bad outcomes, but if we also
enable security controls then customers can configure such that only
admins/gates can submit with < 1 full core and ultimately resulting in a
cluster that can do more.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
