[
https://issues.apache.org/jira/browse/YARN-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15957484#comment-15957484
]
Daniel Tomes commented on YARN-6449:
------------------------------------
[[email protected]] That would work in theory but that would be a
global change, right? Most users are not experienced enough to do this
correctly and should not be allowed or we would surely run into issues. If we
were to allow a small group of admins/schedulers in the enterprise tweak this
for certain jobs we could fit a lot more into our clusters.
This request comes from numerous clients starting to use SPARK on YARN for ETL.
90% of the simple ETL I see does not require anywhere near a full processor to
manage the ETL pipeline.
Thanks for the quick response.
> Enable YARN to accept jobs with < 1 core allocations
> ----------------------------------------------------
>
> Key: YARN-6449
> URL: https://issues.apache.org/jira/browse/YARN-6449
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
> Reporter: Daniel Tomes
> Labels: features, performance
>
> Product Enhancement Request
> In Spark/HIVE/etc. I often need to complete work for which an entire core is
> overkill such as managing a JDBC connection or doing a simple map/transform;
> however, when I do this on large datasets, 1 core X 500 partitions/mappers
> winds up with quite the cluster level footprint even though most of those
> processor cycles are idle.
> I propose that we enable YARN to allow a user to submit jobs that "allocate <
> 1 core". Under the covers, the JVM will still receive one core but YARN/ZK
> could keep track of the fractions of cores being used and allow other jobs to
> consume the same core twice provided that both jobs were submitted with <= .5
> cores. Now, YARN can more effectively utilize multi-threading and decrease
> CPU idle for the power users.
> Obviously this can ultimately result in very bad outcomes, but if we also
> enable security controls then customers can configure such that only
> admins/gates can submit with < 1 full core and ultimately resulting in a
> cluster that can do more.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
