[
https://issues.apache.org/jira/browse/YARN-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15976956#comment-15976956
]
Carlo Curino commented on YARN-2113:
------------------------------------
[~wangda] I lack context here, as I come in late in the conversation. I don't
understand why simple dead-zones cannot avoid oscillations. Even if you change
the "target values" afterwards. This is equivalent of changing your thermostat
target temperature, something like an hysteresis (somewhat akin to a dead-zone)
typically takes care of it.
Regarding the idea of a directed graph, it seems tricky because you would have
to maintain some form of windowing semantics as well, since not all cycles
appear instantaneously, but also over time some preemption cycles will
eventually happen (and if far apart enough they are not a problem, just
naturally changing cluster conditions). Tuning this windowing semantics seems
very tricky, and I am not sure I even get what are the precise semantics it
enforces.
Overall I would stick to the simplest solution possible.
More generally, I would ask you ([~sunilg]?) to provide very crisp description
of the semantics that you target, and most importantly a crisp explanation of
how this plays with the rest of the scheduler features (e.g., node-labels /
non-preemptible queues / delay-scheduling). It is ok to say feature A doesn't
work with feature B, because they break each-other invariants, but this should
be captured clearly.
I suggest to look at YARN-6451, and you use those ideas plus SLS to build
tests that ensure that whatever invariant you have in mind is enforced and
maintained by the implementation.
> Add cross-user preemption within CapacityScheduler's leaf-queue
> ---------------------------------------------------------------
>
> Key: YARN-2113
> URL: https://issues.apache.org/jira/browse/YARN-2113
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Sunil G
> Attachments:
> TestNoIntraQueuePreemptionIfBelowUserLimitAndDifferentPrioritiesWithExtraUsers.txt,
> YARN-2113.0001.patch, YARN-2113.0002.patch, YARN-2113.0003.patch,
> YARN-2113.0004.patch, YARN-2113.0005.patch, YARN-2113.0006.patch,
> YARN-2113.0007.patch, YARN-2113.v0.patch
>
>
> Preemption today only works across queues and moves around resources across
> queues per demand and usage. We should also have user-level preemption within
> a queue, to balance capacity across users in a predictable manner.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
