[
https://issues.apache.org/jira/browse/YARN-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15623344#comment-15623344
]
Karthik Kambatla commented on YARN-5783:
----------------------------------------
bq. If you're assuming a single-threaded context, there's no need to null out
the appBeingProcessed in take().
By nulling out before the blocking call, we are avoiding a match when adding a
starved app.
For the scenario outlined, appBeingProcessed is set to app1 only when the
preemption thread is processing it trying to identify nodes that would match.
As soon as that processing is done, appBeingProcessed is reset to null. If the
app continues to be starved, the update thread attempts to queue it. The app is
added to FSStarvedApps any time it is not actively processed by the preemption
thread.
bq. I don't see where tracking the app being processed is needed for the tests.
While implementing the test, I noticed a couple of things - (1) preemption
kicks in fast enough that it does not make much sense to look at the latest
snapshot of starved apps, (2) The app is added multiple times if we don't track
appBeingProcessed.
In general, since the first patch has no tests, adding more tests will lead to
finding issues that need to be fixed.
Also, this branch is weird in that this could just be one JIRA instead of
multiple JIRAs/commits. The branch is only so the development is in the open
and reviews stay small. For the merge, I would likely want to commit the entire
branch as a single commit. So, as long as the final output is reasonable, I
wouldn't worry too much about which JIRA adds what.
> Unit tests to verify the identification of starved applications
> ---------------------------------------------------------------
>
> Key: YARN-5783
> URL: https://issues.apache.org/jira/browse/YARN-5783
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler
> Affects Versions: 2.8.0
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Labels: oct16-medium
> Attachments: yarn-5783.YARN-4752.1.patch,
> yarn-5783.YARN-4752.2.patch, yarn-5783.YARN-4752.3.patch
>
>
> JIRA to track unit tests to verify the identification of starved
> applications. An application should be marked starved only when:
> # Cluster allocation is over the configured threshold for preemption.
> # Preemption is enabled for a queue and any of the following:
> ## The queue is under its minshare for longer than minsharePreemptionTimeout
> ## One of the queue’s applications is under its fairshare for longer than
> fairsharePreemptionTimeout.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
