[jira] [Updated] (YARN-5938) Minor refactoring to OpportunisticContainerAllocatorAMService
[ https://issues.apache.org/jira/browse/YARN-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5938: -- Attachment: YARN-5938-YARN-5085.003.patch Updating patch after rebasing with trunk. > Minor refactoring to OpportunisticContainerAllocatorAMService > - > > Key: YARN-5938 > URL: https://issues.apache.org/jira/browse/YARN-5938 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5938-YARN-5085.001.patch, > YARN-5938-YARN-5085.002.patch, YARN-5938-YARN-5085.003.patch, > YARN-5938.001.patch > > > Minor code re-organization to do the following: > # The OpportunisticContainerAllocatorAMService currently allocates outside > the ApplicationAttempt lock maintained by the ApplicationMasterService. This > should happen inside the lock. > # Refactored out some code to simplify the allocate() method. > # Removed some unused fields inside the OpportunisticContainerAllocator. > # Re-organized some of the code in the > OpportunisticContainerAllocatorAMService::allocate method to make it a bit > more readable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5085) Add support for change of container ExecutionType
[
https://issues.apache.org/jira/browse/YARN-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-5085:
--
Attachment: YARN-5085.002.patch
Re-attaching again
> Add support for change of container ExecutionType
> -
>
> Key: YARN-5085
> URL: https://issues.apache.org/jira/browse/YARN-5085
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-5085.001.patch, YARN-5085.002.patch,
> YARN-5085.002.patch
>
>
> YARN-2882 introduced the concept of {{ExecutionType}} for containers and it
> also introduced the concept of OPPORTUNISTIC ExecutionType.
> YARN-4335 introduced changes to the ResourceRequest so that AMs may request
> that the Container allocated against the ResourceRequest is of a particular
> {{ExecutionType}}.
> This JIRA proposes to provide support for the AM to change the ExecutionType
> of a previously requested Container.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5959) RM changes to support change of container ExecutionType
[
https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15742711#comment-15742711
]
Arun Suresh edited comment on YARN-5959 at 12/12/16 6:33 PM:
-
Thanks for capturing the salient points from our discussion [~leftnoteasy].
Updating patch based your feedback..
1.
bq. Instead of creating a hard-locality container request, can we wrap the
ResourceRequest to add some additional information to identify it is for
container increase / promotion.
It looks like we already have an {{AbstractResourceRequest}} which both the
{{ResourceRequest}} and {{UpdateContainerRequest}} extends from.
I renamed it to {{SchedulerResourceRequest}} and also included a field
*containerToUpdate* to signify this is an update to an existing container.
2.
bq. if any normal container request with higher priority, it will be allocated
first. However this is discussable: if we give the available resource of a node
to a different normal container, it is possible that the running opportunistic
container will be killed and work will be wasted.
This is IMO still an ordering issue. I have updated the {{SchedulerRequestKey}}
to ensure the behavior you pointed out. Update requests (both promotion as well
as resource updates) will be considered before any normal resourceRequest.
Since the schedulerKey is created using a {{SchedulerResourceRequest}}, which
contains information about the container update and not the old
{{ResourceRequest}}, I do not have to hard code a negative allocationRequestId
as I had done in the previous patch.
3.
{quote}
* Track pending container changes requests.
* Cancel / update pending container change requests
* Makes backend implementation of container increase / promotion to be same.
{quote}
I feel all these cases can be taken care of in the new patch by ensuring that
the {{SchedulerResourceRequest}} and {{SchedulerRequestKey}} understands that
it is pertaining to a container update.
Once you agree on this new design, I can go ahead (I think we can do that in a
separate JIRA) and create a patch that unifies the backend for container
resource increase and promotion.
was (Author: asuresh):
Thanks for capturing the salient points from our discussion [~leftnoteasy].
Updating patch based your feedback..
1.
bq. Instead of creating a hard-locality container request, can we wrap the
ResourceRequest to add some additional information to identify it is for
container increase / promotion.
It looks like we already have an {{AbstractResourceRequest}} which both the
{{ResourceRequest}} and {{UpdateContainerRequest}} extends from.
I renamed it to {{SchedulerResourceRequest}} and also included a field
*containerToUpdate* to signify this is an update to an existing container.
2.
bq. if any normal container request with higher priority, it will be allocated
first. However this is discussable: if we give the available resource of a node
to a different normal container, it is possible that the running opportunistic
container will be killed and work will be wasted.
This is IMO still an ordering issue. I have updated the {{SchedulerRequestKey}}
to ensure the behavior you pointed out. Update requests (both promotion as well
as resource updates) will be considered before any normal resourceRequest
3.
{quote}
* Track pending container changes requests.
* Cancel / update pending container change requests
* Makes backend implementation of container increase / promotion to be same.
{quote}
I feel all these cases can be taken care of in the new patch by ensuring that
the {{SchedulerResourceRequest}} and {{SchedulerRequestKey}} understands that
it is pertaining to a container update.
Once you agree on this new design, I can go ahead an create a patch that
unifies the backed for container resource increase and promotion.
> RM changes to support change of container ExecutionType
> ---
>
> Key: YARN-5959
> URL: https://issues.apache.org/jira/browse/YARN-5959
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-5959.combined.001.patch, YARN-5959.wip.002.patch,
> YARN-5959.wip.003.patch, YARN-5959.wip.patch
>
>
> RM side changes to allow an AM to ask for change of ExecutionType.
> Currently, there are two cases:
> # *Promotion* : OPPORTUNISTIC to GUARANTEED.
> # *Demotion* : GUARANTEED to OPPORTUNISTIC.
> This is similar in YARN-1197 which allows for change in Container resources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5216) Expose configurable preemption policy for OPPORTUNISTIC containers running on the NM
[ https://issues.apache.org/jira/browse/YARN-5216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743071#comment-15743071 ] Arun Suresh commented on YARN-5216: --- Just clarifying my position: * The ability to PAUSE a process is something the environment (executor + runtime + OS) on which the NM is running provides. * W.r.t the NM/ContainerScheduler, if it finds that an opportunistic container is using resources that can be given to a waiting guaranteed container, it just needs the underlying environment (executor + runtime + OS) to reclaim those resources so it can start the guaranteed container. * HOW the environment does this should not really matter to the NM/ContainerScheduler, it just needs a callback from the launcher/executor that the container in question was killed or Paused, to signal if it should place it back in the queue or not. * IMO, I don't see a case where the NM/Scheduler should prefer one over the other, which is the reason why I suggested a common *preempt* method (am not particular of the name, we can call it *reclaimResources* maybe) in the Executor. If later we do find a case, there is already the *kill* method anyway. I infact feel the executor should first try to pause, and if not supported, only then kill the opportunistic container. Given the above, and the fact that most large deployments use their own specialized Executor class / binary anyway (I pointed to YARN-5673, since there is discussion of having dynamically loaded modules / separate binaries in the default distribution based on flags passed during build as well), I was wondering why expose a yarn-site.xml knob in the first place. I am fine making this decision later too. For the time-being, consider adding *reclaimResources* to the ContainerExecutor, which based on some configuration, delegates to kill or pause. > Expose configurable preemption policy for OPPORTUNISTIC containers running on > the NM > > > Key: YARN-5216 > URL: https://issues.apache.org/jira/browse/YARN-5216 > Project: Hadoop YARN > Issue Type: Sub-task > Components: distributed-scheduling >Reporter: Arun Suresh >Assignee: Hitesh Sharma > Labels: oct16-hard > Attachments: YARN-5216-YARN-5972.001.patch, YARN5216.001.patch, > yarn5216.002.patch > > > Currently, the default action taken by the QueuingContainerManager, > introduced in YARN-2883, when a GUARANTEED Container is scheduled on an NM > with OPPORTUNISTIC containers using up resources, is to KILL the running > OPPORTUNISTIC containers. > This JIRA proposes to expose a configurable hook to allow the NM to take a > different action. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3866) AM-RM protocol changes to support container resizing
[
https://issues.apache.org/jira/browse/YARN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730322#comment-15730322
]
Arun Suresh commented on YARN-3866:
---
bq. No. It is already get released in 2.7.3 ..
Ah.. sorry.. was looking at the {{CapacityScheduler}} class which did not have
a working implementation of the feature. Just realized as Wangda mentioned,
only the APIs were updated in 2.7.
bq. We can deprecate these APIs to tell people to use the right API. I think
that is the right way to do.
Agreed.
> AM-RM protocol changes to support container resizing
>
>
> Key: YARN-3866
> URL: https://issues.apache.org/jira/browse/YARN-3866
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api
>Reporter: MENG DING
>Assignee: MENG DING
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: YARN-3866-YARN-1197.4.patch, YARN-3866.1.patch,
> YARN-3866.2.patch, YARN-3866.3.patch
>
>
> YARN-1447 and YARN-1448 are outdated.
> This ticket deals with AM-RM Protocol changes to support container resize
> according to the latest design in YARN-1197.
> 1) Add increase/decrease requests in AllocateRequest
> 2) Get approved increase/decrease requests from RM in AllocateResponse
> 3) Add relevant test cases
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-2877) Extend YARN to support distributed scheduling
[ https://issues.apache.org/jira/browse/YARN-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-2877: -- Fix Version/s: 3.0.0-alpha2 2.9.0 > Extend YARN to support distributed scheduling > - > > Key: YARN-2877 > URL: https://issues.apache.org/jira/browse/YARN-2877 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager, resourcemanager >Reporter: Sriram Rao >Assignee: Konstantinos Karanasos > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: distributed-scheduling-design-doc_v1.pdf > > > This is an umbrella JIRA that proposes to extend YARN to support distributed > scheduling. Briefly, some of the motivations for distributed scheduling are > the following: > 1. Improve cluster utilization by opportunistically executing tasks otherwise > idle resources on individual machines. > 2. Reduce allocation latency. Tasks where the scheduling time dominates > (i.e., task execution time is much less compared to the time required for > obtaining a container from the RM). > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6096) getCpuFrequency for windows returning incorrect value
[ https://issues.apache.org/jira/browse/YARN-6096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-6096: -- Assignee: Roni Burd > getCpuFrequency for windows returning incorrect value > - > > Key: YARN-6096 > URL: https://issues.apache.org/jira/browse/YARN-6096 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.3 > Environment: windows >Reporter: Roni Burd >Assignee: Roni Burd >Priority: Minor > Fix For: 2.7.3 > > > getCpuFrequency in WindowsResourceCalculatorPlugin returns -1 instead of the > actual parsed value -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-5646) Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers
[ https://issues.apache.org/jira/browse/YARN-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh reopened YARN-5646: --- Reopening to add addendum patch to include changes to the site.xml file > Add documentation and update config parameter names for scheduling of > OPPORTUNISTIC containers > -- > > Key: YARN-5646 > URL: https://issues.apache.org/jira/browse/YARN-5646 > Project: Hadoop YARN > Issue Type: Task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos >Priority: Blocker > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5646.001.patch, YARN-5646.002.patch, > YARN-5646.003.patch, YARN-5646.004.patch > > > This is for adding documentation regarding the scheduling of OPPORTUNISTIC > containers. > It includes both the centralized (YARN-5220) and the distributed (YARN-2877) > scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5646) Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers
[ https://issues.apache.org/jira/browse/YARN-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5646: -- Attachment: YARN-5646.addendum.patch > Add documentation and update config parameter names for scheduling of > OPPORTUNISTIC containers > -- > > Key: YARN-5646 > URL: https://issues.apache.org/jira/browse/YARN-5646 > Project: Hadoop YARN > Issue Type: Task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos >Priority: Blocker > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5646.001.patch, YARN-5646.002.patch, > YARN-5646.003.patch, YARN-5646.004.patch, YARN-5646.addendum.patch > > > This is for adding documentation regarding the scheduling of OPPORTUNISTIC > containers. > It includes both the centralized (YARN-5220) and the distributed (YARN-2877) > scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5646) Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers
[ https://issues.apache.org/jira/browse/YARN-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh resolved YARN-5646. --- Resolution: Fixed Committed addendum patch to trunk and branch-2 > Add documentation and update config parameter names for scheduling of > OPPORTUNISTIC containers > -- > > Key: YARN-5646 > URL: https://issues.apache.org/jira/browse/YARN-5646 > Project: Hadoop YARN > Issue Type: Task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos >Priority: Blocker > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5646.001.patch, YARN-5646.002.patch, > YARN-5646.003.patch, YARN-5646.004.patch, YARN-5646.addendum.patch > > > This is for adding documentation regarding the scheduling of OPPORTUNISTIC > containers. > It includes both the centralized (YARN-5220) and the distributed (YARN-2877) > scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5966) AMRMClient changes to support ExecutionType update
[
https://issues.apache.org/jira/browse/YARN-5966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-5966:
--
Attachment: YARN-5966.002.patch
Updating patch. Fixing failed testcase
> AMRMClient changes to support ExecutionType update
> --
>
> Key: YARN-5966
> URL: https://issues.apache.org/jira/browse/YARN-5966
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-5966.001.patch, YARN-5966.002.patch,
> YARN-5966.wip.001.patch
>
>
> {{AMRMClient}} changes to support change of container ExecutionType
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5959) RM changes to support change of container ExecutionType
[
https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717931#comment-15717931
]
Arun Suresh commented on YARN-5959:
---
This can actually be accomplished in a scheduler agnostic manner.
Promotion can be implemented as follows:
# Create a dummy {{ResourceRequest}} with the same resource, priority etc. of
the existing opportunistic container
# Set the required resource name (target location) to the node on which in the
original container was allocated.
# Set *relaxLocality* to *false*.
# Set the *allocationRequestId* of the request to be negative, so it will be
considered before any other {{ResourceRequest}} with the same priority.
# Once the scheduler allocates an {{RMContainer}} against the dummy request,
swap the containerIds with the existing container and release this new
container. This will ensure that the resource accounting is correct. The
allocation path ensures that available resources in the queue, application and
node resources are correctly decremented and the release ensures that only the
opportunistic resource (essentially 0 resource container from the perspective
of the scheduler) are reclaimed.
Demotion will happen in exactly the reverse:
A dummy OPPORTUNISTIC container is created to swap with the existing GUARANTEED
container on the same node. The containerIds are then swapped the dummy
container is release ensuring guaranteed resources are reclaimed.
> RM changes to support change of container ExecutionType
> ---
>
> Key: YARN-5959
> URL: https://issues.apache.org/jira/browse/YARN-5959
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>
> RM side changes to allow an AM to ask for change of ExecutionType.
> Currently, there are two cases:
> # *Promotion* : OPPORTUNISTIC to GUARANTEED.
> # *Demotion* : GUARANTEED to OPPORTUNISTIC.
> This is similar in YARN-1197 which allows for change in Container resources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5959) RM changes to support change of container ExecutionType
[ https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5959: -- Attachment: YARN-5959.combined.001.patch Since the WIP patch depends on YARN-5938, am attaching a combined patch that applies against trunk to kick off Jenkins > RM changes to support change of container ExecutionType > --- > > Key: YARN-5959 > URL: https://issues.apache.org/jira/browse/YARN-5959 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5959.combined.001.patch, YARN-5959.wip.patch > > > RM side changes to allow an AM to ask for change of ExecutionType. > Currently, there are two cases: > # *Promotion* : OPPORTUNISTIC to GUARANTEED. > # *Demotion* : GUARANTEED to OPPORTUNISTIC. > This is similar in YARN-1197 which allows for change in Container resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5959) RM changes to support change of container ExecutionType
[ https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5959: -- Description: RM side changes to allow an AM to ask for change of ExecutionType. Currently, there are two cases: # *Promotion* : OPPORTUNISTIC to GUARANTEED. # *Promotion* : OPPORTUNISTIC to GUARANTEED. > RM changes to support change of container ExecutionType > --- > > Key: YARN-5959 > URL: https://issues.apache.org/jira/browse/YARN-5959 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > > RM side changes to allow an AM to ask for change of ExecutionType. > Currently, there are two cases: > # *Promotion* : OPPORTUNISTIC to GUARANTEED. > # *Promotion* : OPPORTUNISTIC to GUARANTEED. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5087) Expose API to allow AM to request for change of container ExecutionType
[ https://issues.apache.org/jira/browse/YARN-5087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717898#comment-15717898 ] Arun Suresh edited comment on YARN-5087 at 12/3/16 10:45 AM: - Closing this, as YARN-5221 takes care of this was (Author: asuresh): Closing this, as YARN-5221 fixes this > Expose API to allow AM to request for change of container ExecutionType > --- > > Key: YARN-5087 > URL: https://issues.apache.org/jira/browse/YARN-5087 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5087) Expose API to allow AM to request for change of container ExecutionType
[ https://issues.apache.org/jira/browse/YARN-5087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh resolved YARN-5087. --- Resolution: Resolved Closing this, as YARN-5221 fixes this > Expose API to allow AM to request for change of container ExecutionType > --- > > Key: YARN-5087 > URL: https://issues.apache.org/jira/browse/YARN-5087 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5959) RM changes to support change of container ExecutionType
[ https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5959: -- Description: RM side changes to allow an AM to ask for change of ExecutionType. Currently, there are two cases: # *Promotion* : OPPORTUNISTIC to GUARANTEED. # *Demotion* : GUARANTEED to OPPORTUNISTIC. This is similar in YARN-1197 which allows for change in Container resources. was: RM side changes to allow an AM to ask for change of ExecutionType. Currently, there are two cases: # *Promotion* : OPPORTUNISTIC to GUARANTEED. # *Demotion* : GUARANTEED to OPPORTUNISTIC. > RM changes to support change of container ExecutionType > --- > > Key: YARN-5959 > URL: https://issues.apache.org/jira/browse/YARN-5959 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > > RM side changes to allow an AM to ask for change of ExecutionType. > Currently, there are two cases: > # *Promotion* : OPPORTUNISTIC to GUARANTEED. > # *Demotion* : GUARANTEED to OPPORTUNISTIC. > This is similar in YARN-1197 which allows for change in Container resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5959) RM changes to support change of container ExecutionType
[ https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5959: -- Summary: RM changes to support change of container ExecutionType (was: Add support for ExecutionType change from OPPORTUNISTIC to GUARANTEED) > RM changes to support change of container ExecutionType > --- > > Key: YARN-5959 > URL: https://issues.apache.org/jira/browse/YARN-5959 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5959) RM changes to support change of container ExecutionType
[ https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5959: -- Attachment: YARN-5959.wip.patch Attaching a WIP patch with tests to demonstrate the proposal in the previous comment works > RM changes to support change of container ExecutionType > --- > > Key: YARN-5959 > URL: https://issues.apache.org/jira/browse/YARN-5959 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5959.wip.patch > > > RM side changes to allow an AM to ask for change of ExecutionType. > Currently, there are two cases: > # *Promotion* : OPPORTUNISTIC to GUARANTEED. > # *Demotion* : GUARANTEED to OPPORTUNISTIC. > This is similar in YARN-1197 which allows for change in Container resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5959) RM changes to support change of container ExecutionType
[
https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717931#comment-15717931
]
Arun Suresh edited comment on YARN-5959 at 12/3/16 11:21 AM:
-
This can actually be accomplished in a scheduler agnostic manner.
Promotion can be implemented as follows:
# Create a dummy {{ResourceRequest}} with the same resource, priority etc. of
the existing opportunistic container
# Set the required resource name (target location) to the node on which in the
original container was allocated.
# Set *relaxLocality* to *false*.
# Set the *allocationRequestId* of the request to be negative, so it will be
considered before any other {{ResourceRequest}} with the same priority.
# Once the scheduler allocates an {{RMContainer}} against the dummy request,
swap the containerIds with the existing container and release this new
container. This will ensure that the resource accounting is correct. The
allocation path ensures that available resources in the queue, application and
node resources are correctly decremented and the release ensures that only the
opportunistic resource (essentially 0 resource container from the perspective
of the scheduler) are reclaimed.
Demotion will happen in exactly the reverse:
A dummy OPPORTUNISTIC container is created to swap with the existing GUARANTEED
container on the same node. The containerIds are then swapped and the dummy
container is release ensuring guaranteed resources are reclaimed.
was (Author: asuresh):
This can actually be accomplished in a scheduler agnostic manner.
Promotion can be implemented as follows:
# Create a dummy {{ResourceRequest}} with the same resource, priority etc. of
the existing opportunistic container
# Set the required resource name (target location) to the node on which in the
original container was allocated.
# Set *relaxLocality* to *false*.
# Set the *allocationRequestId* of the request to be negative, so it will be
considered before any other {{ResourceRequest}} with the same priority.
# Once the scheduler allocates an {{RMContainer}} against the dummy request,
swap the containerIds with the existing container and release this new
container. This will ensure that the resource accounting is correct. The
allocation path ensures that available resources in the queue, application and
node resources are correctly decremented and the release ensures that only the
opportunistic resource (essentially 0 resource container from the perspective
of the scheduler) are reclaimed.
Demotion will happen in exactly the reverse:
A dummy OPPORTUNISTIC container is created to swap with the existing GUARANTEED
container on the same node. The containerIds are then swapped the dummy
container is release ensuring guaranteed resources are reclaimed.
> RM changes to support change of container ExecutionType
> ---
>
> Key: YARN-5959
> URL: https://issues.apache.org/jira/browse/YARN-5959
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-5959.wip.patch
>
>
> RM side changes to allow an AM to ask for change of ExecutionType.
> Currently, there are two cases:
> # *Promotion* : OPPORTUNISTIC to GUARANTEED.
> # *Demotion* : GUARANTEED to OPPORTUNISTIC.
> This is similar in YARN-1197 which allows for change in Container resources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5959) RM changes to support change of container ExecutionType
[ https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717948#comment-15717948 ] Arun Suresh edited comment on YARN-5959 at 12/3/16 11:23 AM: - Attaching a WIP patch with tests to demonstrate the proposal in the previous comment works Ping : [~leftnoteasy], [~kasha], [~subru], [~curino], do take a look was (Author: asuresh): Attaching a WIP patch with tests to demonstrate the proposal in the previous comment works > RM changes to support change of container ExecutionType > --- > > Key: YARN-5959 > URL: https://issues.apache.org/jira/browse/YARN-5959 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5959.wip.patch > > > RM side changes to allow an AM to ask for change of ExecutionType. > Currently, there are two cases: > # *Promotion* : OPPORTUNISTIC to GUARANTEED. > # *Demotion* : GUARANTEED to OPPORTUNISTIC. > This is similar in YARN-1197 which allows for change in Container resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5959) RM changes to support change of container ExecutionType
[ https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5959: -- Description: RM side changes to allow an AM to ask for change of ExecutionType. Currently, there are two cases: # *Promotion* : OPPORTUNISTIC to GUARANTEED. # *Demotion* : GUARANTEED to OPPORTUNISTIC. was: RM side changes to allow an AM to ask for change of ExecutionType. Currently, there are two cases: # *Promotion* : OPPORTUNISTIC to GUARANTEED. # *Promotion* : OPPORTUNISTIC to GUARANTEED. > RM changes to support change of container ExecutionType > --- > > Key: YARN-5959 > URL: https://issues.apache.org/jira/browse/YARN-5959 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > > RM side changes to allow an AM to ask for change of ExecutionType. > Currently, there are two cases: > # *Promotion* : OPPORTUNISTIC to GUARANTEED. > # *Demotion* : GUARANTEED to OPPORTUNISTIC. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5959) RM changes to support change of container ExecutionType
[
https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717948#comment-15717948
]
Arun Suresh edited comment on YARN-5959 at 12/3/16 11:31 AM:
-
Attaching a WIP patch with tests to demonstrate the proposal in the previous
comment works
Ping : [~leftnoteasy], [~kasha], [~subru], [~curino], [~kkaranasos], do take a
look
The only really interesting changes are in the {{SchedulerAplicationAttempt}}
class. Most of the remaining changes are related to wiring.
was (Author: asuresh):
Attaching a WIP patch with tests to demonstrate the proposal in the previous
comment works
Ping : [~leftnoteasy], [~kasha], [~subru], [~curino], do take a look
> RM changes to support change of container ExecutionType
> ---
>
> Key: YARN-5959
> URL: https://issues.apache.org/jira/browse/YARN-5959
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-5959.combined.001.patch, YARN-5959.wip.patch
>
>
> RM side changes to allow an AM to ask for change of ExecutionType.
> Currently, there are two cases:
> # *Promotion* : OPPORTUNISTIC to GUARANTEED.
> # *Demotion* : GUARANTEED to OPPORTUNISTIC.
> This is similar in YARN-1197 which allows for change in Container resources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5292) Support for PAUSED container state
[
https://issues.apache.org/jira/browse/YARN-5292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15718618#comment-15718618
]
Arun Suresh commented on YARN-5292:
---
Thanks for the patch Hitesh, couple of comments:
* {{ContainerExecutor}}, maybe default behavior should not be to throw an
Exception. We should probably LOG.warn() too.
* {{ContainerImpl}}, In a couple of places, you can maybe collapse a bunch of
transitions like this :
{noformat}
.addTransition(ContainerState.KILLING,
ContainerState.KILLING,
ContainerEventType.CONTAINER_LAUNCHED)
.addTransition(ContainerState.KILLING,
ContainerState.KILLING,
ContainerEventType.PAUSE_CONTAINER)
{noformat}
into
{noformat}
.addTransition(ContainerState.KILLING,
ContainerState.KILLING,
EnumSet.of(ContainerEventType.CONTAINER_LAUNCHED,
ContainerEventType.PAUSE_CONTAINER)
{noformat}
* It looks like when a container is REINITIALIZING, and it receives a PAUSE
event, you are killing the container… Think it might be better to re-queue the
container somehow in this case - so the scheduler can restart it when there is
available resources.
* I was thinking PAUSED and RESUMING should be notified to the RM as SCHEDULED
itself. SCHEDULED should be used signify that the container allocation is
secure, but is not running.
> Support for PAUSED container state
> --
>
> Key: YARN-5292
> URL: https://issues.apache.org/jira/browse/YARN-5292
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Hitesh Sharma
>Assignee: Hitesh Sharma
> Attachments: YARN-5292.001.patch, YARN-5292.002.patch,
> YARN-5292.003.patch, YARN-5292.004.patch, yarn-5292.pdf
>
>
> YARN-2877 introduced OPPORTUNISTIC containers, and YARN-5216 proposes to add
> capability to customize how OPPORTUNISTIC containers get preempted.
> In this JIRA we propose introducing a PAUSED container state.
> When a running container gets preempted, it enters the PAUSED state, where it
> remains until resources get freed up on the node then the preempted container
> can resume to the running state.
>
> One scenario where this capability is useful is work preservation. How
> preemption is done, and whether the container supports it, is implementation
> specific.
> For instance, if the container is a virtual machine, then preempt would pause
> the VM and resume would restore it back to the running state.
> If the container doesn't support preemption, then preempt would default to
> killing the container.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5938) Minor refactoring to OpportunisticContainerAllocatorAMService
[ https://issues.apache.org/jira/browse/YARN-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5938: -- Attachment: YARN-5938-YARN-5085.002.patch Updating patch.. moved SchedulerRequestKey to a common, so it can be used by the OpportunisticAllocator > Minor refactoring to OpportunisticContainerAllocatorAMService > - > > Key: YARN-5938 > URL: https://issues.apache.org/jira/browse/YARN-5938 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5938-YARN-5085.001.patch, > YARN-5938-YARN-5085.002.patch, YARN-5938.001.patch > > > Minor code re-organization to do the following: > # The OpportunisticContainerAllocatorAMService currently allocates outside > the ApplicationAttempt lock maintained by the ApplicationMasterService. This > should happen inside the lock. > # Refactored out some code to simplify the allocate() method. > # Removed some unused fields inside the OpportunisticContainerAllocator. > # Re-organized some of the code in the > OpportunisticContainerAllocatorAMService::allocate method to make it a bit > more readable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5959) Add support for ExecutionType change from OPPORTUNISTIC to GUARANTEED
[ https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh reassigned YARN-5959: - Assignee: Arun Suresh > Add support for ExecutionType change from OPPORTUNISTIC to GUARANTEED > - > > Key: YARN-5959 > URL: https://issues.apache.org/jira/browse/YARN-5959 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5959) Add support for ExecutionType change from OPPORTUNISTIC to GUARANTEED
Arun Suresh created YARN-5959: - Summary: Add support for ExecutionType change from OPPORTUNISTIC to GUARANTEED Key: YARN-5959 URL: https://issues.apache.org/jira/browse/YARN-5959 Project: Hadoop YARN Issue Type: Sub-task Reporter: Arun Suresh -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5938) Minor refactoring to OpportunisticContainerAllocatorAMService
[ https://issues.apache.org/jira/browse/YARN-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5938: -- Attachment: YARN-5938-YARN-5085.001.patch Created new branch for YARN-5085. Updated patch to fix test cases. > Minor refactoring to OpportunisticContainerAllocatorAMService > - > > Key: YARN-5938 > URL: https://issues.apache.org/jira/browse/YARN-5938 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5938-YARN-5085.001.patch, YARN-5938.001.patch > > > Minor code re-organization to do the following: > # The OpportunisticContainerAllocatorAMService currently allocates outside > the ApplicationAttempt lock maintained by the ApplicationMasterService. This > should happen inside the lock. > # Refactored out some code to simplify the allocate() method. > # Removed some unused fields inside the OpportunisticContainerAllocator. > # Re-organized some of the code in the > OpportunisticContainerAllocatorAMService::allocate method to make it a bit > more readable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5966) AMRMClient changes to support ExecutionType update
Arun Suresh created YARN-5966: - Summary: AMRMClient changes to support ExecutionType update Key: YARN-5966 URL: https://issues.apache.org/jira/browse/YARN-5966 Project: Hadoop YARN Issue Type: Sub-task Reporter: Arun Suresh -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5966) AMRMClient changes to support ExecutionType update
[
https://issues.apache.org/jira/browse/YARN-5966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-5966:
--
Description: {{AMRMClient}} changes to support change of container
ExecutionType
> AMRMClient changes to support ExecutionType update
> --
>
> Key: YARN-5966
> URL: https://issues.apache.org/jira/browse/YARN-5966
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>
> {{AMRMClient}} changes to support change of container ExecutionType
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5966) AMRMClient changes to support ExecutionType update
[
https://issues.apache.org/jira/browse/YARN-5966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh reassigned YARN-5966:
-
Assignee: Arun Suresh
> AMRMClient changes to support ExecutionType update
> --
>
> Key: YARN-5966
> URL: https://issues.apache.org/jira/browse/YARN-5966
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>
> {{AMRMClient}} changes to support change of container ExecutionType
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5085) Add support for change of container ExecutionType
[
https://issues.apache.org/jira/browse/YARN-5085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-5085:
--
Attachment: YARN-5085.001.patch
Attaching a combined patch for all the subpatches to kick off Jenkins.
The Sub tasks can be used for focused reviews.
> Add support for change of container ExecutionType
> -
>
> Key: YARN-5085
> URL: https://issues.apache.org/jira/browse/YARN-5085
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-5085.001.patch
>
>
> YARN-2882 introduced the concept of {{ExecutionType}} for containers and it
> also introduced the concept of OPPORTUNISTIC ExecutionType.
> YARN-4335 introduced changes to the ResourceRequest so that AMs may request
> that the Container allocated against the ResourceRequest is of a particular
> {{ExecutionType}}.
> This JIRA proposes to provide support for the AM to change the ExecutionType
> of a previously requested Container.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5959) RM changes to support change of container ExecutionType
[ https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5959: -- Attachment: YARN-5959.wip.002.patch Moved out the AMRMClient changes to YARN-5966 and added more tests. > RM changes to support change of container ExecutionType > --- > > Key: YARN-5959 > URL: https://issues.apache.org/jira/browse/YARN-5959 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5959.combined.001.patch, YARN-5959.wip.002.patch, > YARN-5959.wip.patch > > > RM side changes to allow an AM to ask for change of ExecutionType. > Currently, there are two cases: > # *Promotion* : OPPORTUNISTIC to GUARANTEED. > # *Demotion* : GUARANTEED to OPPORTUNISTIC. > This is similar in YARN-1197 which allows for change in Container resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5966) AMRMClient changes to support ExecutionType update
[
https://issues.apache.org/jira/browse/YARN-5966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-5966:
--
Attachment: YARN-5966.wip.001.patch
Attaching initial patch
> AMRMClient changes to support ExecutionType update
> --
>
> Key: YARN-5966
> URL: https://issues.apache.org/jira/browse/YARN-5966
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-5966.wip.001.patch
>
>
> {{AMRMClient}} changes to support change of container ExecutionType
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5292) Support for PAUSED container state
[
https://issues.apache.org/jira/browse/YARN-5292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723652#comment-15723652
]
Arun Suresh commented on YARN-5292:
---
Thanks for the update [~hrsharma].
I am +1 for the latest patch.
Would like to hear [~sidharta-s]'s and [~vvasudev]'s thoughts on integrating
this functionality with the {{LinuxContainerExecutor}} and/or possibly the
associated Runtimes since docker does support Container pausing which
internally uses cgroups freezer module, which means it should be exposable via
the default runtime as well.
Will commit this tomorrow, if its ok with everyone else tracking this.
> Support for PAUSED container state
> --
>
> Key: YARN-5292
> URL: https://issues.apache.org/jira/browse/YARN-5292
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Hitesh Sharma
>Assignee: Hitesh Sharma
> Attachments: YARN-5292.001.patch, YARN-5292.002.patch,
> YARN-5292.003.patch, YARN-5292.004.patch, YARN-5292.005.patch, yarn-5292.pdf
>
>
> YARN-2877 introduced OPPORTUNISTIC containers, and YARN-5216 proposes to add
> capability to customize how OPPORTUNISTIC containers get preempted.
> In this JIRA we propose introducing a PAUSED container state.
> When a running container gets preempted, it enters the PAUSED state, where it
> remains until resources get freed up on the node then the preempted container
> can resume to the running state.
>
> One scenario where this capability is useful is work preservation. How
> preemption is done, and whether the container supports it, is implementation
> specific.
> For instance, if the container is a virtual machine, then preempt would pause
> the VM and resume would restore it back to the running state.
> If the container doesn't support preemption, then preempt would default to
> killing the container.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[ https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802195#comment-15802195 ] Arun Suresh commented on YARN-6041: --- [~leftnoteasy], So what I plan to do was, like you mentioned, cherry-pick the 10 JIRAs specified in the description (I actually created the patch by doing just that.. and then doing a "git diff ").. Then commit JUST the changes Karthik suggested as "YARN-6041: .." which I will cherry-pick on to trunk as well. > Opportunistic containers : Combined patch for branch-2 > --- > > Key: YARN-6041 > URL: https://issues.apache.org/jira/browse/YARN-6041 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-6041-branch-2.001.patch, > YARN-6041-branch-2.002.patch, YARN-6041-branch-2.003.patch > > > This is a combined patch targeting branch-2 of the following JIRAs which have > already been committed to trunk : > YARN-5938. Refactoring OpportunisticContainerAllocator to use > SchedulerRequestKey instead of Priority and other misc fixes > YARN-5646. Add documentation and update config parameter names for scheduling > of OPPORTUNISTIC containers. > YARN-5982. Simplify opportunistic container parameters and metrics. > YARN-5918. Handle Opportunistic scheduling allocate request failure when NM > is lost. > YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager > container lifecycle. > YARN-5823. Update NMTokens in case of requests with only opportunistic > containers. > YARN-5377. Fix > TestQueuingContainerManager.testKillMultipleOpportunisticContainers. > YARN-2995. Enhance UI to show cluster resource utilization of various > container Execution types. > YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http > Address. > YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method > to handle OPPORTUNISTIC container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6040) Remove usage of ResourceRequest from AppSchedulerInfo / SchedulerApplicationAttempt
[
https://issues.apache.org/jira/browse/YARN-6040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802425#comment-15802425
]
Arun Suresh commented on YARN-6040:
---
Thanks for updating the patch [~leftnoteasy],
# Looks like SchedulingPlacementSet still exposes getPendingAllocationNumber(),
can we change that to match the PendingAsk ? Thinking further, wondering if you
need that method.. you should be doing getPendingAsk(resourceName).getCount()
right ?
# Similarly, looks like you might not need
SchedulerApplicationAttempt::getPendingAllocationNumber() either..
Everything else looks ok to me..
> Remove usage of ResourceRequest from AppSchedulerInfo /
> SchedulerApplicationAttempt
> ---
>
> Key: YARN-6040
> URL: https://issues.apache.org/jira/browse/YARN-6040
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-6040.001.patch, YARN-6040.002.patch,
> YARN-6040.003.patch, YARN-6040.004.patch, YARN-6040.005.patch,
> YARN-6040.006.patch
>
>
> As mentioned by YARN-5906, currently schedulers are using ResourceRequest
> heavily so it will be very hard to adopt the new PowerfulResourceRequest
> (YARN-4902).
> This JIRA is the 2nd step of refactoring, which remove usage of
> ResourceRequest from AppSchedulingInfo / SchedulerApplicationAttempt. Instead
> of returning ResourceRequest, it returns a lightweight and API-independent
> object - {{PendingAsk}}.
> The only remained ResourceRequest API of AppSchedulingInfo will be used by
> web service to get list of ResourceRequests.
> So after this patch, usage of ResourceRequest will be isolated inside
> AppSchedulingInfo, so it will be more flexible to update internal data
> structure and upgrade old ResourceRequest API to new.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6040) Introduce api independent PendingAsk to replace usage of ResourceRequest within Scheduler classes
[
https://issues.apache.org/jira/browse/YARN-6040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-6040:
--
Summary: Introduce api independent PendingAsk to replace usage of
ResourceRequest within Scheduler classes (was: Remove usage of ResourceRequest
from AppSchedulerInfo / SchedulerApplicationAttempt)
> Introduce api independent PendingAsk to replace usage of ResourceRequest
> within Scheduler classes
> -
>
> Key: YARN-6040
> URL: https://issues.apache.org/jira/browse/YARN-6040
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-6040.001.patch, YARN-6040.002.patch,
> YARN-6040.003.patch, YARN-6040.004.patch, YARN-6040.005.patch,
> YARN-6040.006.patch, YARN-6040.007.patch
>
>
> As mentioned by YARN-5906, currently schedulers are using ResourceRequest
> heavily so it will be very hard to adopt the new PowerfulResourceRequest
> (YARN-4902).
> This JIRA is the 2nd step of refactoring, which remove usage of
> ResourceRequest from AppSchedulingInfo / SchedulerApplicationAttempt. Instead
> of returning ResourceRequest, it returns a lightweight and API-independent
> object - {{PendingAsk}}.
> The only remained ResourceRequest API of AppSchedulingInfo will be used by
> web service to get list of ResourceRequests.
> So after this patch, usage of ResourceRequest will be isolated inside
> AppSchedulingInfo, so it will be more flexible to update internal data
> structure and upgrade old ResourceRequest API to new.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6040) Introduce api independent PendingAsk to replace usage of ResourceRequest within Scheduler classes
[
https://issues.apache.org/jira/browse/YARN-6040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-6040:
--
Fix Version/s: 3.0.0-beta1
> Introduce api independent PendingAsk to replace usage of ResourceRequest
> within Scheduler classes
> -
>
> Key: YARN-6040
> URL: https://issues.apache.org/jira/browse/YARN-6040
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Fix For: 3.0.0-beta1
>
> Attachments: YARN-6040.001.patch, YARN-6040.002.patch,
> YARN-6040.003.patch, YARN-6040.004.patch, YARN-6040.005.patch,
> YARN-6040.006.patch, YARN-6040.007.patch
>
>
> As mentioned by YARN-5906, currently schedulers are using ResourceRequest
> heavily so it will be very hard to adopt the new PowerfulResourceRequest
> (YARN-4902).
> This JIRA is the 2nd step of refactoring, which remove usage of
> ResourceRequest from AppSchedulingInfo / SchedulerApplicationAttempt. Instead
> of returning ResourceRequest, it returns a lightweight and API-independent
> object - {{PendingAsk}}.
> The only remained ResourceRequest API of AppSchedulingInfo will be used by
> web service to get list of ResourceRequests.
> So after this patch, usage of ResourceRequest will be isolated inside
> AppSchedulingInfo, so it will be more flexible to update internal data
> structure and upgrade old ResourceRequest API to new.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6040) Introduce api independent PendingAsk to replace usage of ResourceRequest within Scheduler classes
[
https://issues.apache.org/jira/browse/YARN-6040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15805139#comment-15805139
]
Arun Suresh commented on YARN-6040:
---
Committing this shortly...
> Introduce api independent PendingAsk to replace usage of ResourceRequest
> within Scheduler classes
> -
>
> Key: YARN-6040
> URL: https://issues.apache.org/jira/browse/YARN-6040
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-6040.001.patch, YARN-6040.002.patch,
> YARN-6040.003.patch, YARN-6040.004.patch, YARN-6040.005.patch,
> YARN-6040.006.patch, YARN-6040.007.patch
>
>
> As mentioned by YARN-5906, currently schedulers are using ResourceRequest
> heavily so it will be very hard to adopt the new PowerfulResourceRequest
> (YARN-4902).
> This JIRA is the 2nd step of refactoring, which remove usage of
> ResourceRequest from AppSchedulingInfo / SchedulerApplicationAttempt. Instead
> of returning ResourceRequest, it returns a lightweight and API-independent
> object - {{PendingAsk}}.
> The only remained ResourceRequest API of AppSchedulingInfo will be used by
> web service to get list of ResourceRequests.
> So after this patch, usage of ResourceRequest will be isolated inside
> AppSchedulingInfo, so it will be more flexible to update internal data
> structure and upgrade old ResourceRequest API to new.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6040) Introduce api independent PendingAsk to replace usage of ResourceRequest within Scheduler classes
[
https://issues.apache.org/jira/browse/YARN-6040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-6040:
--
Fix Version/s: (was: 3.0.0-beta1)
3.0.0-alpha2
> Introduce api independent PendingAsk to replace usage of ResourceRequest
> within Scheduler classes
> -
>
> Key: YARN-6040
> URL: https://issues.apache.org/jira/browse/YARN-6040
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Fix For: 3.0.0-alpha2
>
> Attachments: YARN-6040.001.patch, YARN-6040.002.patch,
> YARN-6040.003.patch, YARN-6040.004.patch, YARN-6040.005.patch,
> YARN-6040.006.patch, YARN-6040.007.patch
>
>
> As mentioned by YARN-5906, currently schedulers are using ResourceRequest
> heavily so it will be very hard to adopt the new PowerfulResourceRequest
> (YARN-4902).
> This JIRA is the 2nd step of refactoring, which remove usage of
> ResourceRequest from AppSchedulingInfo / SchedulerApplicationAttempt. Instead
> of returning ResourceRequest, it returns a lightweight and API-independent
> object - {{PendingAsk}}.
> The only remained ResourceRequest API of AppSchedulingInfo will be used by
> web service to get list of ResourceRequests.
> So after this patch, usage of ResourceRequest will be isolated inside
> AppSchedulingInfo, so it will be more flexible to update internal data
> structure and upgrade old ResourceRequest API to new.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6040) Introduce api independent PendingAsk to replace usage of ResourceRequest within Scheduler classes
[
https://issues.apache.org/jira/browse/YARN-6040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15805147#comment-15805147
]
Arun Suresh commented on YARN-6040:
---
Committed this to trunk. Thanks [~leftnoteasy].. Looks like we can probably add
some more javadocs. Can you create a followup JIRA for it ?
> Introduce api independent PendingAsk to replace usage of ResourceRequest
> within Scheduler classes
> -
>
> Key: YARN-6040
> URL: https://issues.apache.org/jira/browse/YARN-6040
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Fix For: 3.0.0-alpha2
>
> Attachments: YARN-6040.001.patch, YARN-6040.002.patch,
> YARN-6040.003.patch, YARN-6040.004.patch, YARN-6040.005.patch,
> YARN-6040.006.patch, YARN-6040.007.patch
>
>
> As mentioned by YARN-5906, currently schedulers are using ResourceRequest
> heavily so it will be very hard to adopt the new PowerfulResourceRequest
> (YARN-4902).
> This JIRA is the 2nd step of refactoring, which remove usage of
> ResourceRequest from AppSchedulingInfo / SchedulerApplicationAttempt. Instead
> of returning ResourceRequest, it returns a lightweight and API-independent
> object - {{PendingAsk}}.
> The only remained ResourceRequest API of AppSchedulingInfo will be used by
> web service to get list of ResourceRequests.
> So after this patch, usage of ResourceRequest will be isolated inside
> AppSchedulingInfo, so it will be more flexible to update internal data
> structure and upgrade old ResourceRequest API to new.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6066) Opportunistic containers: Minor fixes: API annotations and config parameter changes
Arun Suresh created YARN-6066: - Summary: Opportunistic containers: Minor fixes: API annotations and config parameter changes Key: YARN-6066 URL: https://issues.apache.org/jira/browse/YARN-6066 Project: Hadoop YARN Issue Type: Bug Reporter: Arun Suresh Priority: Minor Creating this to capture changes suggested by [~leftnoteasy] and [~kasha] in YARN-6041 in its own JIRA. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6066) Opportunistic containers: Minor fixes: API annotations and config parameter changes
[ https://issues.apache.org/jira/browse/YARN-6066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh reassigned YARN-6066: - Assignee: Arun Suresh > Opportunistic containers: Minor fixes: API annotations and config parameter > changes > --- > > Key: YARN-6066 > URL: https://issues.apache.org/jira/browse/YARN-6066 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh >Priority: Minor > > Creating this to capture changes suggested by [~leftnoteasy] and [~kasha] in > YARN-6041 in its own JIRA. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[ https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15805485#comment-15805485 ] Arun Suresh commented on YARN-6041: --- Cherry-picking the JIRAs mentioned in the description to branch-2 shortly. Raised YARN-6066 to commit the changes suggested by [~kasha] and [~leftnoteasy] > Opportunistic containers : Combined patch for branch-2 > --- > > Key: YARN-6041 > URL: https://issues.apache.org/jira/browse/YARN-6041 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-6041-branch-2.001.patch, > YARN-6041-branch-2.002.patch, YARN-6041-branch-2.003.patch > > > This is a combined patch targeting branch-2 of the following JIRAs which have > already been committed to trunk : > YARN-5938. Refactoring OpportunisticContainerAllocator to use > SchedulerRequestKey instead of Priority and other misc fixes > YARN-5646. Add documentation and update config parameter names for scheduling > of OPPORTUNISTIC containers. > YARN-5982. Simplify opportunistic container parameters and metrics. > YARN-5918. Handle Opportunistic scheduling allocate request failure when NM > is lost. > YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager > container lifecycle. > YARN-5823. Update NMTokens in case of requests with only opportunistic > containers. > YARN-5377. Fix > TestQueuingContainerManager.testKillMultipleOpportunisticContainers. > YARN-2995. Enhance UI to show cluster resource utilization of various > container Execution types. > YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http > Address. > YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method > to handle OPPORTUNISTIC container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[ https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-6041: -- Fix Version/s: 2.9.0 > Opportunistic containers : Combined patch for branch-2 > --- > > Key: YARN-6041 > URL: https://issues.apache.org/jira/browse/YARN-6041 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 2.9.0 > > Attachments: YARN-6041-branch-2.001.patch, > YARN-6041-branch-2.002.patch, YARN-6041-branch-2.003.patch > > > This is a combined patch targeting branch-2 of the following JIRAs which have > already been committed to trunk : > YARN-5938. Refactoring OpportunisticContainerAllocator to use > SchedulerRequestKey instead of Priority and other misc fixes > YARN-5646. Add documentation and update config parameter names for scheduling > of OPPORTUNISTIC containers. > YARN-5982. Simplify opportunistic container parameters and metrics. > YARN-5918. Handle Opportunistic scheduling allocate request failure when NM > is lost. > YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager > container lifecycle. > YARN-5823. Update NMTokens in case of requests with only opportunistic > containers. > YARN-5377. Fix > TestQueuingContainerManager.testKillMultipleOpportunisticContainers. > YARN-2995. Enhance UI to show cluster resource utilization of various > container Execution types. > YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http > Address. > YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method > to handle OPPORTUNISTIC container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6066) Opportunistic containers minor fixes: API annotations and config parameter changes
[ https://issues.apache.org/jira/browse/YARN-6066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-6066: -- Attachment: YARN-6066-branch-2.001.patch Attaching patch > Opportunistic containers minor fixes: API annotations and config parameter > changes > -- > > Key: YARN-6066 > URL: https://issues.apache.org/jira/browse/YARN-6066 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh >Priority: Minor > Attachments: YARN-6066-branch-2.001.patch > > > Creating this to capture changes suggested by [~leftnoteasy] and [~kasha] in > YARN-6041 in its own JIRA. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6066) Opportunistic containers minor fixes: API annotations and config parameter changes
[ https://issues.apache.org/jira/browse/YARN-6066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-6066: -- Summary: Opportunistic containers minor fixes: API annotations and config parameter changes (was: Opportunistic containers ninor fixes: API annotations and config parameter changes) > Opportunistic containers minor fixes: API annotations and config parameter > changes > -- > > Key: YARN-6066 > URL: https://issues.apache.org/jira/browse/YARN-6066 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh >Priority: Minor > > Creating this to capture changes suggested by [~leftnoteasy] and [~kasha] in > YARN-6041 in its own JIRA. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6066) Opportunistic containers ninor fixes: API annotations and config parameter changes
[ https://issues.apache.org/jira/browse/YARN-6066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-6066: -- Summary: Opportunistic containers ninor fixes: API annotations and config parameter changes (was: Opportunistic containers: Minor fixes: API annotations and config parameter changes) > Opportunistic containers ninor fixes: API annotations and config parameter > changes > -- > > Key: YARN-6066 > URL: https://issues.apache.org/jira/browse/YARN-6066 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh >Priority: Minor > > Creating this to capture changes suggested by [~leftnoteasy] and [~kasha] in > YARN-6041 in its own JIRA. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5966) AMRMClient changes to support ExecutionType update
[
https://issues.apache.org/jira/browse/YARN-5966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-5966:
--
Attachment: YARN-5966.001.patch
Re-attaching patch, rebasing against trunk
> AMRMClient changes to support ExecutionType update
> --
>
> Key: YARN-5966
> URL: https://issues.apache.org/jira/browse/YARN-5966
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-5966.001.patch, YARN-5966.wip.001.patch
>
>
> {{AMRMClient}} changes to support change of container ExecutionType
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[
https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15787934#comment-15787934
]
Arun Suresh edited comment on YARN-6041 at 12/30/16 4:17 PM:
-
The javac error is from a Testcase (not included in this patch) using a
deprecated API.
The javadoc error will not be fixed, since '_' is an existing identifier.
The checkstyle warnings also will not be fixed, to maintain consistency with
existing code. (I will remove the unused imports when checking in)
The whitespace errors are not actually errors, since they pertain to the
documentation markdown file.
The {{TestRMRestart}} and {{TestContainerManagerSecurity}} testcases are
unrelated and the remaining tests run fine locally.
was (Author: asuresh):
The javac error is from a Testcase (not included in this patch) using a
deprecated API.
The javadoc error will not be fixed, since '_' is an existing identifier.
The checkstyle warnings also will not be fixed, to maintain consistency with
existing code. (I will remove the unused imports when checking in)
The {{TestRMRestart}} and {{TestContainerManagerSecurity}} testcases are
unrelated and the remaining tests run fine locally.
> Opportunistic containers : Combined patch for branch-2
> ---
>
> Key: YARN-6041
> URL: https://issues.apache.org/jira/browse/YARN-6041
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-6041-branch-2.001.patch
>
>
> This is a combined patch targeting branch-2 of the following JIRAs which have
> already been committed to trunk :
> YARN-5938. Refactoring OpportunisticContainerAllocator to use
> SchedulerRequestKey instead of Priority and other misc fixes
> YARN-5646. Add documentation and update config parameter names for scheduling
> of OPPORTUNISTIC containers.
> YARN-5982. Simplify opportunistic container parameters and metrics.
> YARN-5918. Handle Opportunistic scheduling allocate request failure when NM
> is lost.
> YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager
> container lifecycle.
> YARN-5823. Update NMTokens in case of requests with only opportunistic
> containers.
> YARN-5377. Fix
> TestQueuingContainerManager.testKillMultipleOpportunisticContainers.
> YARN-2995. Enhance UI to show cluster resource utilization of various
> container Execution types.
> YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http
> Address.
> YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method
> to handle OPPORTUNISTIC container requests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[
https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15787934#comment-15787934
]
Arun Suresh commented on YARN-6041:
---
The javac error is from a Testcase (not included in this patch) using a
deprecated API.
The javadoc error will not be fixed, since '_' is an existing identifier.
The checkstyle warnings also will not be fixed, to maintain consistency with
existing code. (I will remove the unused imports when checking in)
The {{TestRMRestart}} and {{TestContainerManagerSecurity}} testcases are
unrelated and the remaining tests run fine locally.
> Opportunistic containers : Combined patch for branch-2
> ---
>
> Key: YARN-6041
> URL: https://issues.apache.org/jira/browse/YARN-6041
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-6041-branch-2.001.patch
>
>
> This is a combined patch targeting branch-2 of the following JIRAs which have
> already been committed to trunk :
> YARN-5938. Refactoring OpportunisticContainerAllocator to use
> SchedulerRequestKey instead of Priority and other misc fixes
> YARN-5646. Add documentation and update config parameter names for scheduling
> of OPPORTUNISTIC containers.
> YARN-5982. Simplify opportunistic container parameters and metrics.
> YARN-5918. Handle Opportunistic scheduling allocate request failure when NM
> is lost.
> YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager
> container lifecycle.
> YARN-5823. Update NMTokens in case of requests with only opportunistic
> containers.
> YARN-5377. Fix
> TestQueuingContainerManager.testKillMultipleOpportunisticContainers.
> YARN-2995. Enhance UI to show cluster resource utilization of various
> container Execution types.
> YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http
> Address.
> YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method
> to handle OPPORTUNISTIC container requests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[ https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-6041: -- Attachment: YARN-6041-branch-2.002.patch Thanks for the review [~leftnoteasy] Uploading new patch: # Added the extra @Unstable annotations to ContainerState and YarnConfiguration you had requested. # Fixed some unused imports On the NM side, there really are no API changes pre se (except the addition of the SCHEDULED ContainerState). > Opportunistic containers : Combined patch for branch-2 > --- > > Key: YARN-6041 > URL: https://issues.apache.org/jira/browse/YARN-6041 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-6041-branch-2.001.patch, > YARN-6041-branch-2.002.patch > > > This is a combined patch targeting branch-2 of the following JIRAs which have > already been committed to trunk : > YARN-5938. Refactoring OpportunisticContainerAllocator to use > SchedulerRequestKey instead of Priority and other misc fixes > YARN-5646. Add documentation and update config parameter names for scheduling > of OPPORTUNISTIC containers. > YARN-5982. Simplify opportunistic container parameters and metrics. > YARN-5918. Handle Opportunistic scheduling allocate request failure when NM > is lost. > YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager > container lifecycle. > YARN-5823. Update NMTokens in case of requests with only opportunistic > containers. > YARN-5377. Fix > TestQueuingContainerManager.testKillMultipleOpportunisticContainers. > YARN-2995. Enhance UI to show cluster resource utilization of various > container Execution types. > YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http > Address. > YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method > to handle OPPORTUNISTIC container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5959) RM changes to support change of container ExecutionType
[ https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5959: -- Attachment: YARN-5959-YARN-5085.002.patch Fixing findbugs. > RM changes to support change of container ExecutionType > --- > > Key: YARN-5959 > URL: https://issues.apache.org/jira/browse/YARN-5959 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5959-YARN-5085.001.patch, > YARN-5959-YARN-5085.002.patch, YARN-5959.combined.001.patch, > YARN-5959.wip.002.patch, YARN-5959.wip.003.patch, YARN-5959.wip.patch > > > RM side changes to allow an AM to ask for change of ExecutionType. > Currently, there are two cases: > # *Promotion* : OPPORTUNISTIC to GUARANTEED. > # *Demotion* : GUARANTEED to OPPORTUNISTIC. > This is similar in YARN-1197 which allows for change in Container resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5959) RM changes to support change of container ExecutionType
[
https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-5959:
--
Attachment: YARN-5959-YARN-5985.001.patch
Updating patch.
* The container increase and decreases accounting has been moved to the
{{AppSchedulingInfo}}.
* To isolate the update code path (and to make it easier to review), i've
created a {{ContainerUpdateContext}} which is field of the
{{AppSchedulingInfo}}. It also encapsulates the datastructures used for
accounting the container increase and decrease.
The new functionality can be used for both ExecutionType Promotion/Demotion as
well as Container resource Increase/Decrease. I have currently just hooked it
to the ExecutionType promote and demote. I feel we should raise another JIRA to
work on removing the existing container increase/decrease code-path and move it
to using the new ContainerUpdateContext.
As with the previous patches, the basic principle is the same.. new
ResourceRequests are created for container increases, but these are created
internally in the {{AppSchedulingInfo}}
[~leftnoteasy], do give it a look.
> RM changes to support change of container ExecutionType
> ---
>
> Key: YARN-5959
> URL: https://issues.apache.org/jira/browse/YARN-5959
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-5959-YARN-5985.001.patch,
> YARN-5959.combined.001.patch, YARN-5959.wip.002.patch,
> YARN-5959.wip.003.patch, YARN-5959.wip.patch
>
>
> RM side changes to allow an AM to ask for change of ExecutionType.
> Currently, there are two cases:
> # *Promotion* : OPPORTUNISTIC to GUARANTEED.
> # *Demotion* : GUARANTEED to OPPORTUNISTIC.
> This is similar in YARN-1197 which allows for change in Container resources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5959) RM changes to support change of container ExecutionType
[ https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5959: -- Attachment: YARN-5959-YARN-5085.001.patch Correcting name of patch file to kick of Jenkins against YARN-5085 branch. > RM changes to support change of container ExecutionType > --- > > Key: YARN-5959 > URL: https://issues.apache.org/jira/browse/YARN-5959 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5959-YARN-5085.001.patch, > YARN-5959.combined.001.patch, YARN-5959.wip.002.patch, > YARN-5959.wip.003.patch, YARN-5959.wip.patch > > > RM side changes to allow an AM to ask for change of ExecutionType. > Currently, there are two cases: > # *Promotion* : OPPORTUNISTIC to GUARANTEED. > # *Demotion* : GUARANTEED to OPPORTUNISTIC. > This is similar in YARN-1197 which allows for change in Container resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5959) RM changes to support change of container ExecutionType
[ https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5959: -- Attachment: (was: YARN-5959-YARN-5985.001.patch) > RM changes to support change of container ExecutionType > --- > > Key: YARN-5959 > URL: https://issues.apache.org/jira/browse/YARN-5959 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5959-YARN-5085.001.patch, > YARN-5959.combined.001.patch, YARN-5959.wip.002.patch, > YARN-5959.wip.003.patch, YARN-5959.wip.patch > > > RM side changes to allow an AM to ask for change of ExecutionType. > Currently, there are two cases: > # *Promotion* : OPPORTUNISTIC to GUARANTEED. > # *Demotion* : GUARANTEED to OPPORTUNISTIC. > This is similar in YARN-1197 which allows for change in Container resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6022) Revert changes of AbstractResourceRequest
[
https://issues.apache.org/jira/browse/YARN-6022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795677#comment-15795677
]
Arun Suresh commented on YARN-6022:
---
[~leftnoteasy], [~kasha], do you guys want to just move this to @Public
@Unstable now (thereby unblocking the release) and maybe refactor this later ?
> Revert changes of AbstractResourceRequest
> -
>
> Key: YARN-6022
> URL: https://issues.apache.org/jira/browse/YARN-6022
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Wangda Tan
>Priority: Blocker
>
> YARN-5774 added AbstractResourceRequest to make easier internal scheduler
> change, this is not a correct approach: For example, with this change, we
> need to make AbstractResourceRequest to be public/stable. And end users can
> use it like:
> {code}
> AbstractResourceRequest request = ...
> request.setCapability(...)
> {code}
> But AbstractResourceRequest should not be visible by application at all.
> We need to revert it from branch-2.8 / branch-2 / trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[ https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15796705#comment-15796705 ] Arun Suresh commented on YARN-6041: --- Thanks for taking a look [~kasha], bq. Assuming MRJobConfig. MR_NUM_OPPORTUNISTIC_MAPS_PER_100 was not part of any release. Can you confirm the same? I confirm this hasn't seen light of day in any previous release :) bq. Assuming the incompatible proto changes to yarn_server_common_protos and yarn_server_common_service_protos are okay because the previous versions didn't make it to a release. Can you confirm the same? Ditto.. bq. Nit: For scheduler-related configs, should we use yarn.scheduler.* like other scheduler configs? And, yarn.nodemanager.* for all node-specific opportunistic container configs? Making sure the configs are consistent and simple is a good exercise before we include these in a release. Hmm... I guess you are referring to YARN-5646. Looks like the patch was modified to incorporate your suggestions wrt the parameter names. Also, since this doesn't affect the scheduler per se.. the RM side changes are all yarn.resourcemanager.* and the nodemanager ones are yarn.nodemanager.* Good catch on the method names... will update patch shortly > Opportunistic containers : Combined patch for branch-2 > --- > > Key: YARN-6041 > URL: https://issues.apache.org/jira/browse/YARN-6041 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-6041-branch-2.001.patch, > YARN-6041-branch-2.002.patch > > > This is a combined patch targeting branch-2 of the following JIRAs which have > already been committed to trunk : > YARN-5938. Refactoring OpportunisticContainerAllocator to use > SchedulerRequestKey instead of Priority and other misc fixes > YARN-5646. Add documentation and update config parameter names for scheduling > of OPPORTUNISTIC containers. > YARN-5982. Simplify opportunistic container parameters and metrics. > YARN-5918. Handle Opportunistic scheduling allocate request failure when NM > is lost. > YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager > container lifecycle. > YARN-5823. Update NMTokens in case of requests with only opportunistic > containers. > YARN-5377. Fix > TestQueuingContainerManager.testKillMultipleOpportunisticContainers. > YARN-2995. Enhance UI to show cluster resource utilization of various > container Execution types. > YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http > Address. > YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method > to handle OPPORTUNISTIC container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5959) RM changes to support change of container ExecutionType
[ https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5959: -- Attachment: YARN-5959-YARN-5085.005.patch Updating patch v005 Some minor updates: Moved ContainerUpdateRequests to its own class and removed updateErrors from it. > RM changes to support change of container ExecutionType > --- > > Key: YARN-5959 > URL: https://issues.apache.org/jira/browse/YARN-5959 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5959-YARN-5085.001.patch, > YARN-5959-YARN-5085.002.patch, YARN-5959-YARN-5085.003.patch, > YARN-5959-YARN-5085.004.patch, YARN-5959-YARN-5085.005.patch, > YARN-5959.combined.001.patch, YARN-5959.wip.002.patch, > YARN-5959.wip.003.patch, YARN-5959.wip.patch > > > RM side changes to allow an AM to ask for change of ExecutionType. > Currently, there are two cases: > # *Promotion* : OPPORTUNISTIC to GUARANTEED. > # *Demotion* : GUARANTEED to OPPORTUNISTIC. > This is similar in YARN-1197 which allows for change in Container resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5959) RM changes to support change of container ExecutionType
[ https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799772#comment-15799772 ] Arun Suresh edited comment on YARN-5959 at 1/5/17 12:06 AM: Updated patch. Do give it another review [~leftnoteasy] bq. Can we make scheduler.allocate to have a single ContainerUpdateRequests, which include 4 lists (increase/decrease/promotion/demotion) instead of making all of them become top-level parameter, similarly for Allocation. I've created a holder class ContainerUpdateContext.ContainerUpdateRequests which is created in the RMServerUtils#validate.. method and passed around. I don't think we need one for the Allocation object, since technically, it itself is a holder for all the lists of allocated/updated containers. Agree to all other suggestions, and its incorporated in the patch. was (Author: asuresh): Updated patch. Do give it another review [~leftnoteasy] bq. Can we make scheduler.allocate to have a single ContainerUpdateRequests, which include 4 lists (increase/decrease/promotion/demotion) instead of making all of them become top-level parameter, similarly for Allocation. I've created a holder class ContainerUpdateContext.ContainerUpdateRequests which is created in the RMServerUtils#validate.. method and passed around. I don't think we need one for the Allocation object, since technically, it itself is a holder for all the lists. Agree to all other suggestions, and its incorporated in the patch. > RM changes to support change of container ExecutionType > --- > > Key: YARN-5959 > URL: https://issues.apache.org/jira/browse/YARN-5959 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5959-YARN-5085.001.patch, > YARN-5959-YARN-5085.002.patch, YARN-5959-YARN-5085.003.patch, > YARN-5959-YARN-5085.004.patch, YARN-5959.combined.001.patch, > YARN-5959.wip.002.patch, YARN-5959.wip.003.patch, YARN-5959.wip.patch > > > RM side changes to allow an AM to ask for change of ExecutionType. > Currently, there are two cases: > # *Promotion* : OPPORTUNISTIC to GUARANTEED. > # *Demotion* : GUARANTEED to OPPORTUNISTIC. > This is similar in YARN-1197 which allows for change in Container resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5959) RM changes to support change of container ExecutionType
[ https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5959: -- Attachment: YARN-5959-YARN-5085.004.patch Updated patch. Do give it another review [~leftnoteasy] bq. Can we make scheduler.allocate to have a single ContainerUpdateRequests, which include 4 lists (increase/decrease/promotion/demotion) instead of making all of them become top-level parameter, similarly for Allocation. I've created a holder class ContainerUpdateContext.ContainerUpdateRequests which is created in the RMServerUtils#validate.. method and passed around. I don't think we need one for the Allocation object, since technically, it itself is a holder for all the lists. Agree to all other suggestions, and its incorporated in the patch. > RM changes to support change of container ExecutionType > --- > > Key: YARN-5959 > URL: https://issues.apache.org/jira/browse/YARN-5959 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5959-YARN-5085.001.patch, > YARN-5959-YARN-5085.002.patch, YARN-5959-YARN-5085.003.patch, > YARN-5959-YARN-5085.004.patch, YARN-5959.combined.001.patch, > YARN-5959.wip.002.patch, YARN-5959.wip.003.patch, YARN-5959.wip.patch > > > RM side changes to allow an AM to ask for change of ExecutionType. > Currently, there are two cases: > # *Promotion* : OPPORTUNISTIC to GUARANTEED. > # *Demotion* : GUARANTEED to OPPORTUNISTIC. > This is similar in YARN-1197 which allows for change in Container resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5959) RM changes to support change of container ExecutionType
[ https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5959: -- Attachment: YARN-5959.005.patch Attaching trunk patch.. Thanks for the review [~leftnoteasy]. > RM changes to support change of container ExecutionType > --- > > Key: YARN-5959 > URL: https://issues.apache.org/jira/browse/YARN-5959 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5959-YARN-5085.001.patch, > YARN-5959-YARN-5085.002.patch, YARN-5959-YARN-5085.003.patch, > YARN-5959-YARN-5085.004.patch, YARN-5959-YARN-5085.005.patch, > YARN-5959.005.patch, YARN-5959.combined.001.patch, YARN-5959.wip.002.patch, > YARN-5959.wip.003.patch, YARN-5959.wip.patch > > > RM side changes to allow an AM to ask for change of ExecutionType. > Currently, there are two cases: > # *Promotion* : OPPORTUNISTIC to GUARANTEED. > # *Demotion* : GUARANTEED to OPPORTUNISTIC. > This is similar in YARN-1197 which allows for change in Container resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[ https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-6041: -- Attachment: YARN-6041-branch-2.003.patch Updating patch based on [~kasha]'s suggestions > Opportunistic containers : Combined patch for branch-2 > --- > > Key: YARN-6041 > URL: https://issues.apache.org/jira/browse/YARN-6041 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-6041-branch-2.001.patch, > YARN-6041-branch-2.002.patch, YARN-6041-branch-2.003.patch > > > This is a combined patch targeting branch-2 of the following JIRAs which have > already been committed to trunk : > YARN-5938. Refactoring OpportunisticContainerAllocator to use > SchedulerRequestKey instead of Priority and other misc fixes > YARN-5646. Add documentation and update config parameter names for scheduling > of OPPORTUNISTIC containers. > YARN-5982. Simplify opportunistic container parameters and metrics. > YARN-5918. Handle Opportunistic scheduling allocate request failure when NM > is lost. > YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager > container lifecycle. > YARN-5823. Update NMTokens in case of requests with only opportunistic > containers. > YARN-5377. Fix > TestQueuingContainerManager.testKillMultipleOpportunisticContainers. > YARN-2995. Enhance UI to show cluster resource utilization of various > container Execution types. > YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http > Address. > YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method > to handle OPPORTUNISTIC container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5959) RM changes to support change of container ExecutionType
[
https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-5959:
--
Attachment: YARN-5959-YARN-5085.003.patch
Updating patch.
Thanks for the detailed review [~leftnoteasy].
I agree with all your suggestions. I've updated the patch as follows:
# Reverted all changes to the {{AbstractResourceRequest}}. We don't need the
extra {{containerToUpdate}} field anymore, as you pointed out, it was possible
to accomplish everything via the {{SchedulerRequestKey}}
# Modified the call flow remove the update Execution type call from the OCAAMS
and moved it into the Scheduler itself.
# Modified the Allocation object to handle the promoted and demoted containers.
# Split the ContainerUpdateType::UPDATE_EXECUTION_TYPE to
PROMOTE_EXECUTION_TYPE and DEMOTE_EXECUTION_TYPE.
The size of the patch has increased a bit since I modified the allocate()
method of the scheduler to take the promoted and demoted lists. And this
touches a few test classes.
> RM changes to support change of container ExecutionType
> ---
>
> Key: YARN-5959
> URL: https://issues.apache.org/jira/browse/YARN-5959
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-5959-YARN-5085.001.patch,
> YARN-5959-YARN-5085.002.patch, YARN-5959-YARN-5085.003.patch,
> YARN-5959.combined.001.patch, YARN-5959.wip.002.patch,
> YARN-5959.wip.003.patch, YARN-5959.wip.patch
>
>
> RM side changes to allow an AM to ask for change of ExecutionType.
> Currently, there are two cases:
> # *Promotion* : OPPORTUNISTIC to GUARANTEED.
> # *Demotion* : GUARANTEED to OPPORTUNISTIC.
> This is similar in YARN-1197 which allows for change in Container resources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[ https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15801509#comment-15801509 ] Arun Suresh commented on YARN-6041: --- As mentioned earlier, the javac, javadoc and whitespace errors are better left unfixed to retain the style of the existing files. The Testcase errors are not related. [~kasha] / [~leftnoteasy], do let me know if this is good for checkin > Opportunistic containers : Combined patch for branch-2 > --- > > Key: YARN-6041 > URL: https://issues.apache.org/jira/browse/YARN-6041 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-6041-branch-2.001.patch, > YARN-6041-branch-2.002.patch, YARN-6041-branch-2.003.patch > > > This is a combined patch targeting branch-2 of the following JIRAs which have > already been committed to trunk : > YARN-5938. Refactoring OpportunisticContainerAllocator to use > SchedulerRequestKey instead of Priority and other misc fixes > YARN-5646. Add documentation and update config parameter names for scheduling > of OPPORTUNISTIC containers. > YARN-5982. Simplify opportunistic container parameters and metrics. > YARN-5918. Handle Opportunistic scheduling allocate request failure when NM > is lost. > YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager > container lifecycle. > YARN-5823. Update NMTokens in case of requests with only opportunistic > containers. > YARN-5377. Fix > TestQueuingContainerManager.testKillMultipleOpportunisticContainers. > YARN-2995. Enhance UI to show cluster resource utilization of various > container Execution types. > YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http > Address. > YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method > to handle OPPORTUNISTIC container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6059) Update paused container state in the state store
[ https://issues.apache.org/jira/browse/YARN-6059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15807913#comment-15807913 ] Arun Suresh commented on YARN-6059: --- Thanks for raising this Hitesh.. Can you also maybe also outline the NM recovery strategy for paused containers here? > Update paused container state in the state store > > > Key: YARN-6059 > URL: https://issues.apache.org/jira/browse/YARN-6059 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Hitesh Sharma >Assignee: Hitesh Sharma > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5823) Update NMTokens in case of requests with only opportunistic containers
[
https://issues.apache.org/jira/browse/YARN-5823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-5823:
--
Fix Version/s: 2.9.0
> Update NMTokens in case of requests with only opportunistic containers
> --
>
> Key: YARN-5823
> URL: https://issues.apache.org/jira/browse/YARN-5823
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5823.001.patch, YARN-5823.002.patch,
> YARN-5823.003.patch, YARN-5823.004.patch
>
>
> At the moment, when an {{AllocateRequest}} contains only opportunistic
> {{ResourceRequests}}, the updated NMTokens are not properly added to the
> {{AllocateResponse}}.
> In such a case the AM does not get back the needed NMTokens that are required
> to start the opportunistic containers at the respective nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5823) Update NMTokens in case of requests with only opportunistic containers
[
https://issues.apache.org/jira/browse/YARN-5823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15786345#comment-15786345
]
Arun Suresh commented on YARN-5823:
---
Committing this to branch-2
> Update NMTokens in case of requests with only opportunistic containers
> --
>
> Key: YARN-5823
> URL: https://issues.apache.org/jira/browse/YARN-5823
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5823.001.patch, YARN-5823.002.patch,
> YARN-5823.003.patch, YARN-5823.004.patch
>
>
> At the moment, when an {{AllocateRequest}} contains only opportunistic
> {{ResourceRequests}}, the updated NMTokens are not properly added to the
> {{AllocateResponse}}.
> In such a case the AM does not get back the needed NMTokens that are required
> to start the opportunistic containers at the respective nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5377) Fix TestQueuingContainerManager.testKillMultipleOpportunisticContainers
[
https://issues.apache.org/jira/browse/YARN-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15786343#comment-15786343
]
Arun Suresh commented on YARN-5377:
---
Committing this to branch-2
> Fix TestQueuingContainerManager.testKillMultipleOpportunisticContainers
> ---
>
> Key: YARN-5377
> URL: https://issues.apache.org/jira/browse/YARN-5377
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Konstantinos Karanasos
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5377.001.patch
>
>
> Test case fails jenkin build
> [link|https://builds.apache.org/job/PreCommit-YARN-Build/12228/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt]
> {noformat}
> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 134.586 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager
> testKillMultipleOpportunisticContainers(org.apache.hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager)
> Time elapsed: 32.134 sec <<< FAILURE!
> java.lang.AssertionError: ContainerState is not correct (timedout)
> expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.BaseContainerManagerTest.waitForNMContainerState(BaseContainerManagerTest.java:363)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager.testKillMultipleOpportunisticContainers(TestQueuingContainerManager.java:470)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4597) Introduce ContainerScheduler and a SCHEDULED state to NodeManager container lifecycle
[ https://issues.apache.org/jira/browse/YARN-4597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-4597: -- Fix Version/s: 2.9.0 > Introduce ContainerScheduler and a SCHEDULED state to NodeManager container > lifecycle > - > > Key: YARN-4597 > URL: https://issues.apache.org/jira/browse/YARN-4597 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager >Reporter: Chris Douglas >Assignee: Arun Suresh > Labels: oct16-hard > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-4597.001.patch, YARN-4597.002.patch, > YARN-4597.003.patch, YARN-4597.004.patch, YARN-4597.005.patch, > YARN-4597.006.patch, YARN-4597.007.patch, YARN-4597.008.patch, > YARN-4597.009.patch, YARN-4597.010.patch, YARN-4597.011.patch, > YARN-4597.012.patch, YARN-4597.013.patch > > > Currently, the NM immediately launches containers after resource > localization. Several features could be more cleanly implemented if the NM > included a separate stage for reserving resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6041) Opportunistic containers : Combined patch for branch-2
[ https://issues.apache.org/jira/browse/YARN-6041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-6041: -- Attachment: YARN-6041-branch-2.001.patch Attaching patch > Opportunistic containers : Combined patch for branch-2 > --- > > Key: YARN-6041 > URL: https://issues.apache.org/jira/browse/YARN-6041 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 2.9.0 > > Attachments: YARN-6041-branch-2.001.patch > > > This is a combined patch targeting branch-2 of the following JIRAs which have > already been committed to trunk : > YARN-5938. Refactoring OpportunisticContainerAllocator to use > SchedulerRequestKey instead of Priority and other misc fixes > YARN-5646. Add documentation and update config parameter names for scheduling > of OPPORTUNISTIC containers. > YARN-5982. Simplify opportunistic container parameters and metrics. > YARN-5918. Handle Opportunistic scheduling allocate request failure when NM > is lost. > YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager > container lifecycle. > YARN-5823. Update NMTokens in case of requests with only opportunistic > containers. > YARN-5377. Fix > TestQueuingContainerManager.testKillMultipleOpportunisticContainers. > YARN-2995. Enhance UI to show cluster resource utilization of various > container Execution types. > YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http > Address. > YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method > to handle OPPORTUNISTIC container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4597) Introduce ContainerScheduler and a SCHEDULED state to NodeManager container lifecycle
[ https://issues.apache.org/jira/browse/YARN-4597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15786348#comment-15786348 ] Arun Suresh commented on YARN-4597: --- Committing this to branch-2 > Introduce ContainerScheduler and a SCHEDULED state to NodeManager container > lifecycle > - > > Key: YARN-4597 > URL: https://issues.apache.org/jira/browse/YARN-4597 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager >Reporter: Chris Douglas >Assignee: Arun Suresh > Labels: oct16-hard > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-4597.001.patch, YARN-4597.002.patch, > YARN-4597.003.patch, YARN-4597.004.patch, YARN-4597.005.patch, > YARN-4597.006.patch, YARN-4597.007.patch, YARN-4597.008.patch, > YARN-4597.009.patch, YARN-4597.010.patch, YARN-4597.011.patch, > YARN-4597.012.patch, YARN-4597.013.patch > > > Currently, the NM immediately launches containers after resource > localization. Several features could be more cleanly implemented if the NM > included a separate stage for reserving resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5918) Handle Opportunistic scheduling allocate request failure when NM is lost
[
https://issues.apache.org/jira/browse/YARN-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15786350#comment-15786350
]
Arun Suresh commented on YARN-5918:
---
Committing this to branch-2
> Handle Opportunistic scheduling allocate request failure when NM is lost
>
>
> Key: YARN-5918
> URL: https://issues.apache.org/jira/browse/YARN-5918
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5918.0001.patch, YARN-5918.0002.patch,
> YARN-5918.0003.patch, YARN-5918.0004.patch
>
>
> Allocate request failure during Opportunistic container allocation when
> nodemanager is lost
> {noformat}
> 2016-11-20 10:38:49,011 INFO
> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root
> OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS
> APPID=application_1479637990302_0002
> CONTAINERID=container_e12_1479637990302_0002_01_06
> RESOURCE=
> 2016-11-20 10:38:49,011 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
> Removed node docker2:38297 clusterResource:
> 2016-11-20 10:38:49,434 WARN org.apache.hadoop.ipc.Server: IPC Server handler
> 7 on 8030, call Call#35 Retry#0
> org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.allocate from
> 172.17.0.2:51584
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.convertToRemoteNode(OpportunisticContainerAllocatorAMService.java:420)
> at
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.convertToRemoteNodes(OpportunisticContainerAllocatorAMService.java:412)
> at
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.getLeastLoadedNodes(OpportunisticContainerAllocatorAMService.java:402)
> at
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.allocate(OpportunisticContainerAllocatorAMService.java:236)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:467)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:990)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:846)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:789)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2539)
> 2016-11-20 10:38:50,824 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
> container_e12_1479637990302_0002_01_02 Container Transitioned from
> RUNNING to COMPLETED
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5982) Simplify opportunistic container parameters and metrics
[ https://issues.apache.org/jira/browse/YARN-5982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5982: -- Fix Version/s: 2.9.0 > Simplify opportunistic container parameters and metrics > --- > > Key: YARN-5982 > URL: https://issues.apache.org/jira/browse/YARN-5982 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5982.001.patch, YARN-5982.002.patch, > YARN-5982.003.patch > > > This JIRA removes some of the parameters that are related to opportunistic > containers (e.g., min/max memory/cpu). Instead, we will be using the > parameters already used by guaranteed containers. > The goal is to reduce the number of parameters that need to be used by the > user. > We also fix a small issue related to the container metrics (opportunistic > memory reported in GB in Web UI, although it was in MB). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5938) Refactoring OpportunisticContainerAllocator to use SchedulerRequestKey instead of Priority and other misc fixes
[ https://issues.apache.org/jira/browse/YARN-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15786356#comment-15786356 ] Arun Suresh commented on YARN-5938: --- Committing this to branch-2 > Refactoring OpportunisticContainerAllocator to use SchedulerRequestKey > instead of Priority and other misc fixes > --- > > Key: YARN-5938 > URL: https://issues.apache.org/jira/browse/YARN-5938 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 3.0.0-alpha2 > > Attachments: YARN-5938-YARN-5085.001.patch, > YARN-5938-YARN-5085.002.patch, YARN-5938-YARN-5085.003.patch, > YARN-5938-YARN-5085.004.patch, YARN-5938.001.patch, YARN-5938.002.patch, > YARN-5938.003.patch > > > Minor code re-organization to do the following: > # The OpportunisticContainerAllocatorAMService currently allocates outside > the ApplicationAttempt lock maintained by the ApplicationMasterService. This > should happen inside the lock. > # Refactored out some code to simplify the allocate() method. > # Removed some unused fields inside the OpportunisticContainerAllocator. > # Re-organized some of the code in the > OpportunisticContainerAllocatorAMService::allocate method to make it a bit > more readable. > # Moved SchedulerRequestKey to a new package, so it can be used by the > OpportunisticContainerAllocator/Context. > # Moved all usages of Priority in the OpportunisticContainerAllocator -> > SchedulerRequestKey. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5646) Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers
[ https://issues.apache.org/jira/browse/YARN-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15786354#comment-15786354 ] Arun Suresh commented on YARN-5646: --- Committing this to branch-2 > Add documentation and update config parameter names for scheduling of > OPPORTUNISTIC containers > -- > > Key: YARN-5646 > URL: https://issues.apache.org/jira/browse/YARN-5646 > Project: Hadoop YARN > Issue Type: Task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos >Priority: Blocker > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5646.001.patch, YARN-5646.002.patch, > YARN-5646.003.patch, YARN-5646.004.patch > > > This is for adding documentation regarding the scheduling of OPPORTUNISTIC > containers. > It includes both the centralized (YARN-5220) and the distributed (YARN-2877) > scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5646) Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers
[ https://issues.apache.org/jira/browse/YARN-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5646: -- Fix Version/s: 2.9.0 > Add documentation and update config parameter names for scheduling of > OPPORTUNISTIC containers > -- > > Key: YARN-5646 > URL: https://issues.apache.org/jira/browse/YARN-5646 > Project: Hadoop YARN > Issue Type: Task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos >Priority: Blocker > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5646.001.patch, YARN-5646.002.patch, > YARN-5646.003.patch, YARN-5646.004.patch > > > This is for adding documentation regarding the scheduling of OPPORTUNISTIC > containers. > It includes both the centralized (YARN-5220) and the distributed (YARN-2877) > scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5918) Handle Opportunistic scheduling allocate request failure when NM is lost
[
https://issues.apache.org/jira/browse/YARN-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-5918:
--
Fix Version/s: 2.9.0
> Handle Opportunistic scheduling allocate request failure when NM is lost
>
>
> Key: YARN-5918
> URL: https://issues.apache.org/jira/browse/YARN-5918
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5918.0001.patch, YARN-5918.0002.patch,
> YARN-5918.0003.patch, YARN-5918.0004.patch
>
>
> Allocate request failure during Opportunistic container allocation when
> nodemanager is lost
> {noformat}
> 2016-11-20 10:38:49,011 INFO
> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root
> OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS
> APPID=application_1479637990302_0002
> CONTAINERID=container_e12_1479637990302_0002_01_06
> RESOURCE=
> 2016-11-20 10:38:49,011 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
> Removed node docker2:38297 clusterResource:
> 2016-11-20 10:38:49,434 WARN org.apache.hadoop.ipc.Server: IPC Server handler
> 7 on 8030, call Call#35 Retry#0
> org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB.allocate from
> 172.17.0.2:51584
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.convertToRemoteNode(OpportunisticContainerAllocatorAMService.java:420)
> at
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.convertToRemoteNodes(OpportunisticContainerAllocatorAMService.java:412)
> at
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.getLeastLoadedNodes(OpportunisticContainerAllocatorAMService.java:402)
> at
> org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.allocate(OpportunisticContainerAllocatorAMService.java:236)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:467)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:990)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:846)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:789)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1857)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2539)
> 2016-11-20 10:38:50,824 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
> container_e12_1479637990302_0002_01_02 Container Transitioned from
> RUNNING to COMPLETED
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5982) Simplify opportunistic container parameters and metrics
[ https://issues.apache.org/jira/browse/YARN-5982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15786352#comment-15786352 ] Arun Suresh commented on YARN-5982: --- Committing this to branch-2 > Simplify opportunistic container parameters and metrics > --- > > Key: YARN-5982 > URL: https://issues.apache.org/jira/browse/YARN-5982 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5982.001.patch, YARN-5982.002.patch, > YARN-5982.003.patch > > > This JIRA removes some of the parameters that are related to opportunistic > containers (e.g., min/max memory/cpu). Instead, we will be using the > parameters already used by guaranteed containers. > The goal is to reduce the number of parameters that need to be used by the > user. > We also fix a small issue related to the container metrics (opportunistic > memory reported in GB in Web UI, although it was in MB). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5486) Update OpportunisticContainerAllocatorAMService::allocate method to handle OPPORTUNISTIC container requests
[ https://issues.apache.org/jira/browse/YARN-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5486: -- Fix Version/s: 2.9.0 > Update OpportunisticContainerAllocatorAMService::allocate method to handle > OPPORTUNISTIC container requests > --- > > Key: YARN-5486 > URL: https://issues.apache.org/jira/browse/YARN-5486 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Arun Suresh >Assignee: Konstantinos Karanasos > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5486.001.patch, YARN-5486.002.patch, > YARN-5486.003.patch, YARN-5486.004.patch > > > YARN-5457 refactors the Distributed Scheduling framework to move the > container allocator to yarn-server-common. > This JIRA proposes to update the allocate method in the new AM service to use > the OpportunisticContainerAllocator to allocate opportunistic containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5486) Update OpportunisticContainerAllocatorAMService::allocate method to handle OPPORTUNISTIC container requests
[ https://issues.apache.org/jira/browse/YARN-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15786337#comment-15786337 ] Arun Suresh commented on YARN-5486: --- Committing this to branch-2. YARN-5688 can be worked on independently > Update OpportunisticContainerAllocatorAMService::allocate method to handle > OPPORTUNISTIC container requests > --- > > Key: YARN-5486 > URL: https://issues.apache.org/jira/browse/YARN-5486 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Arun Suresh >Assignee: Konstantinos Karanasos > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5486.001.patch, YARN-5486.002.patch, > YARN-5486.003.patch, YARN-5486.004.patch > > > YARN-5457 refactors the Distributed Scheduling framework to move the > container allocator to yarn-server-common. > This JIRA proposes to update the allocate method in the new AM service to use > the OpportunisticContainerAllocator to allocate opportunistic containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5799) Fix Opportunistic Allocation to set the correct value of Node Http Address
[ https://issues.apache.org/jira/browse/YARN-5799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5799: -- Fix Version/s: 2.9.0 > Fix Opportunistic Allocation to set the correct value of Node Http Address > --- > > Key: YARN-5799 > URL: https://issues.apache.org/jira/browse/YARN-5799 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5799.001.patch, YARN-5799.002.patch > > > This proposes to fix the the OpportunisticAllocator, used to allocate > OPPORTUNISTIC containers (both Centrally as well as in a Distributed manner) > to correctly populate the Node Http Address in the returned Container. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6041) Opportunistic containers : Combined patch for branch-2
Arun Suresh created YARN-6041: - Summary: Opportunistic containers : Combined patch for branch-2 Key: YARN-6041 URL: https://issues.apache.org/jira/browse/YARN-6041 Project: Hadoop YARN Issue Type: Bug Reporter: Arun Suresh Assignee: Arun Suresh Fix For: 2.9.0 This is a combined patch targeting branch-2 of the following JIRAs which have already been committed to trunk : YARN-5938. Refactoring OpportunisticContainerAllocator to use SchedulerRequestKey instead of Priority and other misc fixes YARN-5646. Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers. YARN-5982. Simplify opportunistic container parameters and metrics. YARN-5918. Handle Opportunistic scheduling allocate request failure when NM is lost. YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager container lifecycle. YARN-5823. Update NMTokens in case of requests with only opportunistic containers. YARN-5377. Fix TestQueuingContainerManager.testKillMultipleOpportunisticContainers. YARN-2995. Enhance UI to show cluster resource utilization of various container Execution types. YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http Address. YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method to handle OPPORTUNISTIC container requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5377) Fix TestQueuingContainerManager.testKillMultipleOpportunisticContainers
[
https://issues.apache.org/jira/browse/YARN-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-5377:
--
Fix Version/s: 2.9.0
> Fix TestQueuingContainerManager.testKillMultipleOpportunisticContainers
> ---
>
> Key: YARN-5377
> URL: https://issues.apache.org/jira/browse/YARN-5377
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Konstantinos Karanasos
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5377.001.patch
>
>
> Test case fails jenkin build
> [link|https://builds.apache.org/job/PreCommit-YARN-Build/12228/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt]
> {noformat}
> Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 134.586 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager
> testKillMultipleOpportunisticContainers(org.apache.hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager)
> Time elapsed: 32.134 sec <<< FAILURE!
> java.lang.AssertionError: ContainerState is not correct (timedout)
> expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.BaseContainerManagerTest.waitForNMContainerState(BaseContainerManagerTest.java:363)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager.testKillMultipleOpportunisticContainers(TestQueuingContainerManager.java:470)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2995) Enhance UI to show cluster resource utilization of various container Execution types
[ https://issues.apache.org/jira/browse/YARN-2995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15786342#comment-15786342 ] Arun Suresh commented on YARN-2995: --- Committing this to branch-2 > Enhance UI to show cluster resource utilization of various container > Execution types > > > Key: YARN-2995 > URL: https://issues.apache.org/jira/browse/YARN-2995 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sriram Rao >Assignee: Konstantinos Karanasos >Priority: Blocker > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-2995.001.patch, YARN-2995.002.patch, > YARN-2995.003.patch, YARN-2995.004.patch, all-nodes.png, all-nodes.png, > opp-container.png > > > This JIRA proposes to extend the Resource manager UI to show how cluster > resources are being used to run *guaranteed start* and *queueable* > containers. For example, a graph that shows over time, the fraction of > running containers that are *guaranteed start* and the fraction of running > containers that are *queueable*. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5799) Fix Opportunistic Allocation to set the correct value of Node Http Address
[ https://issues.apache.org/jira/browse/YARN-5799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15786339#comment-15786339 ] Arun Suresh commented on YARN-5799: --- Committing this to branch-2 > Fix Opportunistic Allocation to set the correct value of Node Http Address > --- > > Key: YARN-5799 > URL: https://issues.apache.org/jira/browse/YARN-5799 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5799.001.patch, YARN-5799.002.patch > > > This proposes to fix the the OpportunisticAllocator, used to allocate > OPPORTUNISTIC containers (both Centrally as well as in a Distributed manner) > to correctly populate the Node Http Address in the returned Container. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-2995) Enhance UI to show cluster resource utilization of various container Execution types
[ https://issues.apache.org/jira/browse/YARN-2995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-2995: -- Fix Version/s: 2.9.0 > Enhance UI to show cluster resource utilization of various container > Execution types > > > Key: YARN-2995 > URL: https://issues.apache.org/jira/browse/YARN-2995 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sriram Rao >Assignee: Konstantinos Karanasos >Priority: Blocker > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-2995.001.patch, YARN-2995.002.patch, > YARN-2995.003.patch, YARN-2995.004.patch, all-nodes.png, all-nodes.png, > opp-container.png > > > This JIRA proposes to extend the Resource manager UI to show how cluster > resources are being used to run *guaranteed start* and *queueable* > containers. For example, a graph that shows over time, the fraction of > running containers that are *guaranteed start* and the fraction of running > containers that are *queueable*. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6040) Remove usage of ResourceRequest from AppSchedulerInfo / SchedulerApplicationAttempt
[
https://issues.apache.org/jira/browse/YARN-6040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15798997#comment-15798997
]
Arun Suresh commented on YARN-6040:
---
Thanks for the patch [~leftnoteasy],
Couple of comments:
# Maybe rename PendingAsk::allocationNumber to PendingAsk::count / cardinality ?
# From your comment, it seems you are planning on moving canDelayTo() method to
a separate policy ? I was thinking the PlacementSet itself would encapsulate
the delay scheduling policy, but we can discuss that later.
# In the FSPreemptionThread::run() method, the potentialNodes (line 104 in the
patch), you should be doing a getNodesByResourceName() on the rack name. I
understand though that it requires adding the resourcename into the PendingAsk.
> Remove usage of ResourceRequest from AppSchedulerInfo /
> SchedulerApplicationAttempt
> ---
>
> Key: YARN-6040
> URL: https://issues.apache.org/jira/browse/YARN-6040
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-6040.001.patch, YARN-6040.002.patch,
> YARN-6040.003.patch, YARN-6040.004.patch
>
>
> As mentioned by YARN-5906, currently schedulers are using ResourceRequest
> heavily so it will be very hard to adopt the new PowerfulResourceRequest
> (YARN-4902).
> This JIRA is the 2nd step of refactoring, which remove usage of
> ResourceRequest from AppSchedulingInfo / SchedulerApplicationAttempt. Instead
> of returning ResourceRequest, it returns a lightweight and API-independent
> object - {{PendingAsk}}.
> The only remained ResourceRequest API of AppSchedulingInfo will be used by
> web service to get list of ResourceRequests.
> So after this patch, usage of ResourceRequest will be isolated inside
> AppSchedulingInfo, so it will be more flexible to update internal data
> structure and upgrade old ResourceRequest API to new.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6066) Opportunistic containers minor fixes: API annotations and config parameter changes
[ https://issues.apache.org/jira/browse/YARN-6066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15809617#comment-15809617 ] Arun Suresh commented on YARN-6066: --- Committing this to branch-2 shortly since these changes have already been reviewed in YARN-6041 > Opportunistic containers minor fixes: API annotations and config parameter > changes > -- > > Key: YARN-6066 > URL: https://issues.apache.org/jira/browse/YARN-6066 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh >Priority: Minor > Attachments: YARN-6066-branch-2.001.patch > > > Creating this to capture changes suggested by [~leftnoteasy] and [~kasha] in > YARN-6041 in its own JIRA. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6040) Remove usage of ResourceRequest from AppSchedulerInfo / SchedulerApplicationAttempt
[
https://issues.apache.org/jira/browse/YARN-6040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15803541#comment-15803541
]
Arun Suresh commented on YARN-6040:
---
Thanks for the updated patch Wangda,
The latest patch looks good to me.
Will commit it tomorrow if no objections.
BTW, Some of the checkstyles look fixable.. will take care of it when I commit.
> Remove usage of ResourceRequest from AppSchedulerInfo /
> SchedulerApplicationAttempt
> ---
>
> Key: YARN-6040
> URL: https://issues.apache.org/jira/browse/YARN-6040
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-6040.001.patch, YARN-6040.002.patch,
> YARN-6040.003.patch, YARN-6040.004.patch, YARN-6040.005.patch,
> YARN-6040.006.patch, YARN-6040.007.patch
>
>
> As mentioned by YARN-5906, currently schedulers are using ResourceRequest
> heavily so it will be very hard to adopt the new PowerfulResourceRequest
> (YARN-4902).
> This JIRA is the 2nd step of refactoring, which remove usage of
> ResourceRequest from AppSchedulingInfo / SchedulerApplicationAttempt. Instead
> of returning ResourceRequest, it returns a lightweight and API-independent
> object - {{PendingAsk}}.
> The only remained ResourceRequest API of AppSchedulingInfo will be used by
> web service to get list of ResourceRequests.
> So after this patch, usage of ResourceRequest will be isolated inside
> AppSchedulingInfo, so it will be more flexible to update internal data
> structure and upgrade old ResourceRequest API to new.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5938) Minor refactoring to OpportunisticContainerAllocatorAMService
[ https://issues.apache.org/jira/browse/YARN-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15762880#comment-15762880 ] Arun Suresh commented on YARN-5938: --- Thanks for the review [~leftnoteasy] bq. OpportunisticContainerAllocator#allocateContainers, why change from AllocateRequest to several separated param? Is it possible that we need some other fields in AllocateRequest? So, Firslty, I wanted to decouple the OpportunisticContainerAllocator from the AllocateRequest object and Secondly.. we were doing 1. allocateRequest.setAskList() 2. then OpportunisticContainerAllocator::allocate() 3. then doing a allocateRequest.setAskList() 4. super.allocate() The above sequence seemed a bit "hacky"... and made a code flow a bit difficult to follow.. the refactoring fixes that. It is possible we might need other things later from the AllocateRequest, but we can address that as and when we need to (there are better ways like updating the OpportunisticContainerContext etc.) Will update the patch addressing your remaining comments shortly.. > Minor refactoring to OpportunisticContainerAllocatorAMService > - > > Key: YARN-5938 > URL: https://issues.apache.org/jira/browse/YARN-5938 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5938-YARN-5085.001.patch, > YARN-5938-YARN-5085.002.patch, YARN-5938-YARN-5085.003.patch, > YARN-5938.001.patch > > > Minor code re-organization to do the following: > # The OpportunisticContainerAllocatorAMService currently allocates outside > the ApplicationAttempt lock maintained by the ApplicationMasterService. This > should happen inside the lock. > # Refactored out some code to simplify the allocate() method. > # Removed some unused fields inside the OpportunisticContainerAllocator. > # Re-organized some of the code in the > OpportunisticContainerAllocatorAMService::allocate method to make it a bit > more readable. > # Moved SchedulerRequestKey to a new package, so it can be used by the > OpportunisticContainerAllocator/Context. > # Moved all usages of Priority in the OpportunisticContainerAllocator -> > SchedulerRequestKey. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5938) Minor refactoring to OpportunisticContainerAllocatorAMService
[
https://issues.apache.org/jira/browse/YARN-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-5938:
--
Attachment: YARN-5938-YARN-5085.004.patch
Updating patch based on [~leftnoteasy]'s suggestions.
bq. ApplicationMasterService#allocate -> internalAllocate
I renamed it to {{allocateInternal()}} to maintain consistency with other
places such as ContainerManager#startContainerInternal()
> Minor refactoring to OpportunisticContainerAllocatorAMService
> -
>
> Key: YARN-5938
> URL: https://issues.apache.org/jira/browse/YARN-5938
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-5938-YARN-5085.001.patch,
> YARN-5938-YARN-5085.002.patch, YARN-5938-YARN-5085.003.patch,
> YARN-5938-YARN-5085.004.patch, YARN-5938.001.patch
>
>
> Minor code re-organization to do the following:
> # The OpportunisticContainerAllocatorAMService currently allocates outside
> the ApplicationAttempt lock maintained by the ApplicationMasterService. This
> should happen inside the lock.
> # Refactored out some code to simplify the allocate() method.
> # Removed some unused fields inside the OpportunisticContainerAllocator.
> # Re-organized some of the code in the
> OpportunisticContainerAllocatorAMService::allocate method to make it a bit
> more readable.
> # Moved SchedulerRequestKey to a new package, so it can be used by the
> OpportunisticContainerAllocator/Context.
> # Moved all usages of Priority in the OpportunisticContainerAllocator ->
> SchedulerRequestKey.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5938) Refactoring OpportunisticContainerAllocator use SchedulerRequestKey instead of Priority
[ https://issues.apache.org/jira/browse/YARN-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765095#comment-15765095 ] Arun Suresh commented on YARN-5938: --- Committed to branch YARN-5085 Thanks for the review [~leftnoteasy] > Refactoring OpportunisticContainerAllocator use SchedulerRequestKey instead > of Priority > --- > > Key: YARN-5938 > URL: https://issues.apache.org/jira/browse/YARN-5938 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5938-YARN-5085.001.patch, > YARN-5938-YARN-5085.002.patch, YARN-5938-YARN-5085.003.patch, > YARN-5938-YARN-5085.004.patch, YARN-5938.001.patch > > > Minor code re-organization to do the following: > # The OpportunisticContainerAllocatorAMService currently allocates outside > the ApplicationAttempt lock maintained by the ApplicationMasterService. This > should happen inside the lock. > # Refactored out some code to simplify the allocate() method. > # Removed some unused fields inside the OpportunisticContainerAllocator. > # Re-organized some of the code in the > OpportunisticContainerAllocatorAMService::allocate method to make it a bit > more readable. > # Moved SchedulerRequestKey to a new package, so it can be used by the > OpportunisticContainerAllocator/Context. > # Moved all usages of Priority in the OpportunisticContainerAllocator -> > SchedulerRequestKey. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5938) Refactoring OpportunisticContainerAllocator use SchedulerRequestKey instead of Priority
[ https://issues.apache.org/jira/browse/YARN-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5938: -- Summary: Refactoring OpportunisticContainerAllocator use SchedulerRequestKey instead of Priority (was: Minor refactoring to OpportunisticContainerAllocatorAMService) > Refactoring OpportunisticContainerAllocator use SchedulerRequestKey instead > of Priority > --- > > Key: YARN-5938 > URL: https://issues.apache.org/jira/browse/YARN-5938 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5938-YARN-5085.001.patch, > YARN-5938-YARN-5085.002.patch, YARN-5938-YARN-5085.003.patch, > YARN-5938-YARN-5085.004.patch, YARN-5938.001.patch > > > Minor code re-organization to do the following: > # The OpportunisticContainerAllocatorAMService currently allocates outside > the ApplicationAttempt lock maintained by the ApplicationMasterService. This > should happen inside the lock. > # Refactored out some code to simplify the allocate() method. > # Removed some unused fields inside the OpportunisticContainerAllocator. > # Re-organized some of the code in the > OpportunisticContainerAllocatorAMService::allocate method to make it a bit > more readable. > # Moved SchedulerRequestKey to a new package, so it can be used by the > OpportunisticContainerAllocator/Context. > # Moved all usages of Priority in the OpportunisticContainerAllocator -> > SchedulerRequestKey. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5959) RM changes to support change of container ExecutionType
[ https://issues.apache.org/jira/browse/YARN-5959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15765229#comment-15765229 ] Arun Suresh commented on YARN-5959: --- Based on offline discussions with [~leftnoteasy], I will rework this patch to refactor the internal RM implementation of container increase-decrease based on the approach outlined here to unify the codebase. > RM changes to support change of container ExecutionType > --- > > Key: YARN-5959 > URL: https://issues.apache.org/jira/browse/YARN-5959 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5959.combined.001.patch, YARN-5959.wip.002.patch, > YARN-5959.wip.003.patch, YARN-5959.wip.patch > > > RM side changes to allow an AM to ask for change of ExecutionType. > Currently, there are two cases: > # *Promotion* : OPPORTUNISTIC to GUARANTEED. > # *Demotion* : GUARANTEED to OPPORTUNISTIC. > This is similar in YARN-1197 which allows for change in Container resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5216) Expose configurable preemption policy for OPPORTUNISTIC containers running on the NM
[ https://issues.apache.org/jira/browse/YARN-5216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15763587#comment-15763587 ] Arun Suresh commented on YARN-5216: --- Thanks for updating the patch [~hrsharma] Couple of comments: # In the ContainerScheduler, rename onContainerCompleted() method to onResourcesReclaimed(). Think that might make more sense now. # Instead of explicitly calling "dispatcher.getEventHandler().handle(..)" from within ContainerScheduler, can you create a method inside Container: sendPauseEvent(String) and sendResumeEvent(String) # Actually w.r.t the above comment, consider just updating the sendLaunchEvent() method that is already present to check the container state, and if paused, send the pause event else send the launch event to the launcher. > Expose configurable preemption policy for OPPORTUNISTIC containers running on > the NM > > > Key: YARN-5216 > URL: https://issues.apache.org/jira/browse/YARN-5216 > Project: Hadoop YARN > Issue Type: Sub-task > Components: distributed-scheduling >Reporter: Arun Suresh >Assignee: Hitesh Sharma > Labels: oct16-hard > Attachments: YARN-5216-YARN-5972.001.patch, > YARN-5216-YARN-5972.002.patch, YARN-5216-YARN-5972.003.patch, > YARN-5216-YARN-5972.004.patch, YARN5216.001.patch, yarn5216.002.patch > > > Currently, the default action taken by the QueuingContainerManager, > introduced in YARN-2883, when a GUARANTEED Container is scheduled on an NM > with OPPORTUNISTIC containers using up resources, is to KILL the running > OPPORTUNISTIC containers. > This JIRA proposes to expose a configurable hook to allow the NM to take a > different action. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5938) Refactoring OpportunisticContainerAllocator use SchedulerRequestKey instead of Priority
[ https://issues.apache.org/jira/browse/YARN-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5938: -- Attachment: YARN-5938.003.patch Fixing some of the checkstyle warnings. The testcase failure is not releated.. > Refactoring OpportunisticContainerAllocator use SchedulerRequestKey instead > of Priority > --- > > Key: YARN-5938 > URL: https://issues.apache.org/jira/browse/YARN-5938 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 3.0.0-beta1 > > Attachments: YARN-5938-YARN-5085.001.patch, > YARN-5938-YARN-5085.002.patch, YARN-5938-YARN-5085.003.patch, > YARN-5938-YARN-5085.004.patch, YARN-5938.001.patch, YARN-5938.002.patch, > YARN-5938.003.patch > > > Minor code re-organization to do the following: > # The OpportunisticContainerAllocatorAMService currently allocates outside > the ApplicationAttempt lock maintained by the ApplicationMasterService. This > should happen inside the lock. > # Refactored out some code to simplify the allocate() method. > # Removed some unused fields inside the OpportunisticContainerAllocator. > # Re-organized some of the code in the > OpportunisticContainerAllocatorAMService::allocate method to make it a bit > more readable. > # Moved SchedulerRequestKey to a new package, so it can be used by the > OpportunisticContainerAllocator/Context. > # Moved all usages of Priority in the OpportunisticContainerAllocator -> > SchedulerRequestKey. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5938) Refactoring OpportunisticContainerAllocator use SchedulerRequestKey instead of Priority
[ https://issues.apache.org/jira/browse/YARN-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5938: -- Attachment: (was: YARN-5938.002.patch.) > Refactoring OpportunisticContainerAllocator use SchedulerRequestKey instead > of Priority > --- > > Key: YARN-5938 > URL: https://issues.apache.org/jira/browse/YARN-5938 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 3.0.0-beta1 > > Attachments: YARN-5938-YARN-5085.001.patch, > YARN-5938-YARN-5085.002.patch, YARN-5938-YARN-5085.003.patch, > YARN-5938-YARN-5085.004.patch, YARN-5938.001.patch, YARN-5938.002.patch > > > Minor code re-organization to do the following: > # The OpportunisticContainerAllocatorAMService currently allocates outside > the ApplicationAttempt lock maintained by the ApplicationMasterService. This > should happen inside the lock. > # Refactored out some code to simplify the allocate() method. > # Removed some unused fields inside the OpportunisticContainerAllocator. > # Re-organized some of the code in the > OpportunisticContainerAllocatorAMService::allocate method to make it a bit > more readable. > # Moved SchedulerRequestKey to a new package, so it can be used by the > OpportunisticContainerAllocator/Context. > # Moved all usages of Priority in the OpportunisticContainerAllocator -> > SchedulerRequestKey. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5938) Refactoring OpportunisticContainerAllocator use SchedulerRequestKey instead of Priority
[ https://issues.apache.org/jira/browse/YARN-5938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-5938: -- Attachment: YARN-5938.002.patch > Refactoring OpportunisticContainerAllocator use SchedulerRequestKey instead > of Priority > --- > > Key: YARN-5938 > URL: https://issues.apache.org/jira/browse/YARN-5938 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 3.0.0-beta1 > > Attachments: YARN-5938-YARN-5085.001.patch, > YARN-5938-YARN-5085.002.patch, YARN-5938-YARN-5085.003.patch, > YARN-5938-YARN-5085.004.patch, YARN-5938.001.patch, YARN-5938.002.patch > > > Minor code re-organization to do the following: > # The OpportunisticContainerAllocatorAMService currently allocates outside > the ApplicationAttempt lock maintained by the ApplicationMasterService. This > should happen inside the lock. > # Refactored out some code to simplify the allocate() method. > # Removed some unused fields inside the OpportunisticContainerAllocator. > # Re-organized some of the code in the > OpportunisticContainerAllocatorAMService::allocate method to make it a bit > more readable. > # Moved SchedulerRequestKey to a new package, so it can be used by the > OpportunisticContainerAllocator/Context. > # Moved all usages of Priority in the OpportunisticContainerAllocator -> > SchedulerRequestKey. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
