[
https://issues.apache.org/jira/browse/YARN-1040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15167794#comment-15167794
]
Vinod Kumar Vavilapalli commented on YARN-1040:
-----------------------------------------------
Still catching up on some of your discussion, but quick comments on a few
things that I care
We should really call them instead as {{Allocation}} and {{Container}}. That's
the nomenclature I used at YARN-4726. Till now, YARN has combined the notion of
Allocation and Container, which is the main de-linking that we need to do here.
Process has an OS level connotation, and doesn't work well with more things in
the picture like process-trees / multiple-processes / Docker (YARN-2466).
Taking this further, here's how the overall picture can look like the following
*ResourceManager*
- RM only does allocations in the scheduling path. ResourceManager does all
scheduling based on AllocationRequests and tracks Allocations..
- RM receives AllocationRequests and returns fulfilled Allocations (and
AllocationTokens) to AMs.
*Applications*
- AM can in turn use the Allocations (and AllocationTokens) to launch multiple
Containers on the NM.
-- Simple case: AM only launches containers one-after-another. It's up to
the app to do this.
-- General case: AM launches multiple containers at the same time. This is
essentially container-groups - we should keep this option open.
- AMs can specify *single-use* AllocationRequests, at which point RM can
simply return Containers and Container-Tokens (today's code-path).
- Each Container exits when the process-tree / linux-container exits.
- Each Container has an Identifier.
-- For single-use allocation-requests, RM generates ContainerIDs
-- For multi-use allocation-requests, apps could optionally specify a
container-name that is scoped under the allocation. NM always returns a
(generated or app-specified) ContainerID based off the allocation-ID.
Essentially, allocationID + containerID is unique
*NodeManagers*
- NodeManager also understands incoming Allocations and ties them to Container
groups: it deals with Allocation activation/deactivation and Container
start/stop. but does
-- the following *decoupled from both allocations and containers*:
localizations / re-localizations. This means local-resources should now have
more scopes: container, allocation, application etc.
-- *per allocation*: enforcement of resource-limits
-- all of the following *per container*: (a) process/OS-container
activation / deactivation, (b) process/OS-container auto-restart (YARN-4725)
log-aggregation
> De-link container life cycle from the process and add ability to execute
> multiple processes in the same long-lived container
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-1040
> URL: https://issues.apache.org/jira/browse/YARN-1040
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
> Affects Versions: 3.0.0
> Reporter: Steve Loughran
>
> The AM should be able to exec >1 process in a container, rather than have the
> NM automatically release the container when the single process exits.
> This would let an AM restart a process on the same container repeatedly,
> which for HBase would offer locality on a restarted region server.
> We may also want the ability to exec multiple processes in parallel, so that
> something could be run in the container while a long-lived process was
> already running. This can be useful in monitoring and reconfiguring the
> long-lived process, as well as shutting it down.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
