Gourav Khaneja created TWILL-87:
-----------------------------------
Summary: Adding Container Placement control (Placement Policy API)
Key: TWILL-87
URL: https://issues.apache.org/jira/browse/TWILL-87
Project: Apache Twill
Issue Type: New Feature
Components: api, yarn
Affects Versions: 0.3.0-incubating
Environment: Tested on Hadoop Yarn 2.2 and 2.3 running on Ubuntu-nodes
(4 GB , 8 Cores) cluster.
Reporter: Gourav Khaneja
Fix For: 0.3.0-incubating
Yarn AMRMClient provides API to control container placement. We need to enhance
Twill API so that user can specify container placement policy. Twill could use
AMRMClient to try allocating containers according to specified placement policy.
1. API:
a. Added Placement Policy API in TwillSpecificationBuilder.
b. For now, the only Placement policy added is DISTRIBUTED, which tries to
spawn specified runnables on different hosts.
c. Added a placeholder for 'PlacementHints' through which user can specify
host names they want the runnables to go to.
2. Code Changes:
a. Allocating Containers for one runnable instance at a time, instead of
grouping them together based on Resource Specs.
b. Added PlacementPolicy and PlacementPolicyGroup data structures
/interface for storing placement policies.
b. Storing node name too (apart from hostname) in TwillRunResource.
3. AMRMClient API Usage
a. Using Blacklist to support DISTRIBUTED placement policy.
b. Not using addContainerRequest.relaxLocality = false for now.
4. Unit Tests
a. Added PlacementPolicyTestRun Class with test method
testDistributedPlacementPolicy.
b. Running MiniYarnCluster with 3 nodes instead of 1.
Please review changes in PR - https://github.com/apache/incubator-twill/pull/7
--
This message was sent by Atlassian JIRA
(v6.2#6252)
