[
https://issues.apache.org/jira/browse/SAMZA-886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jake Maes updated SAMZA-886:
----------------------------
Attachment: RelaxedLocality experiments.pdf
tl;dr
1. This was mostly an LI internal problem. We need to correct the
net.toplogy.script.file.name path in core-site.xml for NMs in some clusters.
2. We found and fixed SAMZA-922, but no other code changes are needed
3. The attached experiment results show that with the above fixes, host
affinity performs equally well with or without relaxed locality, rack-locality,
or request prioritization.
Details:
After significant investigation, I found that the main problem was a
misconfigured location for the yarn-topology (rack locality) script in some LI
clusters.
Without the script, Yarn defaults everything to /default-rack. The script is
enabled by configuring the following property in Yarn:
{noformat}
<property>
<name>net.topology.script.file.name</name>
<value>/path/to/yarn-topology.py</value>
</property>
{noformat}
The problem was that it was assumed that the script would only be needed in the
RMs, so the path provided was specific to RMs. However, the script is also used
in the NMs to resolve the rack when the AM manages ContainerRequests and since
those were configured with the RM path, the following exception would occur:
{noformat}
java.io.IOException: Cannot run program "/wrong/path/to/yarn-topology.py" (in
directory
"/app_path/application_1452292535523_0047/container_1452292535523_0047_02_000001"):
error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1042)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:485)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at
org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:251)
at
org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:188)
at
org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
at
org.apache.hadoop.yarn.util.RackResolver.coreResolve(RackResolver.java:101)
at org.apache.hadoop.yarn.util.RackResolver.resolve(RackResolver.java:95)
at
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.resolveRacks(AMRMClientImpl.java:551)
at
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.addContainerRequest(AMRMClientImpl.java:411)
at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.addContainerRequest(AMRMClientAsyncImpl.java:166)
at
org.apache.samza.job.yarn.ContainerRequestState.updateRequestState(ContainerRequestState.java:82)
at
org.apache.samza.job.yarn.AbstractContainerAllocator.requestContainer(AbstractContainerAllocator.java:102)
at
org.apache.samza.job.yarn.AbstractContainerAllocator.requestContainers(AbstractContainerAllocator.java:85)
at
org.apache.samza.job.yarn.SamzaTaskManager.onInit(SamzaTaskManager.java:112)
at
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$run$1.apply(SamzaAppMaster.scala:117)
at
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$run$1.apply(SamzaAppMaster.scala:117)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.samza.job.yarn.SamzaAppMaster$.run(SamzaAppMaster.scala:117)
at org.apache.samza.job.yarn.SamzaAppMaster$.main(SamzaAppMaster.scala:104)
at org.apache.samza.job.yarn.SamzaAppMaster.main(SamzaAppMaster.scala)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:187)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1023)
{noformat}
That exception causes the RackResolver to revert to default rack
{noformat}
2016-03-30 19:14:15.300 [main] RackResolver [INFO] Couldn't resolve
some_valid_hostname.linkedin.com. Falling back to /default-rack
{noformat}
Unfortunately, this causes Yarn to not return a container on the desired host
(no container at all, with strict locality). Why? I think
https://issues.apache.org/jira/browse/YARN-752 provides some clues. It seems
that internally Yarn requires the host and rack to match, and the RackResolver
is intended to ensure this, but apparently if the node is on a specific rack
and the resolver picks default rack, it doesn't work.
Note that this issue has only been seen when we configure
net.toplogy.script.file.name which enables specific racks. Without it, there is
no script to fail, but even RackResolver did fail, all the nodes would be on
default rack so when the RackResolver falls back to default rack, it still
works.
To prove this theory we fixed the path issue on a couple cluster and ran some
experiments. The results are attached. The experiments each had 4 trials and
varied in 3 dimensions:
1. Relaxed locality - true/false
2. Code patches - none/container requests for ANY_HOST are lower priority
(queues)
3. Cluster rack script - has(LI)/doesn't have(OSS)
You can see that with the path fix, host affinity performs rather well. There
were a couple occurrences of a connection error in the script:
{noformat}
ExitCodeException exitCode=1: Traceback (most recent call last):
File "/path/to/yarn-topology.py", line 189, in <module>
output_rack_locality_for_yarn(sys.argv[1:])
File "/path/to/yarn-topology.py", line 176, in output_rack_locality_for_yarn
locality_infos = gather_host_list_locality(host_list)
File "/export/content/lid/apps/samsa-yarn-nm/i001/sbin/yarn-topology.py",
line 64, in gather_host_list_locality
devices = jsonload(urlopen(url).read())
File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib64/python2.6/urllib2.py", line 391, in open
response = self._open(req, data)
File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
'_open', req)
File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
result = func(*args)
File "/usr/lib64/python2.6/urllib2.py", line 1198, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "/usr/lib64/python2.6/urllib2.py", line 1165, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno 104] Connection reset by peer>
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at
org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.runResolveCommand(ScriptBasedMapping.java:251)
at
org.apache.hadoop.net.ScriptBasedMapping$RawScriptBasedMapping.resolve(ScriptBasedMapping.java:188)
at
org.apache.hadoop.net.CachedDNSToSwitchMapping.resolve(CachedDNSToSwitchMapping.java:119)
at
org.apache.hadoop.yarn.util.RackResolver.coreResolve(RackResolver.java:101)
at org.apache.hadoop.yarn.util.RackResolver.resolve(RackResolver.java:95)
at
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.resolveRacks(AMRMClientImpl.java:551)
at
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.addContainerRequest(AMRMClientImpl.java:411)
at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl.addContainerRequest(AMRMClientAsyncImpl.java:166)
at
org.apache.samza.job.yarn.ContainerRequestState.updateRequestState(ContainerRequestState.java:82)
at
org.apache.samza.job.yarn.AbstractContainerAllocator.requestContainer(AbstractContainerAllocator.java:102)
at
org.apache.samza.job.yarn.AbstractContainerAllocator.requestContainers(AbstractContainerAllocator.java:85)
at
org.apache.samza.job.yarn.SamzaTaskManager.onInit(SamzaTaskManager.java:112)
at
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$run$1.apply(SamzaAppMaster.scala:117)
at
org.apache.samza.job.yarn.SamzaAppMaster$$anonfun$run$1.apply(SamzaAppMaster.scala:117)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.samza.job.yarn.SamzaAppMaster$.run(SamzaAppMaster.scala:117)
at org.apache.samza.job.yarn.SamzaAppMaster$.main(SamzaAppMaster.scala:104)
at org.apache.samza.job.yarn.SamzaAppMaster.main(SamzaAppMaster.scala)
2016-03-30 19:14:15.300 [main] RackResolver [INFO] Couldn't resolve
some_valid_hostname.linkedin.com. Falling back to /default-rack
{noformat}
And those had the same effect as the path issue. The container was not
allocated on the preferred host. To fix this, we could just add some retries to
the script.
Otherwise, the results show that there's not much value in making any code
changes for now.
The tests were also run on a cluster with no topology script, which is the
default Yarn configuration, to ensure that adopters who don't configure yarn
for rack locality observe proper host affinity behavior. The feature performed
as expected.
Closing this ticket.
> Investigate 'relax locality' to improve Host Affinity
> ------------------------------------------------------
>
> Key: SAMZA-886
> URL: https://issues.apache.org/jira/browse/SAMZA-886
> Project: Samza
> Issue Type: Bug
> Reporter: Jagadish
> Assignee: Jake Maes
> Attachments: RelaxedLocality experiments.pdf
>
>
> I ran several tests experimenting Samza with a cluster of size 36 nodes. I
> have the following observations:
> 1.On a cluster with about 50% utilization. The percentage of requests that
> are mapped to preferred hosts seems to depend on yarn.container.count. The %
> is higher when yarn.container.count is comparable to the size of the cluster.
> (For example.) I get about 50% of requests matched when yarn.container.count
> is 30. and When yarn.container.count is 10, only 27% of requests are matched.
> (on a 36 node cluster)
> One reason is because, when spawning a large # of containers initially, many
> requests are made in bulk successively, there is a good chance that any
> random host in the cluster will match with the preferred request. However,
> when spawning a particular container during failure, there's only one request
> for the failed container, and it has a lesser chance of a match.
> The results are averaged across 20 runs in each scenario.
> 2. On a cluster with about zero utilization, 100% of requests are matched to
> preferred hosts irrespective of yarn.container.count.
> This ticket is to explore alternatives to see if they will improve % of
> matched hosts.
> I believe these ideas are worth trying:
> 1. Yarn supports the idea of a 'relaxed locality' flag that can be specified
> with the request. We could set 'relaxed locality' to false. (This will ensure
> that we get the request on the exact same host we ask for.) If we don't get
> such a request within a timeout, we may re-request the same request with
> 'relaxed locality' to true. (as we currently do now.)
> 2. Re-issue the same preferred host request again, if the hosts returned
> don't match the request.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
