[
https://issues.apache.org/jira/browse/HADOOP-12878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15323576#comment-15323576
]
Chris Nauroth commented on HADOOP-12878:
----------------------------------------
>From reviewing the description, there might be a misunderstanding about the
>implementation in hadoop-azure.
bq. We should mimic Azures implementation: a config setting
fs.s3a.block.location.impersonatedhost where the user can enter the list of
hostnames in the cluster to return to getFileBlockLocations.
The hadoop-azure implementation currently reports the same host for every block
location. That host is configurable and defaults to "localhost". I'm not
aware of a way to get it to use a mix of different hostnames.
What WASB does differently from S3A right now is that it overrides
{{getFileBlockLocations}} to mimic the concept of block size and use that block
size to divide a file and report that it has multiple block locations. For
something like MapReduce, that translates to multiple input splits, more map
tasks and a greater opportunity for I/O parallelism on jobs that consume a
small number of very large files. S3A is different in that it inherits the
{{getFileBlockLocations}} implementation from the superclass, which always
reports that the file has exactly 1 block location. That could mean that for
example, S3A would experience a bottleneck on a job whose input is a single
very large file, because it would get only 1 input split.
If use of the same host name in every block location can cause scheduling
bottlenecks at the ResourceManager, then I have to assume that WASB is prone to
that same problem today. Echoing Steve's comment, if any code changes done
here are self-contained and reusable, then S3A, WASB and any other file system
could call it to get the benefits.
> Impersonate hosts in s3a for better data locality handling
> ----------------------------------------------------------
>
> Key: HADOOP-12878
> URL: https://issues.apache.org/jira/browse/HADOOP-12878
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.8.0
> Reporter: Thomas Demoor
> Assignee: Thomas Demoor
>
> Currently, {{localhost}} is passed as locality for each block, causing all
> blocks involved in job to initially target the same node (RM), before being
> moved by the scheduler (to a rack-local node). This reduces parallelism for
> jobs (with short-lived mappers).
> We should mimic Azures implementation: a config setting
> {{fs.s3a.block.location.impersonatedhost}} where the user can enter the list
> of hostnames in the cluster to return to {{getFileBlockLocations}}.
> Possible optimization: for larger systems, it might be better to return N
> (5?) random hostnames to prevent passing a huge array (the downstream code
> assumes size = O(3)).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
