[
https://issues.apache.org/jira/browse/TEZ-1157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011708#comment-14011708
]
Rajesh Balamohan commented on TEZ-1157:
---------------------------------------
Thanks for sharing the inputs BIkas. Yes, currently unsorted-kv-input reads
all the data from broadcast edge. There is no support to download data in
chunks (i.e by specifying range within the data that needs to be downloaded).
Separate ShuffleHandler is needed in order to support this.
Some options to reduce number of copies of same data being downloaded in the
same node:
1. Download data locally and let the local NM's ShuffleHandler serve the output
to its local tasks.
- There can be associated issues in terms of handling partially
downloaded data. With custom ShuffleHandler, it might be easier handle this
case where in, other tasks can check if the data-range to be downloaded is
present in the local system.
2. Let the tasks download the data and write it to local directories. Task
needs to lock the file for writing and it can be written to
$NM_LOCAL_DIR_OF_APP/application_id/src_attempt_id . Tasks running in the same
container (belonging to same job) can check whether the file is locally present
or not; If it is not present those tasks can fetch the data via remote read
(i.e Http). Locally downloaded data would be cleared up as a part of job
cleanup acitivity. To prevent partial data from being read by other tasks, a
zero byte indicator file can be created when the download is complete.
> Optimize broadcast :- Tasks pertaining to same job in same machine should not
> download multiple copies of broadcast data
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: TEZ-1157
> URL: https://issues.apache.org/jira/browse/TEZ-1157
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Labels: performance
> Attachments: TEZ-1152.WIP.patch
>
>
> Currently tasks (belonging to same job) running in the same machine download
> its own copy of broadcast data. Optimization could be to download one copy
> in the machine, and the rest of the tasks can refer to this downloaded copy.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
