[
https://issues.apache.org/jira/browse/TEZ-2442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14544413#comment-14544413
]
Siddharth Seth commented on TEZ-2442:
-------------------------------------
We've made similar changes in the past - primarily as hacks - to quickly test
out Shuffle via HDFS. It'll be good to get the changes in with a proper patch.
I glanced over the 5 commits. To Bikas' point, there's no need to create
separate LogicalInputs / Outputs - this is configuration for the existing ones,
which is what it should be.
Couple of things to consider while making these changes
- On both the producer and consumer end - there may be intermediate data being
written out to disk. For the producer, this may be intermediate spills, which
don't need to be transferred to the reducer and instead will be merged locally.
For the consumer - there's data which is fetched, but there may be intermediate
spills while this data is being merged together. I don't think the intermediate
spills need to go through HDFS.
- Security is another aspect to consider. Don't think this will be a big
consideration though. User level security will get managed with tokens for the
filesystem. Across jobs (within the same user) - even today the files are on
the local file system and can be accessed by other jobs running as the same
user (assuming they know the path).
- From looking at the patch, it's modifying the code path which is used for the
Local Disk Fetch optimization. That's very avoidable. This data will always be
read directly from source - which means we'll attempt to read the data over the
network while serving it out or merging it. The main code path fetches the data
to memory (or local disk) in a single chunk, and then performs the merge on it.
That should be how this works as well - where remote data from HDFS is fetched
to the local node.
- TEZ-2450 is making the Fetch component pluggable. That can be extended to be
a data source, one of which can be a DFS based implementation.
> Support DFS based shuffle in addition to HTTP shuffle
> -----------------------------------------------------
>
> Key: TEZ-2442
> URL: https://issues.apache.org/jira/browse/TEZ-2442
> Project: Apache Tez
> Issue Type: Improvement
> Affects Versions: 0.5.3
> Reporter: Kannan Rajah
> Attachments: Tez Shuffle using DFS.pdf
>
>
> In Tez, Shuffle is a mechanism by which intermediate data can be shared
> between stages. Shuffle data is written to local disk and fetched from any
> remote node using HTTP. A DFS like MapR file system can support writing this
> shuffle data directly to its DFS using a notion of local volumes and retrieve
> it using HDFS API from remote node. The current Shuffle implementation
> assumes local data can only be managed by LocalFileSystem. So it uses
> RawLocalFileSystem and LocalDirAllocator. If we can remove this assumption
> and introduce an abstraction to manage local disks, then we can reuse most of
> the shuffle logic (store, sort) and inject a HDFS API based retrieval instead
> of HTTP.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
