[
https://issues.apache.org/jira/browse/MAPREDUCE-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12872372#action_12872372
]
Dmytro Molkov commented on MAPREDUCE-1752:
------------------------------------------
Nicholas, I do see that this approach is somewhat expensive. However it gives
us the locality when we are running a job.
And this time will only be added to the job setup time, right?
I guess my approach was making it right and then looking at the ways we can
optimize it rather then trying to hack up a fast solution right from the start.
Do you have any other ideas that may be worth exploring?
> Implement getFileBlockLocations in HarFilesystem
> ------------------------------------------------
>
> Key: MAPREDUCE-1752
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1752
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: harchive
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: MR-1752.patch
>
>
> To efficiently run map reduce on the data that has been HAR'ed it will be
> great to actually implement getFileBlockLocations for a given filename.
> This way the JobTracker will have information about data locality and will
> schedule tasks appropriately.
> I believe the overhead introduced by doing lookups in the index files can be
> smaller than that of copying data over the wire.
> Will upload the patch shortly, but would love to get some feedback on this.
> And any ideas on how to test it are very welcome.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
