[
https://issues.apache.org/jira/browse/TAJO-983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14104163#comment-14104163
]
Hyunsik Choi commented on TAJO-983:
-----------------------------------
Hi [~mhthanh],
Your approach looks good to me. BTW, I have some better approach for it. I'm
sorry for not mentioning it earlier.
Please see Task::localizeFetchedData() method. This method makes FileFragments
for input data from fetched files. The FileFragments will be ultimately used
for a sequential scanner or merge scanner. I think that we can directly make
FileFragment with a original local path without unnecessary copying the files
stored in local file system.
How do you think about that?
Best regards,
Hyunsik
> Worker should directly read Intermediate data stored in localhost rather than
> fetching
> --------------------------------------------------------------------------------------
>
> Key: TAJO-983
> URL: https://issues.apache.org/jira/browse/TAJO-983
> Project: Tajo
> Issue Type: Bug
> Components: data shuffle
> Reporter: Hyunsik Choi
> Assignee: Mai Hai Thanh
> Attachments: TAJO-983.140820.0.patch.txt
>
>
> Currently, worker always fetches all intermediate via Fetcher and than store
> them in local file system even though some intermediate data already are
> stored in local file system. It is inefficient and causes unnecessary I/O and
> extra storage occupation. We should improve it.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
