----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25242/#review53457 -----------------------------------------------------------
Hi Mai, I've investigated the problem, incurring FileNotFound. I found the cause. ExternalSort directly merges input data, and than it removes its intermediate data of previous run when it finishes merge all files at some run level. In this approach, ExternalSortExec also removes the input files, actually output files of previous stage. But, there are remain tasks, which still requires the output files. Because I should go out right now, I'll leave more comments after a while. - Hyunsik Choi On Sept. 4, 2014, 5:40 p.m., Hai Thanh Mai wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/25242/ > ----------------------------------------------------------- > > (Updated Sept. 4, 2014, 5:40 p.m.) > > > Review request for Tajo and Hyunsik Choi. > > > Repository: tajo > > > Description > ------- > > TAJO-983: Worker should directly read Intermediate data stored in localhost > rather than fetching > > Currently, worker always fetches all intermediate via Fetcher and than store > them in local file system even though some intermediate data already are > stored in local file system. It is inefficient and causes unnecessary I/O and > extra storage occupation. We should improve it. > > https://issues.apache.org/jira/browse/TAJO-983 > > > Diffs > ----- > > tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java 64475fe > tajo-core/src/main/java/org/apache/tajo/worker/Task.java d0665ae > tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java 95c06bb > > tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java > 150ac85 > > tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java > a8b424e > > Diff: https://reviews.apache.org/r/25242/diff/ > > > Testing > ------- > > > Thanks, > > Hai Thanh Mai > >
