> On Sept. 16, 2014, 11:39 a.m., Hyunsik Choi wrote: > > Hi Mai, > > > > I've investigated the problem, incurring FileNotFound. I found the cause. > > ExternalSort directly merges input data, and than it removes its > > intermediate data of previous run when it finishes merge all files at some > > run level. > > > > In this approach, ExternalSortExec also removes the input files, actually > > output files of previous stage. But, there are remain tasks, which still > > requires the output files. > > > > Because I should go out right now, I'll leave more comments after a while.
In order to solve the problem, ExternalSortExec should not remove intermediate files, which actually are output files of previous stage. For it, we need to add some flag to indicate this information. - Hyunsik ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25242/#review53457 ----------------------------------------------------------- On Sept. 4, 2014, 5:40 p.m., Hai Thanh Mai wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/25242/ > ----------------------------------------------------------- > > (Updated Sept. 4, 2014, 5:40 p.m.) > > > Review request for Tajo and Hyunsik Choi. > > > Repository: tajo > > > Description > ------- > > TAJO-983: Worker should directly read Intermediate data stored in localhost > rather than fetching > > Currently, worker always fetches all intermediate via Fetcher and than store > them in local file system even though some intermediate data already are > stored in local file system. It is inefficient and causes unnecessary I/O and > extra storage occupation. We should improve it. > > https://issues.apache.org/jira/browse/TAJO-983 > > > Diffs > ----- > > tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java 64475fe > tajo-core/src/main/java/org/apache/tajo/worker/Task.java d0665ae > tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java 95c06bb > > tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java > 150ac85 > > tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java > a8b424e > > Diff: https://reviews.apache.org/r/25242/diff/ > > > Testing > ------- > > > Thanks, > > Hai Thanh Mai > >
