> On Sept. 16, 2014, 11:39 a.m., Hyunsik Choi wrote: > > Hi Mai, > > > > I've investigated the problem, incurring FileNotFound. I found the cause. > > ExternalSort directly merges input data, and than it removes its > > intermediate data of previous run when it finishes merge all files at some > > run level. > > > > In this approach, ExternalSortExec also removes the input files, actually > > output files of previous stage. But, there are remain tasks, which still > > requires the output files. > > > > Because I should go out right now, I'll leave more comments after a while. > > Hyunsik Choi wrote: > In order to solve the problem, ExternalSortExec should not remove > intermediate files, which actually are output files of previous stage. For > it, we need to add some flag to indicate this information. > > Hai Thanh Mai wrote: > Thank Hyunsik for helpful hints! > I will try to fix this problem. Beside, I guess that adding a flag is > enough to avoid the exception, but not enough to get the correct query > result. The reason is that, in the old pull model, every data chunk will be > fetched into an individual file, so the sort merger will merge the whole > input files. In our new situation, we need to support the scanning of both > whole files and file parts.
You are right. ExternalSortExec is not designed to read only some part of each file. However, RawFile, intermediate file format, is seekable by offsets. We can make ExternalSortExec to scan only some parts instead of reading entire file with FileFragment. - Hyunsik ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25242/#review53457 ----------------------------------------------------------- On Sept. 4, 2014, 5:40 p.m., Hai Thanh Mai wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/25242/ > ----------------------------------------------------------- > > (Updated Sept. 4, 2014, 5:40 p.m.) > > > Review request for Tajo and Hyunsik Choi. > > > Repository: tajo > > > Description > ------- > > TAJO-983: Worker should directly read Intermediate data stored in localhost > rather than fetching > > Currently, worker always fetches all intermediate via Fetcher and than store > them in local file system even though some intermediate data already are > stored in local file system. It is inefficient and causes unnecessary I/O and > extra storage occupation. We should improve it. > > https://issues.apache.org/jira/browse/TAJO-983 > > > Diffs > ----- > > tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java 64475fe > tajo-core/src/main/java/org/apache/tajo/worker/Task.java d0665ae > tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java 95c06bb > > tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java > 150ac85 > > tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java > a8b424e > > Diff: https://reviews.apache.org/r/25242/diff/ > > > Testing > ------- > > > Thanks, > > Hai Thanh Mai > >
