----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25242/#review54015 -----------------------------------------------------------
tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java <https://reviews.apache.org/r/25242/#comment93880> In my view, this condition is not too explicit. It is hard for other code reader to understand the code. In order to fully understand this if-condition, readers need to know lots of knowledges. Especially, code readers should know the fact that all local files whose start offset is 0 and the the end length is equivalent to file length can be removed because we can ensure they are are not shared with other tasks. I'd like to suggest three solutions: * One is that ExternalSort should not remove the input data and only removes intermediate data generated during external sort. * Second solution is to add some flags to indicate if each fragment is originated from local file instead of fetched data. If you have other ideas, feel free to suggest the ideas. - Hyunsik Choi On Sept. 18, 2014, 4:09 p.m., Hai Thanh Mai wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/25242/ > ----------------------------------------------------------- > > (Updated Sept. 18, 2014, 4:09 p.m.) > > > Review request for Tajo and Hyunsik Choi. > > > Repository: tajo > > > Description > ------- > > TAJO-983: Worker should directly read Intermediate data stored in localhost > rather than fetching > > Currently, worker always fetches all intermediate via Fetcher and than store > them in local file system even though some intermediate data already are > stored in local file system. It is inefficient and causes unnecessary I/O and > extra storage occupation. We should improve it. > > https://issues.apache.org/jira/browse/TAJO-983 > > > Diffs > ----- > > > tajo-core/src/main/java/org/apache/tajo/engine/planner/UniformRangePartition.java > db12285 > > tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java > 700e34d > tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java 4867fe4 > tajo-core/src/main/java/org/apache/tajo/worker/Task.java c9c83d1 > tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java b15d523 > tajo-storage/src/main/java/org/apache/tajo/storage/RawFile.java 1f57675 > > tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/PullServerAuxService.java > e6e7ce3 > > tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java > f7bc489 > > tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java > a8b424e > > Diff: https://reviews.apache.org/r/25242/diff/ > > > Testing > ------- > > > Thanks, > > Hai Thanh Mai > >
