> On Sept. 19, 2014, 8:18 p.m., Hyunsik Choi wrote: > > tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java, > > line 406 > > <https://reviews.apache.org/r/25242/diff/4/?file=693383#file693383line406> > > > > In my view, this condition is not too explicit. It is hard for other > > code reader to understand the code. > > > > In order to fully understand this if-condition, readers need to know > > lots of knowledges. Especially, code readers should know the fact that all > > local files whose start offset is 0 and the the end length is equivalent to > > file length can be removed because we can ensure they are are not shared > > with other tasks. > > > > I'd like to suggest three solutions: > > * One is that ExternalSort should not remove the input data and only > > removes intermediate data generated during external sort. > > * Second solution is to add some flags to indicate if each fragment is > > originated from local file instead of fetched data. > > > > If you have other ideas, feel free to suggest the ideas.
If the main purpose is to make the if-condition more clear, I will add comments to explain which intermediate file fragments should be deleted or not. I think that all intermediate files that will never be used in the future should be deleted to save disk space, no matter where the origins of the files are. - Hai Thanh ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25242/#review54015 ----------------------------------------------------------- On Sept. 18, 2014, 7:09 a.m., Hai Thanh Mai wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/25242/ > ----------------------------------------------------------- > > (Updated Sept. 18, 2014, 7:09 a.m.) > > > Review request for Tajo and Hyunsik Choi. > > > Repository: tajo > > > Description > ------- > > TAJO-983: Worker should directly read Intermediate data stored in localhost > rather than fetching > > Currently, worker always fetches all intermediate via Fetcher and than store > them in local file system even though some intermediate data already are > stored in local file system. It is inefficient and causes unnecessary I/O and > extra storage occupation. We should improve it. > > https://issues.apache.org/jira/browse/TAJO-983 > > > Diffs > ----- > > > tajo-core/src/main/java/org/apache/tajo/engine/planner/UniformRangePartition.java > db12285 > > tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java > 700e34d > tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java 4867fe4 > tajo-core/src/main/java/org/apache/tajo/worker/Task.java c9c83d1 > tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java b15d523 > tajo-storage/src/main/java/org/apache/tajo/storage/RawFile.java 1f57675 > > tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/PullServerAuxService.java > e6e7ce3 > > tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java > f7bc489 > > tajo-yarn-pullserver/src/main/java/org/apache/tajo/pullserver/retriever/FileChunk.java > a8b424e > > Diff: https://reviews.apache.org/r/25242/diff/ > > > Testing > ------- > > > Thanks, > > Hai Thanh Mai > >
