----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34635/#review85079 -----------------------------------------------------------
Ship it! Ship It! - Daniel Dai On May 23, 2015, 7:24 p.m., Rohini Palaniswamy wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/34635/ > ----------------------------------------------------------- > > (Updated May 23, 2015, 7:24 p.m.) > > > Review request for pig. > > > Bugs: PIG-4564 > https://issues.apache.org/jira/browse/PIG-4564 > > > Repository: pig > > > Description > ------- > > This patch also incorporates PIG-4012 fixing > > The problem is we need to have POPartialAgg to be synchronous spill so > that its memory consumption is reduced before we invoke System.gc() in > SpillableMemoryManager. But for POPartialAgg to do the aggregation, it needs > to read the next record which can block on registerSpillable if there is a > bag in the record causing deadlock. This patch allows registering to the > spillables if doing POPartialAgg. For other normal spills it just blocks as > before. registerSpillable is still synchronized on spillables to avoid adding > to spillables when the spillablesSR is being constructed. But it mainly > relies on blockRegisterOnSpill variable to block from adding to spillables > when the actual spill is in progress. > > > Diffs > ----- > > > http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/util/SpillableMemoryManager.java > 1681368 > > Diff: https://reviews.apache.org/r/34635/diff/ > > > Testing > ------- > > Manually tested for a large job with big bags and lot of spills. > > > Thanks, > > Rohini Palaniswamy > >
