-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34635/
-----------------------------------------------------------
Review request for pig.
Bugs: PIG-4564
https://issues.apache.org/jira/browse/PIG-4564
Repository: pig
Description
-------
This patch also incorporates PIG-4012 fixing
The problem is we need to have POPartialAgg to be synchronous spill so that
its memory consumption is reduced before we invoke System.gc() in
SpillableMemoryManager. But for POPartialAgg to do the aggregation, it needs to
read the next record which can block on registerSpillable if there is a bag in
the record causing deadlock. This patch allows registering to the spillables if
doing POPartialAgg. For other normal spills it just blocks as before.
registerSpillable is still synchronized on spillables to avoid adding to
spillables when the spillablesSR is being constructed. But it mainly relies on
blockRegisterOnSpill variable to block from adding to spillables when the
actual spill is in progress.
Diffs
-----
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/util/SpillableMemoryManager.java
1681368
Diff: https://reviews.apache.org/r/34635/diff/
Testing
-------
Manually tested for a large job with big bags and lot of spills.
Thanks,
Rohini Palaniswamy
