Great, thanks Tim. In that case, I suggest branching 2.10. at the following commit. Please let me know if someone disagrees. Also, thanks everyone for cleaning up 2.10.0 targeted jiras. I don't see any potential release blockers.
commit 2a7c8b9011905bfeb21b0610f0739f9df9daacef Author: Tim Armstrong <[email protected]> Date: Fri Aug 18 14:47:52 2017 -0700 IMPALA-5713: always reserve memory for preaggs Before this change the preaggregation was frequently disabled when running under some memory pressure, e.g. if the aggregation is at the end of a pipeline of joins and those joins eat up all the memory. This can result in huge performance degradation since all rows must then be sent over the network. This change always reserves 16 * (buffer size + 64kb) bytes per preaggregation so that it is always able to build some hash tables and reduce the input somewhat. This has two parts: * Changing the frontend reservation calculation * Removing dead code in the backend that handled the case when the initial partitions and hash tables could not be allocated. Testing: Passes exhaustive tests. On Thu, Aug 24, 2017 at 5:55 PM, Tim Armstrong <[email protected]> wrote: > All of the IMPALA-3200 work is now in master! > > On Wed, Aug 23, 2017 at 12:21 PM, Tim Armstrong <[email protected]> > wrote: > > > I was looking through open JIRAs to make sure I didn't drop the ball on > > any buffer pool changes and discovered we have 100+ open JIRAs targeted > for > > 2.10: https://issues.apache.org/jira/issues/?filter=12341748 > > > > It would be great to clean those up. I tried to clean up the ones that I > > know something about but most of them I'm not familiar with. It looks > like > > a lot aren't being actively worked on so probably belong in the backlog - > > the target version seems to just be expressing a hope that someone else > > will fix it soon. > > > > You can check your own 2.10 JIRAs with this filter: > > https://issues.apache.org/jira/issues/?filter=12341563 > > > > There are also a bunch of unassigned ones: https://issues.apache.org/ > > jira/issues/?filter=12341750 > > > > > > > > On Mon, Aug 14, 2017 at 11:09 AM, Bharath Vissapragada < > > [email protected]> wrote: > > > >> Agreed Tim. > >> > >> On Mon, Aug 14, 2017 at 9:13 AM, Tim Armstrong <[email protected] > > > >> wrote: > >> > >> > Sounds good to me. We should coordinate to make sure that all of > >> > https://issues.apache.org/jira/browse/IMPALA-3200 (the buffer pool > >> > changes) > >> > and related fixes make it into the release. > >> > > >> > - Tim > >> > > >> > On Mon, Aug 14, 2017 at 5:52 AM, Jim Apple <[email protected]> > >> wrote: > >> > > >> > > This sounds like a good idea to me. Thank you for volunteering! > >> > > > >> > > On Mon, Aug 14, 2017 at 12:37 AM, Bharath Vissapragada > >> > > <[email protected]> wrote: > >> > > > Folks, > >> > > > > >> > > > It has been almost 2 months since we released Apache Impala > >> > (incubating) > >> > > > 2.9.0 and there have been new feature improvements and a good > >> number of > >> > > bug > >> > > > fixes checked in since then. > >> > > > > >> > > > I propose that we release 2.10.0 soon and I volunteer to be its > >> release > >> > > > manager. Please speak up and let the community know if anyone has > >> any > >> > > > objections to this. > >> > > > > >> > > > Thanks, > >> > > > Bharath > >> > > > >> > > >> > > > > >
