Re: Drill: Memory Spilling for the Hash Aggregate Operator

2017-01-16 Thread Boaz Ben-Zvi
ian Hyde <jh...@apache.org> Sent: Friday, January 13, 2017 11:00 PM To: d...@drill.apache.org Subject: Re: Drill: Memory Spilling for the Hash Aggregate Operator The attachment didn't come through. I'm hoping that you settled on a "hybrid" hash algorithm that can write to disk, or w

Re: Drill: Memory Spilling for the Hash Aggregate Operator

2017-01-16 Thread Boaz Ben-Zvi
Hi Julian, Yes, the model used for the memory spilling is the “hybrid” one — the memory space is pre-divided into “partitions” (based on the hash values of the key columns), and then as needed only some of the partitions would end up spilled to secondary storage (e.g. HDD), hence after