Created DRILL-4236 <https://issues.apache.org/jira/browse/DRILL-4236> to keep track of this improvement.
On Wed, Dec 30, 2015 at 11:01 AM, Jacques Nadeau <[email protected]> wrote: > Since the accounting changed (more accurate), the termination condition for > the sort operator will be different than before. In fact, this likely will > be sooner since our accounting is much larger than previously (since we > correctly consider the entire allocation rather than simply the used > allocation). > > Hakim, > Steven and I were discussing the need to update the ExternalSort operator > to use the new allocator functionality to better manage its memory > envelope. Would you be interested in working on this since you seem to be > working with that code the most? Basically, it used to be that there was no > way the sort operator would be able to correctly detect a memory condition > and so it jumped through a bunch of hoops to try to figure out the > termination condition.With the transfer accounting in place, this code can > be greatly simplified to just use the current operator memory allocation. > > -- > Jacques Nadeau > CTO and Co-Founder, Dremio > > On Wed, Dec 30, 2015 at 10:48 AM, rahul challapalli < > [email protected]> wrote: > > > I installed the latest master and ran this query. So > > planner.memory.max_query_memory_per_node should have been the default > > value. I switched back to 1.4.0 branch and this query completed > > successfully. > > > > On Wed, Dec 30, 2015 at 10:37 AM, Abdel Hakim Deneche < > > [email protected] > > > wrote: > > > > > Rahul, > > > > > > How much memory was assigned to the sort operator ( > > > planner.memory.max_query_memory_per_node) ? > > > > > > On Wed, Dec 30, 2015 at 9:54 AM, rahul challapalli < > > > [email protected]> wrote: > > > > > > > I am seeing an OOM error while executing a simple CTAS query. I > raised > > > > DRILL-4324 for this. The query mentioned in the JIRA used to complete > > > > successfully without any issue prior to 1.5. Any idea what could have > > > > caused the regression? > > > > > > > > - Rahul > > > > > > > > > > > > > > > > -- > > > > > > Abdelhakim Deneche > > > > > > Software Engineer > > > > > > <http://www.mapr.com/> > > > > > > > > > Now Available - Free Hadoop On-Demand Training > > > < > > > > > > http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available > > > > > > > > > > -- Abdelhakim Deneche Software Engineer <http://www.mapr.com/> Now Available - Free Hadoop On-Demand Training <http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
