Steven, were you able to successfully run the regression tests on the transfer patch ? I just tried and saw several queries running out of memory !
On Wed, Dec 30, 2015 at 11:46 AM, Abdel Hakim Deneche <[email protected] > wrote: > Created DRILL-4236 <https://issues.apache.org/jira/browse/DRILL-4236> to > keep track of this improvement. > > On Wed, Dec 30, 2015 at 11:01 AM, Jacques Nadeau <[email protected]> > wrote: > >> Since the accounting changed (more accurate), the termination condition >> for >> the sort operator will be different than before. In fact, this likely will >> be sooner since our accounting is much larger than previously (since we >> correctly consider the entire allocation rather than simply the used >> allocation). >> >> Hakim, >> Steven and I were discussing the need to update the ExternalSort operator >> to use the new allocator functionality to better manage its memory >> envelope. Would you be interested in working on this since you seem to be >> working with that code the most? Basically, it used to be that there was >> no >> way the sort operator would be able to correctly detect a memory condition >> and so it jumped through a bunch of hoops to try to figure out the >> termination condition.With the transfer accounting in place, this code can >> be greatly simplified to just use the current operator memory allocation. >> >> -- >> Jacques Nadeau >> CTO and Co-Founder, Dremio >> >> On Wed, Dec 30, 2015 at 10:48 AM, rahul challapalli < >> [email protected]> wrote: >> >> > I installed the latest master and ran this query. So >> > planner.memory.max_query_memory_per_node should have been the default >> > value. I switched back to 1.4.0 branch and this query completed >> > successfully. >> > >> > On Wed, Dec 30, 2015 at 10:37 AM, Abdel Hakim Deneche < >> > [email protected] >> > > wrote: >> > >> > > Rahul, >> > > >> > > How much memory was assigned to the sort operator ( >> > > planner.memory.max_query_memory_per_node) ? >> > > >> > > On Wed, Dec 30, 2015 at 9:54 AM, rahul challapalli < >> > > [email protected]> wrote: >> > > >> > > > I am seeing an OOM error while executing a simple CTAS query. I >> raised >> > > > DRILL-4324 for this. The query mentioned in the JIRA used to >> complete >> > > > successfully without any issue prior to 1.5. Any idea what could >> have >> > > > caused the regression? >> > > > >> > > > - Rahul >> > > > >> > > >> > > >> > > >> > > -- >> > > >> > > Abdelhakim Deneche >> > > >> > > Software Engineer >> > > >> > > <http://www.mapr.com/> >> > > >> > > >> > > Now Available - Free Hadoop On-Demand Training >> > > < >> > > >> > >> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available >> > > > >> > > >> > >> > > > > -- > > Abdelhakim Deneche > > Software Engineer > > <http://www.mapr.com/> > > > Now Available - Free Hadoop On-Demand Training > <http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available> > -- Abdelhakim Deneche Software Engineer <http://www.mapr.com/> Now Available - Free Hadoop On-Demand Training <http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
