Thanks Paul, I will update with your suggested memory allocations also and retry.
Zookeeper crashed too which might explain more? I have attached the logs from Zookeeper too. Thanks On Sat, Jan 27, 2018 at 6:45 AM, Paul Rogers <[email protected]> wrote: > Hi Francis, > > Thanks much for the log. The log shows running a query, then immediately > shows entries that occur when starting Drill. I'm guessing that Drill > literally crashed at this point? This is more severe than the usual error > in which a query exhausts memory. > > Some general observations. The Drill memory is 60 GB, but system memory is > 61 GB. Perhaps try dropping total Drill memory some to give the OS and > other tasks more headroom. For a SELECT * memory, Drill needs far less than > what you have, so maybe try giving Drill 48 GB total. > > Then, Drill needs direct memory much more than heap. So, maybe give Drill > 39 GB direct, 8 GB heap and 1 GB (the default) for code cache. These > settings are in drill-env.sh. > > Kunal, you have more experience with these issues. Can you make additional > suggestions by looking at the log? > > Thanks, > > - Paul > > > > On Thursday, January 25, 2018, 10:20:29 PM PST, Francis McGregor-Macdonald > <[email protected]> wrote: > > > Hi all, > > I am guessing that each of your EMR nodes are quite large? EMR nodes are: > r4.2xlarge ('vcpu': 8, 'memory': 61) > > Property "planner.width.max_per_node" is set to = 6 > > What is the system memory and what are the allocations for heap and direct? > System Memory: 61GB (EMR nodes above) > drill_mem_heap: 12G > drill_mem_max: 48G > > The view is simple: SELECT * FROM s3://myparquet.parquet (14GB) > > planner.memory.max_query_memor y_per_node = 10479720202 > > Drillbit.log attached (I think I have the correct selection included). > > Thanks > > On Fri, Jan 26, 2018 at 2:41 PM, Kunal Khatua <[email protected]> wrote: > > What is the system memory and what are the allocations for heap and > direct? The memory crash might be occurring due to insufficient heap. The > limits parameter applies to the direct memory and not Heap. > > Can you share details in the logs from the crash? > > -----Original Message----- > From: Timothy Farkas [mailto:[email protected]] > Sent: Thursday, January 25, 2018 2:58 PM > To: [email protected] > Subject: Re: Creating a Tableau extracts with Drill 1.12 uses unlimited > memory > > Hi Francis, > > I am guessing that each of your EMR nodes are quite large (32 or 64 > vcpus). On large machines Drill's planner over parallelizes and over > allocates memory. There is a property "planner.width.max_per_node" which > limits the number of operators that can simultaneously execute on a > Drillbit for a query. If you configure the width per node to something like > 5 or 10 (you may have to play around with it) things should start working. > > Thanks, > Tim > > ______________________________ __ > From: Francis McGregor-Macdonald <[email protected]> > Sent: Thursday, January 25, 2018 1:58:22 PM > To: [email protected] > Subject: Creating a Tableau extracts with Drill 1.12 uses unlimited memory > > Creating a creating a Tableau (with 10.3, 10.5 desktop) extract from a > Drill (1.12 on EMR) cluster memory appears not to adhere to the limits set > by planner.memory.max_query_memor y_per_node. > > The extract query consumes all memory and then crashes drill. > > Running the same query as a create table memory behaves as expected. > > The query complexity is trivial: > select * from view only a single parquet with no calculated fields. > > Has anyone else observed this behavior? > > > >
