Thanks Paul and Kunal, I think I have the right information now. With Paul's changes (and fixing up a zoo.cfg error) it isn't crashing, rather failing. Logs attached, still blowing past memory limits. It does the same thing when re-running the query from the web console so presumably its not actually Tableau related despite me first generating it that way.
Thanks. On Sat, Jan 27, 2018 at 1:15 PM, Francis McGregor-Macdonald < [email protected]> wrote: > Thanks Paul, > > I will update with your suggested memory allocations also and retry. > > Zookeeper crashed too which might explain more? I have attached the logs > from Zookeeper too. > > Thanks > > On Sat, Jan 27, 2018 at 6:45 AM, Paul Rogers <[email protected]> wrote: > >> Hi Francis, >> >> Thanks much for the log. The log shows running a query, then immediately >> shows entries that occur when starting Drill. I'm guessing that Drill >> literally crashed at this point? This is more severe than the usual error >> in which a query exhausts memory. >> >> Some general observations. The Drill memory is 60 GB, but system memory >> is 61 GB. Perhaps try dropping total Drill memory some to give the OS and >> other tasks more headroom. For a SELECT * memory, Drill needs far less than >> what you have, so maybe try giving Drill 48 GB total. >> >> Then, Drill needs direct memory much more than heap. So, maybe give Drill >> 39 GB direct, 8 GB heap and 1 GB (the default) for code cache. These >> settings are in drill-env.sh. >> >> Kunal, you have more experience with these issues. Can you make >> additional suggestions by looking at the log? >> >> Thanks, >> >> - Paul >> >> >> >> On Thursday, January 25, 2018, 10:20:29 PM PST, Francis >> McGregor-Macdonald <[email protected]> wrote: >> >> >> Hi all, >> >> I am guessing that each of your EMR nodes are quite large? EMR nodes are: >> r4.2xlarge ('vcpu': 8, 'memory': 61) >> >> Property "planner.width.max_per_node" is set to = 6 >> >> What is the system memory and what are the allocations for heap and >> direct? >> System Memory: 61GB (EMR nodes above) >> drill_mem_heap: 12G >> drill_mem_max: 48G >> >> The view is simple: SELECT * FROM s3://myparquet.parquet (14GB) >> >> planner.memory.max_query_memor y_per_node = 10479720202 >> >> Drillbit.log attached (I think I have the correct selection included). >> >> Thanks >> >> On Fri, Jan 26, 2018 at 2:41 PM, Kunal Khatua <[email protected]> wrote: >> >> What is the system memory and what are the allocations for heap and >> direct? The memory crash might be occurring due to insufficient heap. The >> limits parameter applies to the direct memory and not Heap. >> >> Can you share details in the logs from the crash? >> >> -----Original Message----- >> From: Timothy Farkas [mailto:[email protected]] >> Sent: Thursday, January 25, 2018 2:58 PM >> To: [email protected] >> Subject: Re: Creating a Tableau extracts with Drill 1.12 uses unlimited >> memory >> >> Hi Francis, >> >> I am guessing that each of your EMR nodes are quite large (32 or 64 >> vcpus). On large machines Drill's planner over parallelizes and over >> allocates memory. There is a property "planner.width.max_per_node" which >> limits the number of operators that can simultaneously execute on a >> Drillbit for a query. If you configure the width per node to something like >> 5 or 10 (you may have to play around with it) things should start working. >> >> Thanks, >> Tim >> >> ______________________________ __ >> From: Francis McGregor-Macdonald <[email protected]> >> Sent: Thursday, January 25, 2018 1:58:22 PM >> To: [email protected] >> Subject: Creating a Tableau extracts with Drill 1.12 uses unlimited memory >> >> Creating a creating a Tableau (with 10.3, 10.5 desktop) extract from a >> Drill (1.12 on EMR) cluster memory appears not to adhere to the limits set >> by planner.memory.max_query_memor y_per_node. >> >> The extract query consumes all memory and then crashes drill. >> >> Running the same query as a create table memory behaves as expected. >> >> The query complexity is trivial: >> select * from view only a single parquet with no calculated fields. >> >> Has anyone else observed this behavior? >> >> >> >> >
