What is the system memory and what are the allocations for heap and direct? The memory crash might be occurring due to insufficient heap. The limits parameter applies to the direct memory and not Heap.
Can you share details in the logs from the crash? -----Original Message----- From: Timothy Farkas [mailto:[email protected]] Sent: Thursday, January 25, 2018 2:58 PM To: [email protected] Subject: Re: Creating a Tableau extracts with Drill 1.12 uses unlimited memory Hi Francis, I am guessing that each of your EMR nodes are quite large (32 or 64 vcpus). On large machines Drill's planner over parallelizes and over allocates memory. There is a property "planner.width.max_per_node" which limits the number of operators that can simultaneously execute on a Drillbit for a query. If you configure the width per node to something like 5 or 10 (you may have to play around with it) things should start working. Thanks, Tim ________________________________ From: Francis McGregor-Macdonald <[email protected]> Sent: Thursday, January 25, 2018 1:58:22 PM To: [email protected] Subject: Creating a Tableau extracts with Drill 1.12 uses unlimited memory Creating a creating a Tableau (with 10.3, 10.5 desktop) extract from a Drill (1.12 on EMR) cluster memory appears not to adhere to the limits set by planner.memory.max_query_memory_per_node. The extract query consumes all memory and then crashes drill. Running the same query as a create table memory behaves as expected. The query complexity is trivial: select * from view only a single parquet with no calculated fields. Has anyone else observed this behavior?
