Hi Francis, I am guessing that each of your EMR nodes are quite large (32 or 64 vcpus). On large machines Drill's planner over parallelizes and over allocates memory. There is a property "planner.width.max_per_node" which limits the number of operators that can simultaneously execute on a Drillbit for a query. If you configure the width per node to something like 5 or 10 (you may have to play around with it) things should start working.
Thanks, Tim ________________________________ From: Francis McGregor-Macdonald <[email protected]> Sent: Thursday, January 25, 2018 1:58:22 PM To: [email protected] Subject: Creating a Tableau extracts with Drill 1.12 uses unlimited memory Creating a creating a Tableau (with 10.3, 10.5 desktop) extract from a Drill (1.12 on EMR) cluster memory appears not to adhere to the limits set by planner.memory.max_query_memory_per_node. The extract query consumes all memory and then crashes drill. Running the same query as a create table memory behaves as expected. The query complexity is trivial: select * from view only a single parquet with no calculated fields. Has anyone else observed this behavior?
