Hi Team, Could you collect SQL profile for the problematic query and also drillbit.out + /var/log/messages from OS level? One finding is: open files (-n) 1024 It should be increased when you setup the cluster. It should be at least: open files (-n) 65535
Thanks, Hao From: Francis McGregor-Macdonald <fran...@mc-mac.com<mailto:fran...@mc-mac.com>> Reply-To: "user@drill.apache.org<mailto:user@drill.apache.org>" <user@drill.apache.org<mailto:user@drill.apache.org>> Date: Thursday, January 25, 2018 at 10:19 PM To: "user@drill.apache.org<mailto:user@drill.apache.org>" <user@drill.apache.org<mailto:user@drill.apache.org>> Subject: Fwd: Creating a Tableau extracts with Drill 1.12 uses unlimited memory Hi all, I am guessing that each of your EMR nodes are quite large? EMR nodes are: r4.2xlarge ('vcpu': 8, 'memory': 61) Property "planner.width.max_per_node" is set to = 6 What is the system memory and what are the allocations for heap and direct? System Memory: 61GB (EMR nodes above) drill_mem_heap: 12G drill_mem_max: 48G The view is simple: SELECT * FROM s3://myparquet.parquet (14GB) planner.memory.max_query_memory_per_node = 10479720202 Drillbit.log attached (I think I have the correct selection included). Thanks On Fri, Jan 26, 2018 at 2:41 PM, Kunal Khatua <kkha...@mapr.com<mailto:kkha...@mapr.com>> wrote: What is the system memory and what are the allocations for heap and direct? The memory crash might be occurring due to insufficient heap. The limits parameter applies to the direct memory and not Heap. Can you share details in the logs from the crash? -----Original Message----- From: Timothy Farkas [mailto:tfar...@mapr.com<mailto:tfar...@mapr.com>] Sent: Thursday, January 25, 2018 2:58 PM To: user@drill.apache.org<mailto:user@drill.apache.org> Subject: Re: Creating a Tableau extracts with Drill 1.12 uses unlimited memory Hi Francis, I am guessing that each of your EMR nodes are quite large (32 or 64 vcpus). On large machines Drill's planner over parallelizes and over allocates memory. There is a property "planner.width.max_per_node" which limits the number of operators that can simultaneously execute on a Drillbit for a query. If you configure the width per node to something like 5 or 10 (you may have to play around with it) things should start working. Thanks, Tim ________________________________ From: Francis McGregor-Macdonald <fran...@mc-mac.com<mailto:fran...@mc-mac.com>> Sent: Thursday, January 25, 2018 1:58:22 PM To: user@drill.apache.org<mailto:user@drill.apache.org> Subject: Creating a Tableau extracts with Drill 1.12 uses unlimited memory Creating a creating a Tableau (with 10.3, 10.5 desktop) extract from a Drill (1.12 on EMR) cluster memory appears not to adhere to the limits set by planner.memory.max_query_memory_per_node. The extract query consumes all memory and then crashes drill. Running the same query as a create table memory behaves as expected. The query complexity is trivial: select * from view only a single parquet with no calculated fields. Has anyone else observed this behavior?
