Hi Team, I have the below condition:
1. I have a cluster of ec2 instances with 10 nodes of m4.4x large size. Meaning, each m4.4x large node would have 16 vCPUs and 64 G memory with 300 IOPs. Since I mentioned in override-conf all the 10 IPs, and because I am able to see all the drill instances in one page, if i go to one node instance IP in the UI (http://one of the IPs in 10 nodes:8047), I am assume i am running drill in cluster of 10 ec2 instances/nodes 2. I have installed zookeeper on 9 of those same 10 nodes and 1 drillbit in each of those 10 nodes. It means, I am running 10 drillbits in 10 node cluster. 9 instances of zookeeper in the same 10 nodes cluster. 3. free -h command on each ec2 instance/node always varies depending what I am doing on that instance 4. However the memory configurations always gives me different messages on setting and available free memory or max proc memory. [WARN] You have an allocation of 26 GB that is currently unused from a total of 64 GB. You can increase your existing memory configuration to use this extra memory DRILLBIT_MAX_PROC_MEM=64G DRILL_HEAP=8G DRILL_MAX_DIRECT_MEMORY=29G DRILLBIT_CODE_CACHE_SIZE=1024m if I change the config according to the above message, sometimes i get this [INFO] Attempting to start up Drill with the following settings DRILL_HEAP=8G DRILL_MAX_DIRECT_MEMORY=39G DRILLBIT_CODE_CACHE_SIZE=1024m [WARN] Total Memory Allocation for Drillbit (48GB) exceeds available free memory (45GB) [WARN] Drillbit will start up, but can potentially crash due to oversubscribing of system memory. # free -h total used free shared buff/cache available Mem: 62G 7.0G 45G 73M 10G 54G Swap: 0B 0B 0B So, can't really determine on what basis the calculations are done. Whether it is based out free memory or on set max proc mem. However, i have these questions. 1. For an ec2 instance which has 64 GB memory, what should be the value of DRILL_HEAP=8G DRILL_MAX_DIRECT_MEMORY=39G DRILLBIT_CODE_CACHE_SIZE=1024m planner.memory.max_query_memory_per_node planner.memory.percent_per_query planner.width.max_per_node drill.exec.memory.operator.output_batch_size Are there any additional settings that I need to consider, to get the full benefit of large cluster that I have, in order to get faster results. The data size I want to work upon is in terms of TBs, but currently i am not able to execute queries on MBs of partioned tables also. To create a table with 7 GB data, it takes 140.493 seconds To create partitioned table with 7 GB data, it takes 4029.633 seconds To select values from a partitioned table, i get this 3,036 rows selected (34.639 seconds) 2. How does drill distribute this configuration across all the nodes? I mean, if i provide 39 GB of direct memory, is it only to that node or the whole cluster? why direct memory is not shown in UI as per the parameter that we set in drill-env.sh 3. I had the problem of select query execution earlier, that said i can't find XYZ file across cluster. So, i configured EFS for all the data files to reside in one place that is now mounted to all the nodes in the cluster. Some times, I can't select values from partitioned tables. The query doesn't respond at all. Response above would be highly appreciated. -- *Kind Regards* *Kiran Kumar NS* ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
