Hi Sreeparna, I know your dataset is 15 million rows and 16 columns, but how big is the result set you are expecting from that query?
I think that result set is too large for Drill's REST interface to handle especially with only 16G heap. I try to keep the REST queries in Drill to about 10k rows with limited number of columns. JDBC or ODBC can handle MUCH larger volumes without issue. Best, Rafael On Fri, May 1, 2020 at 1:39 PM sreeparna bhabani < [email protected]> wrote: > Hi Team, > > Kindly suggest on the below problem which we are facing in Apache Drill > while running query in Web interface. When we are executing query from > Fiddler, the query is failing after 2 mins saying "There is not enough heap > memory to run this query using the web interface" . We have tried few > memory settings. But the issue persists. The query performance is not as > expected either. Please find the details below- > > Dataset details- > File type - Parquet > Size of file - 401 MB > Number of selected columns - 16 > Number of rows - 15136976 > There is no sort, no joins. > > Drill Setup- > 2 Drillbits are used > DRILL_HEAP - 16 G > DRILL_MAX_DIRECT_MEMORY - 32 G > planner.memory.max_query_memory_per_node - 31147483648 > planner.width.max_per_node- 0 (DEFAULT value. Can you please share some > details why the default is 0) > drill.exec.memory.operator.output_batch_size - 16777216 > > Operator profile- > Type Max process Time Max wait time Max Peak Memory > SCREEN 0.048s 2m40s 4MB > PROJECT 0.068s 0 4MB > UNORDERED_RECEIVER 0.088s 0.020s 418MB > SINGLE_SENDER 0.011s 2m31s 8MB > PROJECT 4.187s 0 8MB > PARQUET_ROW_GROUP_SCAN 1.765s 0.020s 12MB > How to find the reason of the wait time of SINGLE_SENDER. It is not clear > from the profile. > I found one Jira regarding the Batch sizing of SINGLE_SENDER. Not sure > whether it is related to this issue or not. > https://issues.apache.org/jira/browse/DRILL-7093 > > Please let me know if any other information is required. > > Thanks n Regards, > *Sreeparna Bhabani* >
