[
https://issues.apache.org/jira/browse/DRILL-7338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998363#comment-16998363
]
mehran commented on DRILL-7338:
-------------------------------
It is not only about REST API but also this error is occurred with ODBC client.
It is a blocking issue and database hangs and should be restarted with this
issue. Besides I think some times it is a directory pruning issue when we give
a high level directory and have a constrain on dir0 or dir1. In my case it is
occurred in planning phase. I do not know why this planning phase should get so
long time. (it may be because of scanning parquet files).
> REST API calls to Drill fail due to insufficient heap memory
> ------------------------------------------------------------
>
> Key: DRILL-7338
> URL: https://issues.apache.org/jira/browse/DRILL-7338
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
> Affects Versions: 1.15.0
> Reporter: Aditya Allamraju
> Assignee: Kunal Khatua
> Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.17.0
>
>
> Drill queries that use REST API calls have started failing(given below) after
> recent changes.
> {code:java}
> RESOURCE ERROR: There is not enough heap memory to run this query using the
> web interface.
> Please try a query with fewer columns or with a filter or limit condition to
> limit the data returned.
> You can also try an ODBC/JDBC client.{code}
> They were running fine earlier as the ResultSet returned was just few rows.
> These queries now fail for even very small resultSets( < 10rows).
> Investigating the issue revealed that we introduced a check to limit the Heap
> usage.
> The Wrapper code from
> *_exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/QueryWrapper.java_*
> that throws this error, i see certain issues. It does seem we use a
> threshold of *85%* of heap usage before throwing that warning and exiting the
> query.
>
> {code:java}
> public class QueryWrapper {
> private static final org.slf4j.Logger logger =
> org.slf4j.LoggerFactory.getLogger(QueryWrapper.class);
> // Heap usage threshold/trigger to provide resiliency on web server for
> queries submitted via HTTP
> private static final double HEAP_MEMORY_FAILURE_THRESHOLD = 0.85;
> ...
> private static MemoryMXBean memMXBean = ManagementFactory.getMemoryMXBean();
> ...
> // Wait until the query execution is complete or there is error submitting
> the query
> logger.debug("Wait until the query execution is complete or there is
> error submitting the query");
> do {
> try {
> isComplete = webUserConnection.await(TimeUnit.SECONDS.toMillis(1));
> //periodically timeout 1 sec to check heap
> } catch (InterruptedException e) {}
> usagePercent = getHeapUsage();
> if (usagePercent > HEAP_MEMORY_FAILURE_THRESHOLD) {
> nearlyOutOfHeapSpace = true;
> }
> } while (!isComplete && !nearlyOutOfHeapSpace);
> {code}
> By using above check, we unintentionally invited all those issues that happen
> with Java’s Heap usage. JVM does try to make maximum usage of HEAP until
> Minor or Major GC kicks in i.e GC kicks after there is no more space left in
> heap(eden or young gen).
> The workarounds i can think of in order to resolve this issue are:
> # Remove this check altogether so we know why it is filling up Heap.
> # Advise the users to stop using REST for querying data.(We did this
> already). *But not all users may not be happy with this suggestion.* There
> could be few dynamic applications(dashboard, monitoring etc).
> # Make the threshold high enough so that GC kicks in much better.
> If not above options, we have to tune the Heap sizes of drillbit. A quick fix
> would be to increase the threshold from 85% to 100%(option-3 above).
> *For documentation*
> New Drill configuration property -
> drill.exec.http.memory.heap.failure.threshold
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
