[
https://issues.apache.org/jira/browse/DRILL-7338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kunal Khatua resolved DRILL-7338.
---------------------------------
Resolution: Fixed
Reviewer: Arina Ielchiieva
> REST API calls to Drill fail due to insufficient heap memory
> ------------------------------------------------------------
>
> Key: DRILL-7338
> URL: https://issues.apache.org/jira/browse/DRILL-7338
> Project: Apache Drill
> Issue Type: Bug
> Components: Web Server
> Affects Versions: 1.15.0
> Reporter: Aditya Allamraju
> Assignee: Kunal Khatua
> Priority: Major
> Fix For: 1.17.0
>
>
> Drill queries that use REST API calls have started failing(given below) after
> recent changes.
> {code:java}
> RESOURCE ERROR: There is not enough heap memory to run this query using the
> web interface.
> Please try a query with fewer columns or with a filter or limit condition to
> limit the data returned.
> You can also try an ODBC/JDBC client.{code}
> They were running fine earlier as the ResultSet returned was just few rows.
> These queries now fail for even very small resultSets( < 10rows).
> Investigating the issue revealed that we introduced a check to limit the Heap
> usage.
> The Wrapper code from
> *_exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/QueryWrapper.java_*
> that throws this error, i see certain issues. It does seem we use a
> threshold of *85%* of heap usage before throwing that warning and exiting the
> query.
>
> {code:java}
> public class QueryWrapper {
> private static final org.slf4j.Logger logger =
> org.slf4j.LoggerFactory.getLogger(QueryWrapper.class);
> // Heap usage threshold/trigger to provide resiliency on web server for
> queries submitted via HTTP
> private static final double HEAP_MEMORY_FAILURE_THRESHOLD = 0.85;
> ...
> private static MemoryMXBean memMXBean = ManagementFactory.getMemoryMXBean();
> ...
> // Wait until the query execution is complete or there is error submitting
> the query
> logger.debug("Wait until the query execution is complete or there is
> error submitting the query");
> do {
> try {
> isComplete = webUserConnection.await(TimeUnit.SECONDS.toMillis(1));
> //periodically timeout 1 sec to check heap
> } catch (InterruptedException e) {}
> usagePercent = getHeapUsage();
> if (usagePercent > HEAP_MEMORY_FAILURE_THRESHOLD) {
> nearlyOutOfHeapSpace = true;
> }
> } while (!isComplete && !nearlyOutOfHeapSpace);
> {code}
> By using above check, we unintentionally invited all those issues that happen
> with Java’s Heap usage. JVM does try to make maximum usage of HEAP until
> Minor or Major GC kicks in i.e GC kicks after there is no more space left in
> heap(eden or young gen).
> The workarounds i can think of in order to resolve this issue are:
> # Remove this check altogether so we know why it is filling up Heap.
> # Advise the users to stop using REST for querying data.(We did this
> already). *But not all users may not be happy with this suggestion.* There
> could be few dynamic applications(dashboard, monitoring etc).
> # Make the threshold high enough so that GC kicks in much better.
> If not above options, we have to tune the Heap sizes of drillbit. A quick fix
> would be to increase the threshold from 85% to 100%(option-3 above).
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
