[
https://issues.apache.org/jira/browse/DRILL-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16963888#comment-16963888
]
ASF GitHub Bot commented on DRILL-7372:
---------------------------------------
vvysotskyi commented on pull request #1887: DRILL-7372: MethodAnalyzer consumes
too many memory
URL: https://github.com/apache/drill/pull/1887
Jira: [DRILL-7372](https://issues.apache.org/jira/browse/DRILL-7372).
- Replaced `Set<Integer>` with `BitSet` - it allowed to reduce heap usage
significantly.
- Avoided copying `localVariablesSet` and `labelsStack` from the source
frame when a new frame is created since the branch-sensitive analysis was
already done for a source frame.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> MethodAnalyzer consumes too many memory
> ---------------------------------------
>
> Key: DRILL-7372
> URL: https://issues.apache.org/jira/browse/DRILL-7372
> Project: Apache Drill
> Issue Type: Bug
> Affects Versions: 1.16.0
> Reporter: Vova Vysotskyi
> Assignee: Vova Vysotskyi
> Priority: Major
> Fix For: 1.17.0
>
>
> In the scope of DRILL-6524 was added logic for determining whether a variable
> is assigned in conditional block to prevent incorrect scalar replacement for
> such cases.
> But for some queries, this logic consumes too many memory, for example, for
> the following query:
> {code:sql}
> SELECT *
> FROM cp.`employee.json`
> WHERE employee_id+0 < employee_id
> OR employee_id+1 < employee_id
> AND employee_id+2 < employee_id
> OR employee_id+3 < employee_id
> AND employee_id+4 < employee_id
> OR employee_id+5 < employee_id
> AND employee_id+6 < employee_id
> OR employee_id+7 < employee_id
> AND employee_id+8 < employee_id
> OR employee_id+9 < employee_id
> AND employee_id+10 < employee_id
> OR employee_id+11 < employee_id
> AND employee_id+12 < employee_id
> OR employee_id+13 < employee_id
> AND employee_id+14 < employee_id
> OR employee_id+15 < employee_id
> AND employee_id+16 < employee_id
> OR employee_id+17 < employee_id
> AND employee_id+18 < employee_id
> OR employee_id+19 < employee_id
> AND employee_id+20 < employee_id
> OR employee_id+21 < employee_id
> AND employee_id+22 < employee_id
> OR employee_id+23 < employee_id
> AND employee_id+24 < employee_id
> OR employee_id+25 < employee_id
> AND employee_id+26 < employee_id
> OR employee_id+27 < employee_id
> AND employee_id+28 < employee_id
> OR employee_id+29 < employee_id
> AND employee_id+30 < employee_id
> OR employee_id+31 < employee_id
> AND employee_id+32 < employee_id
> OR employee_id+33 < employee_id
> AND employee_id+34 < employee_id
> OR employee_id+35 < employee_id
> AND employee_id+36 < employee_id
> OR employee_id+37 < employee_id
> AND employee_id+38 < employee_id
> OR employee_id+39 < employee_id
> AND employee_id+40 < employee_id
> OR employee_id+41 < employee_id
> AND employee_id+42 < employee_id
> OR employee_id+43 < employee_id
> AND employee_id+44 < employee_id
> OR employee_id+45 < employee_id
> AND employee_id+46 < employee_id
> OR employee_id+47 < employee_id
> AND employee_id+48 < employee_id
> OR employee_id+49 < employee_id
> AND TRUE;
> {code}
> Drill consumes more than 6 GB memory.
> One of the issues to fix is to replace {{Deque<Set<Integer>>
> localVariablesSet;}} with {{Deque<BitSet>}}, it will reduce memory usage
> significantly.
> Additionally should be investigated why these objects cannot be collected by
> GC.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
