[
https://issues.apache.org/jira/browse/DRILL-7372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16966800#comment-16966800
]
ASF GitHub Bot commented on DRILL-7372:
---------------------------------------
vvysotskyi commented on issue #1887: DRILL-7372: MethodAnalyzer consumes too
much memory
URL: https://github.com/apache/drill/pull/1887#issuecomment-549433415
@paul-rogers, unfortunately, I didn't see the results of these performance
measurements, but I believe that there are some corner cases where our byte
code analysis will bring some benefit, for example, when the generated method
is too large for JVM optimizations and general time for query execution is much
larger compared to the time required for producing scalar replacement and
classes merging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> MethodAnalyzer consumes too much memory
> ---------------------------------------
>
> Key: DRILL-7372
> URL: https://issues.apache.org/jira/browse/DRILL-7372
> Project: Apache Drill
> Issue Type: Bug
> Affects Versions: 1.16.0
> Reporter: Vova Vysotskyi
> Assignee: Vova Vysotskyi
> Priority: Major
> Labels: ready-to-commit
> Fix For: 1.17.0
>
>
> In the scope of DRILL-6524 was added logic for determining whether a variable
> is assigned in conditional block to prevent incorrect scalar replacement for
> such cases.
> But for some queries, this logic consumes too many memory, for example, for
> the following query:
> {code:sql}
> SELECT *
> FROM cp.`employee.json`
> WHERE employee_id+0 < employee_id
> OR employee_id+1 < employee_id
> AND employee_id+2 < employee_id
> OR employee_id+3 < employee_id
> AND employee_id+4 < employee_id
> OR employee_id+5 < employee_id
> AND employee_id+6 < employee_id
> OR employee_id+7 < employee_id
> AND employee_id+8 < employee_id
> OR employee_id+9 < employee_id
> AND employee_id+10 < employee_id
> OR employee_id+11 < employee_id
> AND employee_id+12 < employee_id
> OR employee_id+13 < employee_id
> AND employee_id+14 < employee_id
> OR employee_id+15 < employee_id
> AND employee_id+16 < employee_id
> OR employee_id+17 < employee_id
> AND employee_id+18 < employee_id
> OR employee_id+19 < employee_id
> AND employee_id+20 < employee_id
> OR employee_id+21 < employee_id
> AND employee_id+22 < employee_id
> OR employee_id+23 < employee_id
> AND employee_id+24 < employee_id
> OR employee_id+25 < employee_id
> AND employee_id+26 < employee_id
> OR employee_id+27 < employee_id
> AND employee_id+28 < employee_id
> OR employee_id+29 < employee_id
> AND employee_id+30 < employee_id
> OR employee_id+31 < employee_id
> AND employee_id+32 < employee_id
> OR employee_id+33 < employee_id
> AND employee_id+34 < employee_id
> OR employee_id+35 < employee_id
> AND employee_id+36 < employee_id
> OR employee_id+37 < employee_id
> AND employee_id+38 < employee_id
> OR employee_id+39 < employee_id
> AND employee_id+40 < employee_id
> OR employee_id+41 < employee_id
> AND employee_id+42 < employee_id
> OR employee_id+43 < employee_id
> AND employee_id+44 < employee_id
> OR employee_id+45 < employee_id
> AND employee_id+46 < employee_id
> OR employee_id+47 < employee_id
> AND employee_id+48 < employee_id
> OR employee_id+49 < employee_id
> AND TRUE;
> {code}
> Drill consumes more than 6 GB memory.
> One of the issues to fix is to replace {{Deque<Set<Integer>>
> localVariablesSet;}} with {{Deque<BitSet>}}, it will reduce memory usage
> significantly.
> Additionally should be investigated why these objects cannot be collected by
> GC.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
