[
https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-27854 started by chenruotao.
-----------------------------------------
> Driver compile may causing memory leak
> --------------------------------------
>
> Key: HIVE-27854
> URL: https://issues.apache.org/jira/browse/HIVE-27854
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Affects Versions: 3.1.2, 4.0.0-beta-1
> Reporter: chenruotao
> Assignee: chenruotao
> Priority: Major
> Labels: pull-request-available
> Attachments: image-2023-11-06-18-11-47-832.png,
> image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png,
> image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png,
> image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png,
> image-2023-11-06-19-40-49-488.png, image-2023-11-06-19-45-03-388.png
>
>
> when submit a sql to hiveserver, if the sql query timeout before compile, it
> will continue compile and execute the addShutdownHook method, but now no
> thread to call the close method of driver, so it alway stay in
> ShutdownHookManager's hooks, causing a memory leak
> !image-2023-11-06-18-11-47-832.png|width=943,height=241!
> !image-2023-11-06-19-45-03-388.png!
> the cancel from SQLOperation will set it's driver = null, so it can not call
> the close method of driver after SQLOperation exexute cancel
> !image-2023-11-06-18-13-43-182.png!
> !image-2023-11-06-18-13-00-842.png!
> it will not trigger everytime, but we could do this to trigger it
> 1、sleep 5s before compile
> !image-2023-11-06-18-21-19-056.png!
> 2、set hive.query.timeout.seconds to 4
> !image-2023-11-06-18-22-45-812.png!
> 3、now everytime I submit sql , the shutdownHookManager$HookEntry will add one
> !image-2023-11-06-18-24-41-982.png|width=842,height=134!
> i solve it in hive 3.1.2 by adding this code :
> !image-2023-11-06-19-30-14-120.png!
> Obviously we see no more memory leak, and shutdownHookManager$HookEntry num
> will not increase every time i submit sql
> !image-2023-11-06-19-40-49-488.png|width=956,height=140!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
