[
https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenruotao updated HIVE-27854:
------------------------------
Description:
when submit a sql to hiveserver, if the sql query timeout before compile, it
will continue compile and execute addShutdownHook, but now no thread to call
the close method of driver, so it alway stay in ShutdownHookManager's hooks,
causing a memory leak
!image-2023-11-06-18-11-47-832.png!
the cancel from SQLOperation will set it's driver = null, so it can not call
the close method of driver
!image-2023-11-06-18-13-43-182.png!
!image-2023-11-06-18-13-00-842.png!
it will not trigger everytime, but we could do this to trigger it
1、sleep 5s before compile
!image-2023-11-06-18-21-19-056.png!
2、set hive.query.timeout.seconds to 4
!image-2023-11-06-18-22-45-812.png!
3、now everytime I submit sql , the ShutdownHookManager's hooks will add one
!image-2023-11-06-18-24-41-982.png|width=842,height=134!
i solve it in hive 3.1.2 by add this code :
!image-2023-11-06-19-30-14-120.png!
and obviously we see no more memory leak
was:
when submit a sql to hiveserver, if the sql query timeout before compile, it
will continue compile and execute addShutdownHook, but now no thread to call
the close method of driver, so it alway stay in ShutdownHookManager's hooks,
causing a memory leak
!image-2023-11-06-18-11-47-832.png!
the cancel from SQLOperation will set it's driver = null, so it can not call
the close method of driver
!image-2023-11-06-18-13-43-182.png!
!image-2023-11-06-18-13-00-842.png!
it will not trigger everytime, but we could do this to trigger it
1、sleep 5s before compile
!image-2023-11-06-18-21-19-056.png!
2、set hive.query.timeout.seconds to 4
!image-2023-11-06-18-22-45-812.png!
3、now everytime I submit sql , the ShutdownHookManager's hooks will add one
!image-2023-11-06-18-24-41-982.png|width=842,height=134!
> Driver compile may causing memory leak
> --------------------------------------
>
> Key: HIVE-27854
> URL: https://issues.apache.org/jira/browse/HIVE-27854
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Affects Versions: 3.1.2, 4.0.0-beta-1
> Reporter: chenruotao
> Assignee: chenruotao
> Priority: Major
> Attachments: image-2023-11-06-18-11-47-832.png,
> image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png,
> image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png,
> image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png
>
>
> when submit a sql to hiveserver, if the sql query timeout before compile, it
> will continue compile and execute addShutdownHook, but now no thread to call
> the close method of driver, so it alway stay in ShutdownHookManager's hooks,
> causing a memory leak
> !image-2023-11-06-18-11-47-832.png!
> the cancel from SQLOperation will set it's driver = null, so it can not call
> the close method of driver
> !image-2023-11-06-18-13-43-182.png!
> !image-2023-11-06-18-13-00-842.png!
> it will not trigger everytime, but we could do this to trigger it
> 1、sleep 5s before compile
> !image-2023-11-06-18-21-19-056.png!
> 2、set hive.query.timeout.seconds to 4
> !image-2023-11-06-18-22-45-812.png!
> 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one
> !image-2023-11-06-18-24-41-982.png|width=842,height=134!
> i solve it in hive 3.1.2 by add this code :
> !image-2023-11-06-19-30-14-120.png!
> and obviously we see no more memory leak
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
