[
https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenruotao updated HIVE-27854:
------------------------------
Description:
when submit a sql to hiveserver, if the sql query timeout before compile, it
will continue compile and execute the addShutdownHook method, but now no thread
to call the close method of driver, so it alway stay in ShutdownHookManager's
hooks, causing a memory leak
!image-2023-11-06-18-11-47-832.png|width=943,height=241!
!image-2023-11-06-19-45-03-388.png!
the cancel from SQLOperation will set it's driver = null, so it can not call
the close method of driver after SQLOperation exexute cancel
!image-2023-11-06-18-13-43-182.png!
!image-2023-11-06-18-13-00-842.png!
and now the driverState is INITIALIZED, so the lDrvState can not set abort to
interrupted the
!image-2023-11-06-20-26-07-406.png|width=479,height=365!
!image-2023-11-06-20-28-08-239.png|width=731,height=97!
it will not trigger everytime, but we could do this to trigger it
1、sleep 5s before compile
!image-2023-11-06-18-21-19-056.png|width=763,height=257!
2、set hive.query.timeout.seconds to 4
!image-2023-11-06-18-22-45-812.png!
3、now everytime I submit sql , the shutdownHookManager$HookEntry will add one
!image-2023-11-06-18-24-41-982.png|width=842,height=134!
i solve it in hive 3.1.2 by adding this code :
!image-2023-11-06-19-30-14-120.png!
Obviously we see no more memory leak, and shutdownHookManager$HookEntry num
will not increase every time i submit sql
!image-2023-11-06-19-40-49-488.png|width=956,height=140!
was:
when submit a sql to hiveserver, if the sql query timeout before compile, it
will continue compile and execute the addShutdownHook method, but now no thread
to call the close method of driver, so it alway stay in ShutdownHookManager's
hooks, causing a memory leak
!image-2023-11-06-18-11-47-832.png|width=943,height=241!
!image-2023-11-06-19-45-03-388.png!
the cancel from SQLOperation will set it's driver = null, so it can not call
the close method of driver after SQLOperation exexute cancel
!image-2023-11-06-18-13-43-182.png!
!image-2023-11-06-18-13-00-842.png!
it will not trigger everytime, but we could do this to trigger it
1、sleep 5s before compile
!image-2023-11-06-18-21-19-056.png!
2、set hive.query.timeout.seconds to 4
!image-2023-11-06-18-22-45-812.png!
3、now everytime I submit sql , the shutdownHookManager$HookEntry will add one
!image-2023-11-06-18-24-41-982.png|width=842,height=134!
i solve it in hive 3.1.2 by adding this code :
!image-2023-11-06-19-30-14-120.png!
Obviously we see no more memory leak, and shutdownHookManager$HookEntry num
will not increase every time i submit sql
!image-2023-11-06-19-40-49-488.png|width=956,height=140!
> Driver compile may causing memory leak
> --------------------------------------
>
> Key: HIVE-27854
> URL: https://issues.apache.org/jira/browse/HIVE-27854
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Affects Versions: 3.1.2, 4.0.0-beta-1
> Reporter: chenruotao
> Assignee: chenruotao
> Priority: Major
> Labels: pull-request-available
> Attachments: image-2023-11-06-18-11-47-832.png,
> image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png,
> image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png,
> image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png,
> image-2023-11-06-19-40-49-488.png, image-2023-11-06-19-45-03-388.png,
> image-2023-11-06-20-26-07-406.png, image-2023-11-06-20-28-08-239.png
>
>
> when submit a sql to hiveserver, if the sql query timeout before compile, it
> will continue compile and execute the addShutdownHook method, but now no
> thread to call the close method of driver, so it alway stay in
> ShutdownHookManager's hooks, causing a memory leak
> !image-2023-11-06-18-11-47-832.png|width=943,height=241!
> !image-2023-11-06-19-45-03-388.png!
> the cancel from SQLOperation will set it's driver = null, so it can not call
> the close method of driver after SQLOperation exexute cancel
> !image-2023-11-06-18-13-43-182.png!
> !image-2023-11-06-18-13-00-842.png!
> and now the driverState is INITIALIZED, so the lDrvState can not set abort to
> interrupted the
> !image-2023-11-06-20-26-07-406.png|width=479,height=365!
> !image-2023-11-06-20-28-08-239.png|width=731,height=97!
> it will not trigger everytime, but we could do this to trigger it
> 1、sleep 5s before compile
> !image-2023-11-06-18-21-19-056.png|width=763,height=257!
> 2、set hive.query.timeout.seconds to 4
> !image-2023-11-06-18-22-45-812.png!
> 3、now everytime I submit sql , the shutdownHookManager$HookEntry will add one
> !image-2023-11-06-18-24-41-982.png|width=842,height=134!
> i solve it in hive 3.1.2 by adding this code :
> !image-2023-11-06-19-30-14-120.png!
> Obviously we see no more memory leak, and shutdownHookManager$HookEntry num
> will not increase every time i submit sql
> !image-2023-11-06-19-40-49-488.png|width=956,height=140!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
