[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Description: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute the addShutdownHook method, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! !image-2023-11-06-19-45-03-388.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver after SQLOperation exexute cancel !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! and now the driverState is INITIALIZED, so the lDrvState can not set abort to interrupted the compile !image-2023-11-06-20-26-07-406.png|width=479,height=365! !image-2023-11-06-20-28-08-239.png|width=731,height=97! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png|width=763,height=257! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the shutdownHookManager$HookEntry will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by adding this code : !image-2023-11-06-19-30-14-120.png! Obviously we see no more memory leak, and shutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! was: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute the addShutdownHook method, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! !image-2023-11-06-19-45-03-388.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver after SQLOperation exexute cancel !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! and now the driverState is INITIALIZED, so the lDrvState can not set abort to interrupted the !image-2023-11-06-20-26-07-406.png|width=479,height=365! !image-2023-11-06-20-28-08-239.png|width=731,height=97! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png|width=763,height=257! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the shutdownHookManager$HookEntry will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by adding this code : !image-2023-11-06-19-30-14-120.png! Obviously we see no more memory leak, and shutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png, image-2023-11-06-19-45-03-388.png, > image-2023-11-06-20-26-07-406.png, image-2023-11-06-20-28-08-239.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute the addShutdownHook method, but now no > thread to call the close method of driver, so it alway stay in > ShutdownHookManager's hooks, causing a memory leak > !image-2023-11-06-18-11-47-832.png|width=943,height=241! > !image-2023-11-06-19-45-03-388.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver after SQLOperation exexute cancel > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > and now the driverState is INITIALIZED, so the lDrvState can not set abort to > interrupted the compile > !image-2023-11-06-20-26-07-406.png|width=479,height=365! > !image-2023-11-06-20-28-08-239.png|width=731,height=97! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png|width=763,height=257! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png!
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Attachment: image-2023-11-06-20-28-08-239.png > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png, image-2023-11-06-19-45-03-388.png, > image-2023-11-06-20-26-07-406.png, image-2023-11-06-20-28-08-239.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute the addShutdownHook method, but now no > thread to call the close method of driver, so it alway stay in > ShutdownHookManager's hooks, causing a memory leak > !image-2023-11-06-18-11-47-832.png|width=943,height=241! > !image-2023-11-06-19-45-03-388.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver after SQLOperation exexute cancel > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the shutdownHookManager$HookEntry will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by adding this code : > !image-2023-11-06-19-30-14-120.png! > Obviously we see no more memory leak, and shutdownHookManager$HookEntry num > will not increase every time i submit sql > !image-2023-11-06-19-40-49-488.png|width=956,height=140! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Description: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute the addShutdownHook method, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! !image-2023-11-06-19-45-03-388.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver after SQLOperation exexute cancel !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! and now the driverState is INITIALIZED, so the lDrvState can not set abort to interrupted the !image-2023-11-06-20-26-07-406.png|width=479,height=365! !image-2023-11-06-20-28-08-239.png|width=731,height=97! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png|width=763,height=257! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the shutdownHookManager$HookEntry will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by adding this code : !image-2023-11-06-19-30-14-120.png! Obviously we see no more memory leak, and shutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! was: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute the addShutdownHook method, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! !image-2023-11-06-19-45-03-388.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver after SQLOperation exexute cancel !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the shutdownHookManager$HookEntry will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by adding this code : !image-2023-11-06-19-30-14-120.png! Obviously we see no more memory leak, and shutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png, image-2023-11-06-19-45-03-388.png, > image-2023-11-06-20-26-07-406.png, image-2023-11-06-20-28-08-239.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute the addShutdownHook method, but now no > thread to call the close method of driver, so it alway stay in > ShutdownHookManager's hooks, causing a memory leak > !image-2023-11-06-18-11-47-832.png|width=943,height=241! > !image-2023-11-06-19-45-03-388.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver after SQLOperation exexute cancel > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > and now the driverState is INITIALIZED, so the lDrvState can not set abort to > interrupted the > !image-2023-11-06-20-26-07-406.png|width=479,height=365! > !image-2023-11-06-20-28-08-239.png|width=731,height=97! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png|width=763,height=257! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the shutdownHookManager$HookEntry will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by adding this code : > !image-2023-11-06-19-30-14-120.png! > Obviously we see
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Attachment: image-2023-11-06-20-26-07-406.png > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png, image-2023-11-06-19-45-03-388.png, > image-2023-11-06-20-26-07-406.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute the addShutdownHook method, but now no > thread to call the close method of driver, so it alway stay in > ShutdownHookManager's hooks, causing a memory leak > !image-2023-11-06-18-11-47-832.png|width=943,height=241! > !image-2023-11-06-19-45-03-388.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver after SQLOperation exexute cancel > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the shutdownHookManager$HookEntry will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by adding this code : > !image-2023-11-06-19-30-14-120.png! > Obviously we see no more memory leak, and shutdownHookManager$HookEntry num > will not increase every time i submit sql > !image-2023-11-06-19-40-49-488.png|width=956,height=140! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-27854 started by chenruotao. - > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png, image-2023-11-06-19-45-03-388.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute the addShutdownHook method, but now no > thread to call the close method of driver, so it alway stay in > ShutdownHookManager's hooks, causing a memory leak > !image-2023-11-06-18-11-47-832.png|width=943,height=241! > !image-2023-11-06-19-45-03-388.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver after SQLOperation exexute cancel > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the shutdownHookManager$HookEntry will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by adding this code : > !image-2023-11-06-19-30-14-120.png! > Obviously we see no more memory leak, and shutdownHookManager$HookEntry num > will not increase every time i submit sql > !image-2023-11-06-19-40-49-488.png|width=956,height=140! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Description: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute the addShutdownHook method, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! !image-2023-11-06-19-45-03-388.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver after SQLOperation exexute cancel !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the shutdownHookManager$HookEntry will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by adding this code : !image-2023-11-06-19-30-14-120.png! Obviously we see no more memory leak, and shutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! was: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute the addShutdownHook method, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! !image-2023-11-06-19-45-03-388.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver after SQLOperation exexute cancel !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by adding this code : !image-2023-11-06-19-30-14-120.png! Obviously we see no more memory leak, and shutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png, image-2023-11-06-19-45-03-388.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute the addShutdownHook method, but now no > thread to call the close method of driver, so it alway stay in > ShutdownHookManager's hooks, causing a memory leak > !image-2023-11-06-18-11-47-832.png|width=943,height=241! > !image-2023-11-06-19-45-03-388.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver after SQLOperation exexute cancel > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the shutdownHookManager$HookEntry will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by adding this code : > !image-2023-11-06-19-30-14-120.png! > Obviously we see no more memory leak, and shutdownHookManager$HookEntry num > will not increase every time i submit sql > !image-2023-11-06-19-40-49-488.png|width=956,height=140! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Description: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute the addShutdownHook method, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! !image-2023-11-06-19-45-03-388.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver after SQLOperation exexute cancel !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by adding this code : !image-2023-11-06-19-30-14-120.png! Obviously we see no more memory leak, and shutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! was: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute the addShutdownHook method, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! !image-2023-11-06-19-45-03-388.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver after SQLOperation exexute cancel !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by add this code : !image-2023-11-06-19-30-14-120.png! and obviously we see no more memory leak, ShutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png, image-2023-11-06-19-45-03-388.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute the addShutdownHook method, but now no > thread to call the close method of driver, so it alway stay in > ShutdownHookManager's hooks, causing a memory leak > !image-2023-11-06-18-11-47-832.png|width=943,height=241! > !image-2023-11-06-19-45-03-388.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver after SQLOperation exexute cancel > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by adding this code : > !image-2023-11-06-19-30-14-120.png! > Obviously we see no more memory leak, and shutdownHookManager$HookEntry num > will not increase every time i submit sql > !image-2023-11-06-19-40-49-488.png|width=956,height=140! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Description: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute the addShutdownHook method, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! !image-2023-11-06-19-45-03-388.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver after SQLOperation exexute cancel !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by add this code : !image-2023-11-06-19-30-14-120.png! and obviously we see no more memory leak, ShutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! was: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute the addShutdownHook method, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! !image-2023-11-06-19-45-03-388.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by add this code : !image-2023-11-06-19-30-14-120.png! and obviously we see no more memory leak, ShutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png, image-2023-11-06-19-45-03-388.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute the addShutdownHook method, but now no > thread to call the close method of driver, so it alway stay in > ShutdownHookManager's hooks, causing a memory leak > !image-2023-11-06-18-11-47-832.png|width=943,height=241! > !image-2023-11-06-19-45-03-388.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver after SQLOperation exexute cancel > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by add this code : > !image-2023-11-06-19-30-14-120.png! > and obviously we see no more memory leak, ShutdownHookManager$HookEntry num > will not increase every time i submit sql > !image-2023-11-06-19-40-49-488.png|width=956,height=140! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Attachment: image-2023-11-06-19-45-03-388.png > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png, image-2023-11-06-19-45-03-388.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute addShutdownHook, but now no thread to call > the close method of driver, so it alway stay in ShutdownHookManager's hooks, > causing a memory leak > !image-2023-11-06-18-11-47-832.png|width=943,height=241! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by add this code : > !image-2023-11-06-19-30-14-120.png! > and obviously we see no more memory leak, ShutdownHookManager$HookEntry num > will not increase every time i submit sql > !image-2023-11-06-19-40-49-488.png|width=956,height=140! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Description: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute the addShutdownHook method, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! !image-2023-11-06-19-45-03-388.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by add this code : !image-2023-11-06-19-30-14-120.png! and obviously we see no more memory leak, ShutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! was: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute addShutdownHook, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by add this code : !image-2023-11-06-19-30-14-120.png! and obviously we see no more memory leak, ShutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png, image-2023-11-06-19-45-03-388.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute the addShutdownHook method, but now no > thread to call the close method of driver, so it alway stay in > ShutdownHookManager's hooks, causing a memory leak > !image-2023-11-06-18-11-47-832.png|width=943,height=241! > !image-2023-11-06-19-45-03-388.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by add this code : > !image-2023-11-06-19-30-14-120.png! > and obviously we see no more memory leak, ShutdownHookManager$HookEntry num > will not increase every time i submit sql > !image-2023-11-06-19-40-49-488.png|width=956,height=140! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Description: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute addShutdownHook, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by add this code : !image-2023-11-06-19-30-14-120.png! and obviously we see no more memory leak, ShutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! was: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute addShutdownHook, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by add this code : !image-2023-11-06-19-30-14-120.png! and obviously we see no more memory leak, ShutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png! > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute addShutdownHook, but now no thread to call > the close method of driver, so it alway stay in ShutdownHookManager's hooks, > causing a memory leak > !image-2023-11-06-18-11-47-832.png|width=943,height=241! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by add this code : > !image-2023-11-06-19-30-14-120.png! > and obviously we see no more memory leak, ShutdownHookManager$HookEntry num > will not increase every time i submit sql > !image-2023-11-06-19-40-49-488.png|width=956,height=140! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Description: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute addShutdownHook, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by add this code : !image-2023-11-06-19-30-14-120.png! and obviously we see no more memory leak, ShutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png! was: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute addShutdownHook, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by add this code : !image-2023-11-06-19-30-14-120.png! and obviously we see no more memory leak > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute addShutdownHook, but now no thread to call > the close method of driver, so it alway stay in ShutdownHookManager's hooks, > causing a memory leak > !image-2023-11-06-18-11-47-832.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by add this code : > !image-2023-11-06-19-30-14-120.png! > and obviously we see no more memory leak, ShutdownHookManager$HookEntry num > will not increase every time i submit sql > !image-2023-11-06-19-40-49-488.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Attachment: image-2023-11-06-19-40-49-488.png > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute addShutdownHook, but now no thread to call > the close method of driver, so it alway stay in ShutdownHookManager's hooks, > causing a memory leak > !image-2023-11-06-18-11-47-832.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by add this code : > !image-2023-11-06-19-30-14-120.png! > and obviously we see no more memory leak -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Description: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute addShutdownHook, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by add this code : !image-2023-11-06-19-30-14-120.png! and obviously we see no more memory leak was: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute addShutdownHook, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute addShutdownHook, but now no thread to call > the close method of driver, so it alway stay in ShutdownHookManager's hooks, > causing a memory leak > !image-2023-11-06-18-11-47-832.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by add this code : > !image-2023-11-06-19-30-14-120.png! > and obviously we see no more memory leak -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Attachment: image-2023-11-06-19-30-14-120.png > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute addShutdownHook, but now no thread to call > the close method of driver, so it alway stay in ShutdownHookManager's hooks, > causing a memory leak > !image-2023-11-06-18-11-47-832.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Description: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute addShutdownHook, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! was: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute addShutdownHook, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute addShutdownHook, but now no thread to call > the close method of driver, so it alway stay in ShutdownHookManager's hooks, > causing a memory leak > !image-2023-11-06-18-11-47-832.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Attachment: image-2023-11-06-18-24-41-982.png > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute addShutdownHook, but now no thread to call > the close method of driver, so it alway stay in ShutdownHookManager's hooks, > causing a memory leak > !image-2023-11-06-18-11-47-832.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Attachment: image-2023-11-06-18-22-45-812.png > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute addShutdownHook, but now no thread to call > the close method of driver, so it alway stay in ShutdownHookManager's hooks, > causing a memory leak > !image-2023-11-06-18-11-47-832.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Attachment: image-2023-11-06-18-21-19-056.png > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute addShutdownHook, but now no thread to call > the close method of driver, so it alway stay in ShutdownHookManager's hooks, > causing a memory leak > !image-2023-11-06-18-11-47-832.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Attachment: image-2023-11-06-18-13-43-182.png image-2023-11-06-18-13-00-842.png image-2023-11-06-18-11-47-832.png Description: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute addShutdownHook, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! was:when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute addShutdownHook, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute addShutdownHook, but now no thread to call > the close method of driver, so it alway stay in ShutdownHookManager's hooks, > causing a memory leak > !image-2023-11-06-18-11-47-832.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27854) Driver compile may causing memory leak
chenruotao created HIVE-27854: - Summary: Driver compile may causing memory leak Key: HIVE-27854 URL: https://issues.apache.org/jira/browse/HIVE-27854 Project: Hive Issue Type: Bug Components: Hive Affects Versions: 4.0.0-beta-1, 3.1.2 Reporter: chenruotao Assignee: chenruotao when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute addShutdownHook, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27140) Set HADOOP_PROXY_USER cause hiveMetaStoreClient close everytime
[
https://issues.apache.org/jira/browse/HIVE-27140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenruotao updated HIVE-27140:
--
Description:
In order to use proxy user to access hive metastore server with kerberos, I set
HADOOP_PROXY_USER = proxy user.
this is the code of HiveMetaStoreClient
!image-2023-03-14-20-51-32-317.png|width=606,height=393!
but when I run a spark job with hive metatsore client, the connection to hive
metatsore server always close
!image-2023-03-14-20-53-40-132.png|width=687,height=375!
and I notice that some log before close :
{color:#ff}Mestastore configuration hive.metastore.token.signature changed
from DelegationTokenForHiveMetaStoreServer to{color}
so we can find why the connection close always, hive metatsore client will
compare currentMetaVars with hiveconf
!image-2023-03-14-20-56-00-757.png|width=903,height=463!
but if we set HADOOP_PROXY_USER and init hive metatsore client, the
ConfVars.METASTORE_TOKEN_SIGNATURE always is set
!image-2023-03-14-21-00-18-049.png|width=905,height=592!
so I think this solutions can fix the bug:
ConfVars.METASTORE_TOKEN_SIGNATURE is no need to check in hive metatsore client
!image-2023-03-14-21-03-52-764.png|width=905,height=606!
obviously that is no more close
!image-2023-03-14-21-56-27-408.png|width=903,height=452!
was:
In order to use proxy user to access hive metastore server with kerberos, I set
HADOOP_PROXY_USER = proxy user.
this is the code of HiveMetaStoreClient
!image-2023-03-14-20-51-32-317.png|width=606,height=393!
but when I run a spark job with hive metatsore client, the connection to hive
metatsore server always close
!image-2023-03-14-20-53-40-132.png|width=687,height=375!
and I notice that some log before close :
{color:#FF}Mestastore configuration hive.metastore.token.signature changed
from DelegationTokenForHiveMetaStoreServer to{color}
so we can find why the connection close always, hive metatsore client will
compare currentMetaVars with hiveconf
!image-2023-03-14-20-56-00-757.png|width=903,height=463!
but if we set HADOOP_PROXY_USER and init hive metatsore client, the
ConfVars.METASTORE_TOKEN_SIGNATURE always is set
!image-2023-03-14-21-00-18-049.png|width=905,height=592!
so I think this solutions can fix the bug:
ConfVars.METASTORE_TOKEN_SIGNATURE is no need to check in hive metatsore client
!image-2023-03-14-21-03-52-764.png|width=905,height=606!
> Set HADOOP_PROXY_USER cause hiveMetaStoreClient close everytime
>
>
> Key: HIVE-27140
> URL: https://issues.apache.org/jira/browse/HIVE-27140
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.3.8, 3.1.2
>Reporter: chenruotao
>Assignee: chenruotao
>Priority: Major
> Attachments: image-2023-03-14-20-51-32-317.png,
> image-2023-03-14-20-53-40-132.png, image-2023-03-14-20-56-00-757.png,
> image-2023-03-14-21-00-18-049.png, image-2023-03-14-21-03-52-764.png,
> image-2023-03-14-21-56-06-605.png, image-2023-03-14-21-56-27-408.png
>
>
> In order to use proxy user to access hive metastore server with kerberos, I
> set HADOOP_PROXY_USER = proxy user.
> this is the code of HiveMetaStoreClient
> !image-2023-03-14-20-51-32-317.png|width=606,height=393!
> but when I run a spark job with hive metatsore client, the connection to
> hive metatsore server always close
> !image-2023-03-14-20-53-40-132.png|width=687,height=375!
> and I notice that some log before close :
> {color:#ff}Mestastore configuration hive.metastore.token.signature
> changed from DelegationTokenForHiveMetaStoreServer to{color}
> so we can find why the connection close always, hive metatsore client will
> compare currentMetaVars with hiveconf
> !image-2023-03-14-20-56-00-757.png|width=903,height=463!
> but if we set HADOOP_PROXY_USER and init hive metatsore client, the
> ConfVars.METASTORE_TOKEN_SIGNATURE always is set
> !image-2023-03-14-21-00-18-049.png|width=905,height=592!
> so I think this solutions can fix the bug:
> ConfVars.METASTORE_TOKEN_SIGNATURE is no need to check in hive metatsore
> client
> !image-2023-03-14-21-03-52-764.png|width=905,height=606!
> obviously that is no more close
> !image-2023-03-14-21-56-27-408.png|width=903,height=452!
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27140) Set HADOOP_PROXY_USER cause hiveMetaStoreClient close everytime
[
https://issues.apache.org/jira/browse/HIVE-27140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenruotao updated HIVE-27140:
--
Attachment: image-2023-03-14-21-56-06-605.png
> Set HADOOP_PROXY_USER cause hiveMetaStoreClient close everytime
>
>
> Key: HIVE-27140
> URL: https://issues.apache.org/jira/browse/HIVE-27140
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.3.8, 3.1.2
>Reporter: chenruotao
>Assignee: chenruotao
>Priority: Major
> Attachments: image-2023-03-14-20-51-32-317.png,
> image-2023-03-14-20-53-40-132.png, image-2023-03-14-20-56-00-757.png,
> image-2023-03-14-21-00-18-049.png, image-2023-03-14-21-03-52-764.png,
> image-2023-03-14-21-56-06-605.png, image-2023-03-14-21-56-27-408.png
>
>
> In order to use proxy user to access hive metastore server with kerberos, I
> set HADOOP_PROXY_USER = proxy user.
> this is the code of HiveMetaStoreClient
> !image-2023-03-14-20-51-32-317.png|width=606,height=393!
> but when I run a spark job with hive metatsore client, the connection to
> hive metatsore server always close
> !image-2023-03-14-20-53-40-132.png|width=687,height=375!
> and I notice that some log before close :
> {color:#FF}Mestastore configuration hive.metastore.token.signature
> changed from DelegationTokenForHiveMetaStoreServer to{color}
> so we can find why the connection close always, hive metatsore client will
> compare currentMetaVars with hiveconf
> !image-2023-03-14-20-56-00-757.png|width=903,height=463!
> but if we set HADOOP_PROXY_USER and init hive metatsore client, the
> ConfVars.METASTORE_TOKEN_SIGNATURE always is set
> !image-2023-03-14-21-00-18-049.png|width=905,height=592!
> so I think this solutions can fix the bug:
> ConfVars.METASTORE_TOKEN_SIGNATURE is no need to check in hive metatsore
> client
> !image-2023-03-14-21-03-52-764.png|width=905,height=606!
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27140) Set HADOOP_PROXY_USER cause hiveMetaStoreClient close everytime
[
https://issues.apache.org/jira/browse/HIVE-27140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenruotao updated HIVE-27140:
--
Attachment: image-2023-03-14-21-56-27-408.png
> Set HADOOP_PROXY_USER cause hiveMetaStoreClient close everytime
>
>
> Key: HIVE-27140
> URL: https://issues.apache.org/jira/browse/HIVE-27140
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.3.8, 3.1.2
>Reporter: chenruotao
>Assignee: chenruotao
>Priority: Major
> Attachments: image-2023-03-14-20-51-32-317.png,
> image-2023-03-14-20-53-40-132.png, image-2023-03-14-20-56-00-757.png,
> image-2023-03-14-21-00-18-049.png, image-2023-03-14-21-03-52-764.png,
> image-2023-03-14-21-56-06-605.png, image-2023-03-14-21-56-27-408.png
>
>
> In order to use proxy user to access hive metastore server with kerberos, I
> set HADOOP_PROXY_USER = proxy user.
> this is the code of HiveMetaStoreClient
> !image-2023-03-14-20-51-32-317.png|width=606,height=393!
> but when I run a spark job with hive metatsore client, the connection to
> hive metatsore server always close
> !image-2023-03-14-20-53-40-132.png|width=687,height=375!
> and I notice that some log before close :
> {color:#FF}Mestastore configuration hive.metastore.token.signature
> changed from DelegationTokenForHiveMetaStoreServer to{color}
> so we can find why the connection close always, hive metatsore client will
> compare currentMetaVars with hiveconf
> !image-2023-03-14-20-56-00-757.png|width=903,height=463!
> but if we set HADOOP_PROXY_USER and init hive metatsore client, the
> ConfVars.METASTORE_TOKEN_SIGNATURE always is set
> !image-2023-03-14-21-00-18-049.png|width=905,height=592!
> so I think this solutions can fix the bug:
> ConfVars.METASTORE_TOKEN_SIGNATURE is no need to check in hive metatsore
> client
> !image-2023-03-14-21-03-52-764.png|width=905,height=606!
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27140) Set HADOOP_PROXY_USER cause hiveMetaStoreClient close everytime
[
https://issues.apache.org/jira/browse/HIVE-27140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenruotao updated HIVE-27140:
--
Description:
In order to use proxy user to access hive metastore server with kerberos, I set
HADOOP_PROXY_USER = proxy user.
this is the code of HiveMetaStoreClient
!image-2023-03-14-20-51-32-317.png|width=606,height=393!
but when I run a spark job with hive metatsore client, the connection to hive
metatsore server always close
!image-2023-03-14-20-53-40-132.png|width=687,height=375!
and I notice that some log before close :
{color:#FF}Mestastore configuration hive.metastore.token.signature changed
from DelegationTokenForHiveMetaStoreServer to{color}
so we can find why the connection close always, hive metatsore client will
compare currentMetaVars with hiveconf
!image-2023-03-14-20-56-00-757.png|width=903,height=463!
but if we set HADOOP_PROXY_USER and init hive metatsore client, the
ConfVars.METASTORE_TOKEN_SIGNATURE always is set
!image-2023-03-14-21-00-18-049.png|width=905,height=592!
so I think this solutions can fix the bug:
ConfVars.METASTORE_TOKEN_SIGNATURE is no need to check in hive metatsore client
!image-2023-03-14-21-03-52-764.png|width=905,height=606!
was:In order to use proxy user to access hive metastore server with kerberos,
I set
> Set HADOOP_PROXY_USER cause hiveMetaStoreClient close everytime
>
>
> Key: HIVE-27140
> URL: https://issues.apache.org/jira/browse/HIVE-27140
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.3.8, 3.1.2
>Reporter: chenruotao
>Assignee: chenruotao
>Priority: Major
> Attachments: image-2023-03-14-20-51-32-317.png,
> image-2023-03-14-20-53-40-132.png, image-2023-03-14-20-56-00-757.png,
> image-2023-03-14-21-00-18-049.png, image-2023-03-14-21-03-52-764.png
>
>
> In order to use proxy user to access hive metastore server with kerberos, I
> set HADOOP_PROXY_USER = proxy user.
> this is the code of HiveMetaStoreClient
> !image-2023-03-14-20-51-32-317.png|width=606,height=393!
> but when I run a spark job with hive metatsore client, the connection to
> hive metatsore server always close
> !image-2023-03-14-20-53-40-132.png|width=687,height=375!
> and I notice that some log before close :
> {color:#FF}Mestastore configuration hive.metastore.token.signature
> changed from DelegationTokenForHiveMetaStoreServer to{color}
> so we can find why the connection close always, hive metatsore client will
> compare currentMetaVars with hiveconf
> !image-2023-03-14-20-56-00-757.png|width=903,height=463!
> but if we set HADOOP_PROXY_USER and init hive metatsore client, the
> ConfVars.METASTORE_TOKEN_SIGNATURE always is set
> !image-2023-03-14-21-00-18-049.png|width=905,height=592!
> so I think this solutions can fix the bug:
> ConfVars.METASTORE_TOKEN_SIGNATURE is no need to check in hive metatsore
> client
> !image-2023-03-14-21-03-52-764.png|width=905,height=606!
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27140) Set HADOOP_PROXY_USER cause hiveMetaStoreClient close everytime
[ https://issues.apache.org/jira/browse/HIVE-27140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27140: -- Attachment: image-2023-03-14-21-03-52-764.png > Set HADOOP_PROXY_USER cause hiveMetaStoreClient close everytime > > > Key: HIVE-27140 > URL: https://issues.apache.org/jira/browse/HIVE-27140 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.3.8, 3.1.2 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Attachments: image-2023-03-14-20-51-32-317.png, > image-2023-03-14-20-53-40-132.png, image-2023-03-14-20-56-00-757.png, > image-2023-03-14-21-00-18-049.png, image-2023-03-14-21-03-52-764.png > > > In order to use proxy user to access hive metastore server with kerberos, I > set -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27140) Set HADOOP_PROXY_USER cause hiveMetaStoreClient close everytime
[ https://issues.apache.org/jira/browse/HIVE-27140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27140: -- Attachment: image-2023-03-14-21-00-18-049.png > Set HADOOP_PROXY_USER cause hiveMetaStoreClient close everytime > > > Key: HIVE-27140 > URL: https://issues.apache.org/jira/browse/HIVE-27140 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.3.8, 3.1.2 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Attachments: image-2023-03-14-20-51-32-317.png, > image-2023-03-14-20-53-40-132.png, image-2023-03-14-20-56-00-757.png, > image-2023-03-14-21-00-18-049.png > > > In order to use proxy user to access hive metastore server with kerberos, I > set -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27140) Set HADOOP_PROXY_USER cause hiveMetaStoreClient close everytime
[ https://issues.apache.org/jira/browse/HIVE-27140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27140: -- Attachment: image-2023-03-14-20-56-00-757.png > Set HADOOP_PROXY_USER cause hiveMetaStoreClient close everytime > > > Key: HIVE-27140 > URL: https://issues.apache.org/jira/browse/HIVE-27140 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.3.8, 3.1.2 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Attachments: image-2023-03-14-20-51-32-317.png, > image-2023-03-14-20-53-40-132.png, image-2023-03-14-20-56-00-757.png > > > In order to use proxy user to access hive metastore server with kerberos, I > set -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27140) Set HADOOP_PROXY_USER cause hiveMetaStoreClient close everytime
[ https://issues.apache.org/jira/browse/HIVE-27140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27140: -- Attachment: image-2023-03-14-20-53-40-132.png > Set HADOOP_PROXY_USER cause hiveMetaStoreClient close everytime > > > Key: HIVE-27140 > URL: https://issues.apache.org/jira/browse/HIVE-27140 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.3.8, 3.1.2 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Attachments: image-2023-03-14-20-51-32-317.png, > image-2023-03-14-20-53-40-132.png > > > In order to use proxy user to access hive metastore server with kerberos, I > set -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27140) Set HADOOP_PROXY_USER cause hiveMetaStoreClient close everytime
[ https://issues.apache.org/jira/browse/HIVE-27140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27140: -- Attachment: image-2023-03-14-20-51-32-317.png > Set HADOOP_PROXY_USER cause hiveMetaStoreClient close everytime > > > Key: HIVE-27140 > URL: https://issues.apache.org/jira/browse/HIVE-27140 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.3.8, 3.1.2 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Attachments: image-2023-03-14-20-51-32-317.png > > > In order to use proxy user to access hive metastore server with kerberos, I > set -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27140) Set HADOOP_PROXY_USER cause hiveMetaStoreClient close everytime
[ https://issues.apache.org/jira/browse/HIVE-27140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27140: -- Description: In order to use proxy user to access hive metastore server with kerberos, I set > Set HADOOP_PROXY_USER cause hiveMetaStoreClient close everytime > > > Key: HIVE-27140 > URL: https://issues.apache.org/jira/browse/HIVE-27140 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.3.8, 3.1.2 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > > In order to use proxy user to access hive metastore server with kerberos, I > set -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-27140) Set HADOOP_PROXY_USER cause hiveMetaStoreClient close everytime
[ https://issues.apache.org/jira/browse/HIVE-27140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao reassigned HIVE-27140: - > Set HADOOP_PROXY_USER cause hiveMetaStoreClient close everytime > > > Key: HIVE-27140 > URL: https://issues.apache.org/jira/browse/HIVE-27140 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.1.2, 2.3.8 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-25641) Hive多分区表添加字段提示Partition not found
[ https://issues.apache.org/jira/browse/HIVE-25641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17499949#comment-17499949 ] chenruotao commented on HIVE-25641: --- 打开debug看详细的日志 > Hive多分区表添加字段提示Partition not found > - > > Key: HIVE-25641 > URL: https://issues.apache.org/jira/browse/HIVE-25641 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.0.1 >Reporter: zhusijie >Priority: Major > Attachments: image-2021-10-25-17-32-32-207.png > > > 执行 ALTER TABLE cf_rds.cf_rds_jxd_clue_basic_di ADD COLUMNS (channel bigint > COMMENT '渠道号') CASCADE; > 提示：Partition not found > !image-2021-10-25-17-32-32-207.png! -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25992) Can not init HiveMetaStoreClient when using proxy user
[
https://issues.apache.org/jira/browse/HIVE-25992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenruotao updated HIVE-25992:
--
Description:
when I use this code to access hms
{code:java}
UserGroupInformation proxyUser =
UserGroupInformation.createProxyUser(proxyUserName, userGroupInformation);
proxyUser.doAs((PrivilegedExceptionAction) () -> {
XX
return true;
});
{code}
there will throw exception
!image-2022-03-01-16-33-49-012.png|width=868,height=346! so I debug with this,
and find something maybe wrong in the class named HiveMetaStoreClient:
!image-2022-03-01-16-36-52-539.png!
because the value of UserGroupInformation.getLoginUser().getRealUser() that in
proxy.doas is null, so i change to UserGroupInformation.getLoginUser() and i
works
!image-2022-03-01-16-39-57-935.png!
so what I did is no problem?
was:
when I use this code to access hms
{code:java}
UserGroupInformation proxyUser =
UserGroupInformation.createProxyUser(proxyUserName, userGroupInformation);
proxyUser.doAs((PrivilegedExceptionAction) () -> {
XX
return true;
});
{code}
there will throw exception
!image-2022-03-01-16-33-49-012.png! so I debug with this, and find something
maybe wrong in the class named HiveMetaStoreClient:
!image-2022-03-01-16-36-52-539.png!
because the value of UserGroupInformation.getLoginUser().getRealUser() that in
proxy.doas is null, so i change to UserGroupInformation.getLoginUser() and i
works
!image-2022-03-01-16-39-57-935.png!
so what I did is no problem?
> Can not init HiveMetaStoreClient when using proxy user
> --
>
> Key: HIVE-25992
> URL: https://issues.apache.org/jira/browse/HIVE-25992
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.3.9
>Reporter: chenruotao
>Priority: Major
> Attachments: image-2022-03-01-16-33-49-012.png,
> image-2022-03-01-16-36-52-539.png, image-2022-03-01-16-39-57-935.png
>
>
> when I use this code to access hms
> {code:java}
> UserGroupInformation proxyUser =
> UserGroupInformation.createProxyUser(proxyUserName, userGroupInformation);
> proxyUser.doAs((PrivilegedExceptionAction) () -> {
> XX
> return true;
> });
> {code}
> there will throw exception
> !image-2022-03-01-16-33-49-012.png|width=868,height=346! so I debug with
> this, and find something maybe wrong in the class named HiveMetaStoreClient:
> !image-2022-03-01-16-36-52-539.png!
> because the value of UserGroupInformation.getLoginUser().getRealUser() that
> in proxy.doas is null, so i change to UserGroupInformation.getLoginUser() and
> i works
> !image-2022-03-01-16-39-57-935.png!
> so what I did is no problem?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25992) Can not init HiveMetaStoreClient when using proxy user
[
https://issues.apache.org/jira/browse/HIVE-25992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenruotao updated HIVE-25992:
--
Description:
when I use this code to access hms
{code:java}
UserGroupInformation proxyUser =
UserGroupInformation.createProxyUser(proxyUserName, userGroupInformation);
proxyUser.doAs((PrivilegedExceptionAction) () -> {
XX
return true;
});
{code}
there will throw exception
!image-2022-03-01-16-33-49-012.png! so I debug with this, and find something
maybe wrong in the class named HiveMetaStoreClient:
!image-2022-03-01-16-36-52-539.png!
because the value of UserGroupInformation.getLoginUser().getRealUser() that in
proxy.doas is null, so i change to UserGroupInformation.getLoginUser() and i
works
!image-2022-03-01-16-39-57-935.png!
so what I did is no problem?
was:
when I use this code to access hms
{code:java}
UserGroupInformation proxyUser =
UserGroupInformation.createProxyUser(proxyUserName, userGroupInformation);
proxyUser.doAs((PrivilegedExceptionAction) () -> {
XX
return true;
});
{code}
there will throw exception
!image-2022-03-01-16-33-49-012.png!
so I debug with this, and find something maybe wrong in the class named
HiveMetaStoreClient:
!image-2022-03-01-16-36-52-539.png!
because the value of UserGroupInformation.getLoginUser().getRealUser() that in
proxy.doas is null, so i change to UserGroupInformation.getLoginUser() and i
works
!image-2022-03-01-16-39-57-935.png!
so what I did is no problem?
> Can not init HiveMetaStoreClient when using proxy user
> --
>
> Key: HIVE-25992
> URL: https://issues.apache.org/jira/browse/HIVE-25992
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.3.9
>Reporter: chenruotao
>Priority: Major
> Attachments: image-2022-03-01-16-33-49-012.png,
> image-2022-03-01-16-36-52-539.png, image-2022-03-01-16-39-57-935.png
>
>
> when I use this code to access hms
> {code:java}
> UserGroupInformation proxyUser =
> UserGroupInformation.createProxyUser(proxyUserName, userGroupInformation);
> proxyUser.doAs((PrivilegedExceptionAction) () -> {
> XX
> return true;
> });
> {code}
> there will throw exception
> !image-2022-03-01-16-33-49-012.png! so I debug with this, and find something
> maybe wrong in the class named HiveMetaStoreClient:
> !image-2022-03-01-16-36-52-539.png!
> because the value of UserGroupInformation.getLoginUser().getRealUser() that
> in proxy.doas is null, so i change to UserGroupInformation.getLoginUser() and
> i works
> !image-2022-03-01-16-39-57-935.png!
> so what I did is no problem?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-19981) Managed tables converted to external tables by the HiveStrictManagedMigration utility should be set to delete data when the table is dropped
[ https://issues.apache.org/jira/browse/HIVE-19981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17255363#comment-17255363 ] chenruotao commented on HIVE-19981: --- I have one problem about this with hive 2.3.4，by using hive jdbc handler，i found that though the ext table of jdbc handler has been droped, the location of table in HDFS extis. As time goes by, there are so many empty directory, can I fix the problem using this patch? > Managed tables converted to external tables by the HiveStrictManagedMigration > utility should be set to delete data when the table is dropped > > > Key: HIVE-19981 > URL: https://issues.apache.org/jira/browse/HIVE-19981 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-19981.1.patch > > > Using the HiveStrictManagedMigration utility, tables can be converted to > conform to the Hive strict managed tables mode. > For managed tables that are converted to external tables by the utility, > these tables should keep the "drop data on delete" semantics they had when > they were managed tables. > One way to do this is to introduce a table property "external.table.purge", > which if true (and if the table is an external table), will let Hive know to > delete the table data when the table is dropped. This property will be set by > the HiveStrictManagedMigration utility when managed tables are converted to > external tables. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-21964) jdbc handler class cast exception
[
https://issues.apache.org/jira/browse/HIVE-21964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199318#comment-17199318
]
chenruotao edited comment on HIVE-21964 at 9/21/20, 10:35 AM:
--
I has the same problem like this, but not Decimal.
the type of date and timestramp would throw the same exception when job
running, so I do not use the type provided by
hive(org.apache.hadoop.hive.common.type), and it worked. the code like this :
case DATE:
// if (rowVal instanceof java.sql.Date){
// LocalDate localDate = ((java.sql.Date) rowVal).toLocalDate();
//rowVal=Date.of(localDate.getYear(),localDate.getMonthValue(),localDate.getDayOfMonth();
// }
else
{ // rowVal = Date.valueOf (rowVal.toString()); // }
rowVal = Date.valueOf (rowVal.toString());
break;
case TIMESTAMP:
// if (rowVal instanceof java.sql.Timestamp)
{ // LocalDateTime localDateTime = ((java.sql.Timestamp)
rowVal).toLocalDateTime();
//rowVal=Timestamp.ofEpochSecond(localDateTime.toEpochSecond(UTC),localDateTime.geNano());
// }
else
{ // rowVal = Timestamp.valueOf (rowVal.toString()); // }
rowVal = Timestamp.valueOf (rowVal.toString());
was (Author: chenruotao):
I has the same problem like this, but not Decimal.
the type of date and timestramp would throw the same exception when job
running, so I do not use the type provided by
hive(org.apache.hadoop.hive.common.type), and it worked. the code like this :
case DATE:
// if (rowVal instanceof java.sql.Date){
// LocalDate localDate = ((java.sql.Date) rowVal).toLocalDate();
//rowVal=Date.of(localDate.getYear(),localDate.getMonthValue(),localDate.getDayOfMonth();
// }else{
// rowVal = Date.valueOf (rowVal.toString());
// }
rowVal = Date.valueOf (rowVal.toString());
break;
case TIMESTAMP:
// if (rowVal instanceof java.sql.Timestamp){
// LocalDateTime localDateTime = ((java.sql.Timestamp)
rowVal).toLocalDateTime();
//rowVal=Timestamp.ofEpochSecond(localDateTime.toEpochSecond(UTC),localDateTime.geNano());
// }else{
// rowVal = Timestamp.valueOf (rowVal.toString());
// }
rowVal = Timestamp.valueOf (rowVal.toString());
> jdbc handler class cast exception
> -
>
> Key: HIVE-21964
> URL: https://issues.apache.org/jira/browse/HIVE-21964
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Affects Versions: 3.1.1
>Reporter: Aloys Zhang
>Priority: Major
>
> Using hive jdbc handler to query external mysql data source with type decimal
> type, it throws class cast Exception :
>
> {code:java}
> 2019-07-08T11:11:50,424 ERROR [7787918f-3111-4706-a3b3-0097fa1bc117 main]
> CliDriver: Failed with exception
> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
> org.apache.hadoop.hive.common.type.HiveDecimal
> java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
> org.apache.hadoop.hive.common.type.HiveDecimal
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:162)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2691)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
> org.apache.hadoop.hive.common.type.HiveDecimal
> at
> org.apache.hadoop.hive.ql.exec.ListSinkOperator.process(ListSinkOperator.java:98)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:995)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:941)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:995)
> at
[jira] [Comment Edited] (HIVE-21964) jdbc handler class cast exception
[
https://issues.apache.org/jira/browse/HIVE-21964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199318#comment-17199318
]
chenruotao edited comment on HIVE-21964 at 9/21/20, 10:35 AM:
--
I has the same problem like this, but not Decimal.
the type of date and timestramp would throw the same exception when job
running, so I do not use the type provided by
hive(org.apache.hadoop.hive.common.type), and it worked. the code like this :
case DATE:
rowVal = Date.valueOf (rowVal.toString());
break;
case TIMESTAMP:
rowVal = Timestamp.valueOf (rowVal.toString());
was (Author: chenruotao):
I has the same problem like this, but not Decimal.
the type of date and timestramp would throw the same exception when job
running, so I do not use the type provided by
hive(org.apache.hadoop.hive.common.type), and it worked. the code like this :
case DATE:
// if (rowVal instanceof java.sql.Date){
// LocalDate localDate = ((java.sql.Date) rowVal).toLocalDate();
//rowVal=Date.of(localDate.getYear(),localDate.getMonthValue(),localDate.getDayOfMonth();
// }
else
{ // rowVal = Date.valueOf (rowVal.toString()); // }
rowVal = Date.valueOf (rowVal.toString());
break;
case TIMESTAMP:
// if (rowVal instanceof java.sql.Timestamp)
{ // LocalDateTime localDateTime = ((java.sql.Timestamp)
rowVal).toLocalDateTime();
//rowVal=Timestamp.ofEpochSecond(localDateTime.toEpochSecond(UTC),localDateTime.geNano());
// }
else
{ // rowVal = Timestamp.valueOf (rowVal.toString()); // }
rowVal = Timestamp.valueOf (rowVal.toString());
> jdbc handler class cast exception
> -
>
> Key: HIVE-21964
> URL: https://issues.apache.org/jira/browse/HIVE-21964
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Affects Versions: 3.1.1
>Reporter: Aloys Zhang
>Priority: Major
>
> Using hive jdbc handler to query external mysql data source with type decimal
> type, it throws class cast Exception :
>
> {code:java}
> 2019-07-08T11:11:50,424 ERROR [7787918f-3111-4706-a3b3-0097fa1bc117 main]
> CliDriver: Failed with exception
> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
> org.apache.hadoop.hive.common.type.HiveDecimal
> java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
> org.apache.hadoop.hive.common.type.HiveDecimal
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:162)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2691)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
> org.apache.hadoop.hive.common.type.HiveDecimal
> at
> org.apache.hadoop.hive.ql.exec.ListSinkOperator.process(ListSinkOperator.java:98)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:995)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:941)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:995)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:941)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:519)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:511)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146)
> ... 14 more
> Caused by: java.lang.ClassCastException: java.math.BigDecimal cannot be cast
> to org.apache.hadoop.hive.common.type.HiveDecimal
> at
>
[jira] [Comment Edited] (HIVE-21964) jdbc handler class cast exception
[
https://issues.apache.org/jira/browse/HIVE-21964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199318#comment-17199318
]
chenruotao edited comment on HIVE-21964 at 9/21/20, 10:35 AM:
--
I has the same problem like this, but not Decimal.
the type of date and timestramp would throw the same exception when job
running, so I do not use the type provided by
hive(org.apache.hadoop.hive.common.type), and it worked. the code like this :
case DATE:
// if (rowVal instanceof java.sql.Date){
// LocalDate localDate = ((java.sql.Date) rowVal).toLocalDate();
//rowVal=Date.of(localDate.getYear(),localDate.getMonthValue(),localDate.getDayOfMonth();
// }else{
// rowVal = Date.valueOf (rowVal.toString());
// }
rowVal = Date.valueOf (rowVal.toString());
break;
case TIMESTAMP:
// if (rowVal instanceof java.sql.Timestamp){
// LocalDateTime localDateTime = ((java.sql.Timestamp)
rowVal).toLocalDateTime();
//rowVal=Timestamp.ofEpochSecond(localDateTime.toEpochSecond(UTC),localDateTime.geNano());
// }else{
// rowVal = Timestamp.valueOf (rowVal.toString());
// }
rowVal = Timestamp.valueOf (rowVal.toString());
was (Author: chenruotao):
I has the same problem like this, but not Decimal.
the type of date and timestramp would throw the same exception when job
running, so I do not use the type provided by
hive(org.apache.hadoop.hive.common.type), and it worked. the code like this :
case DATE:
// if (rowVal instanceof java.sql.Date) {
// LocalDate localDate = ((java.sql.Date) rowVal).toLocalDate();
// rowVal = Date.of(localDate.getYear(), localDate.getMonthValue(),
localDate.getDayOfMonth());
// } else {
// rowVal = Date.valueOf (rowVal.toString());
// }
rowVal = Date.valueOf (rowVal.toString());
break;
case TIMESTAMP:
// if (rowVal instanceof java.sql.Timestamp) {
// LocalDateTime localDateTime = ((java.sql.Timestamp)
rowVal).toLocalDateTime();
// rowVal = Timestamp.ofEpochSecond(localDateTime.toEpochSecond(UTC),
localDateTime.getNano());
// } else {
// rowVal = Timestamp.valueOf (rowVal.toString());
// }
rowVal = Timestamp.valueOf (rowVal.toString());
> jdbc handler class cast exception
> -
>
> Key: HIVE-21964
> URL: https://issues.apache.org/jira/browse/HIVE-21964
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Affects Versions: 3.1.1
>Reporter: Aloys Zhang
>Priority: Major
>
> Using hive jdbc handler to query external mysql data source with type decimal
> type, it throws class cast Exception :
>
> {code:java}
> 2019-07-08T11:11:50,424 ERROR [7787918f-3111-4706-a3b3-0097fa1bc117 main]
> CliDriver: Failed with exception
> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
> org.apache.hadoop.hive.common.type.HiveDecimal
> java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
> org.apache.hadoop.hive.common.type.HiveDecimal
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:162)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2691)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
> org.apache.hadoop.hive.common.type.HiveDecimal
> at
> org.apache.hadoop.hive.ql.exec.ListSinkOperator.process(ListSinkOperator.java:98)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:995)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:941)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:995)
> at
[jira] [Commented] (HIVE-21964) jdbc handler class cast exception
[
https://issues.apache.org/jira/browse/HIVE-21964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199318#comment-17199318
]
chenruotao commented on HIVE-21964:
---
I has the same problem like this, but not Decimal.
the type of date and timestramp would throw the same exception when job
running, so I do not use the type provided by
hive(org.apache.hadoop.hive.common.type), and it worked. the code like this :
case DATE:
// if (rowVal instanceof java.sql.Date) {
// LocalDate localDate = ((java.sql.Date) rowVal).toLocalDate();
// rowVal = Date.of(localDate.getYear(), localDate.getMonthValue(),
localDate.getDayOfMonth());
// } else {
// rowVal = Date.valueOf (rowVal.toString());
// }
rowVal = Date.valueOf (rowVal.toString());
break;
case TIMESTAMP:
// if (rowVal instanceof java.sql.Timestamp) {
// LocalDateTime localDateTime = ((java.sql.Timestamp)
rowVal).toLocalDateTime();
// rowVal = Timestamp.ofEpochSecond(localDateTime.toEpochSecond(UTC),
localDateTime.getNano());
// } else {
// rowVal = Timestamp.valueOf (rowVal.toString());
// }
rowVal = Timestamp.valueOf (rowVal.toString());
> jdbc handler class cast exception
> -
>
> Key: HIVE-21964
> URL: https://issues.apache.org/jira/browse/HIVE-21964
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Affects Versions: 3.1.1
>Reporter: Aloys Zhang
>Priority: Major
>
> Using hive jdbc handler to query external mysql data source with type decimal
> type, it throws class cast Exception :
>
> {code:java}
> 2019-07-08T11:11:50,424 ERROR [7787918f-3111-4706-a3b3-0097fa1bc117 main]
> CliDriver: Failed with exception
> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
> org.apache.hadoop.hive.common.type.HiveDecimal
> java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
> org.apache.hadoop.hive.common.type.HiveDecimal
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:162)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2691)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:141)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.math.BigDecimal cannot be cast to
> org.apache.hadoop.hive.common.type.HiveDecimal
> at
> org.apache.hadoop.hive.ql.exec.ListSinkOperator.process(ListSinkOperator.java:98)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:995)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:941)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:995)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:941)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:519)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:511)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146)
> ... 14 more
> Caused by: java.lang.ClassCastException: java.math.BigDecimal cannot be cast
> to org.apache.hadoop.hive.common.type.HiveDecimal
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaHiveDecimalObjectInspector.getPrimitiveJavaObject(JavaHiveDecimalObjectInspector.java:55)
> at
> org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitiveUTF8(LazyUtils.java:329)
> at
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(LazySimpleSerDe.java:292)
> at
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serializeField(LazySimpleSerDe.java:247)
> at
>
[jira] [Commented] (HIVE-22352) Hive JDBC Storage Handler, simple select query failed with NPE if executed using Fetch Task
[
https://issues.apache.org/jira/browse/HIVE-22352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183003#comment-17183003
]
chenruotao commented on HIVE-22352:
---
maybe you should try to set hive.sql.query.fieldNames=id,date and set
hive.sql.query.fieldTypes=int,timestamp
or update the ext table column name col1 to id and col2 to date
> Hive JDBC Storage Handler, simple select query failed with NPE if executed
> using Fetch Task
> ---
>
> Key: HIVE-22352
> URL: https://issues.apache.org/jira/browse/HIVE-22352
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.1
> Environment: Hive-3.1
>Reporter: Rajkumar Singh
>Priority: Blocker
>
> Steps To Repro:
>
> {code:java}
> // MySQL Table
> CREATE TABLE `visitors` ( `id` bigint(20) unsigned NOT NULL, `date` timestamp
> NOT NULL DEFAULT CURRENT_TIMESTAMP )
> // hive table
> CREATE EXTERNAL TABLE `hive_visitors`( `col1` bigint COMMENT 'from
> deserializer', `col2` timestamp COMMENT 'from deserializer') ROW FORMAT SERDE
> 'org.apache.hive.storage.jdbc.JdbcSerDe' STORED BY
> 'org.apache.hive.storage.jdbc.JdbcStorageHandler' WITH SERDEPROPERTIES (
> 'serialization.format'='1') TBLPROPERTIES ( 'bucketing_version'='2',
> 'hive.sql.database.type'='MYSQL', 'hive.sql.dbcp.maxActive'='1',
> 'hive.sql.dbcp.password'='hive', 'hive.sql.dbcp.username'='hive',
> 'hive.sql.jdbc.driver'='com.mysql.jdbc.Driver',
> 'hive.sql.jdbc.url'='jdbc:mysql://hostname/test',
> 'hive.sql.table'='visitors', 'transient_lastDdlTime'='1554910389')
> Query:
> select * from hive_visitors ;
> Exception:
> 2019-10-16T04:04:39,483 WARN [HiveServer2-Handler-Pool: Thread-71]:
> thrift.ThriftCLIService (:()) - Error fetching results:
> org.apache.hive.service.cli.HiveSQLException: java.io.IOException:
> java.lang.NullPointerException at
> org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:478)
> ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at
> org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:328)
> ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at
> org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:952)
> ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at
> sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) ~[?:?] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_112] at java.lang.reflect.Method.invoke(Method.java:498)
> ~[?:1.8.0_112] at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
> ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at
> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
> ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at
> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
> ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at
> java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_112] at
> javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112] at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> ~[hadoop-common-3.1.1.3.1.4.0-315.jar:?] at
> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
> ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at
> com.sun.proxy.$Proxy42.fetchResults(Unknown Source) ~[?:?] at
> org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:565)
> ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at
> org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:792)
> ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837)
> ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.1000-SNAPSHOT] at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822)
> ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.1000-SNAPSHOT] at
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.1000-SNAPSHOT] at
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.1000-SNAPSHOT] at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> ~[hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315] at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.1000-SNAPSHOT] at
>
