[jira] [Commented] (HIVE-27789) Iceberg: Add a way to expire snapshots with retain last
[ https://issues.apache.org/jira/browse/HIVE-27789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17783446#comment-17783446 ] Ayush Saxena commented on HIVE-27789: - Committed to master. Thanx [~dkuzmenko] & [~zhangbutao] for the reviews!!! > Iceberg: Add a way to expire snapshots with retain last > --- > > Key: HIVE-27789 > URL: https://issues.apache.org/jira/browse/HIVE-27789 > Project: Hive > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > > Expire snapshots can be performed on iceberg table with retain last param, > add a command for it. > ``` > ALTER TABLE EXECUTE EXPIRE_SNAPSHOT RETAIN LAST 5; > ``` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27789) Iceberg: Add a way to expire snapshots with retain last
[ https://issues.apache.org/jira/browse/HIVE-27789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HIVE-27789. - Fix Version/s: 4.0.0 Resolution: Fixed > Iceberg: Add a way to expire snapshots with retain last > --- > > Key: HIVE-27789 > URL: https://issues.apache.org/jira/browse/HIVE-27789 > Project: Hive > Issue Type: Improvement >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > Expire snapshots can be performed on iceberg table with retain last param, > add a command for it. > ``` > ALTER TABLE EXECUTE EXPIRE_SNAPSHOT RETAIN LAST 5; > ``` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27855) Create external tables for Hive and Tez protologging events
Kiran Velumuri created HIVE-27855: - Summary: Create external tables for Hive and Tez protologging events Key: HIVE-27855 URL: https://issues.apache.org/jira/browse/HIVE-27855 Project: Hive Issue Type: Task Components: Hive, Logging, Tez Reporter: Kiran Velumuri Assignee: Kiran Velumuri Come up with external tables for log files generated by HiveProtoLoggingHook in Hive and ProtoHistoryLoggingService in Tez. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27161) MetaException when executing CTAS query in Druid storage handler
[
https://issues.apache.org/jira/browse/HIVE-27161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Basapuram Kumar updated HIVE-27161:
---
Affects Version/s: 4.0.0-alpha-1
> MetaException when executing CTAS query in Druid storage handler
>
>
> Key: HIVE-27161
> URL: https://issues.apache.org/jira/browse/HIVE-27161
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Affects Versions: 4.0.0-alpha-1, 4.0.0-alpha-2
>Reporter: Stamatis Zampetakis
>Priority: Major
>
> Any kind of CTAS query targeting the Druid storage handler fails with the
> following exception:
> {noformat}
> org.apache.hadoop.hive.ql.metadata.HiveException:
> MetaException(message:LOCATION may not be specified for Druid)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1347)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1352)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.createTableNonReplaceMode(CreateTableOperation.java:158)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.execute(CreateTableOperation.java:116)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:84)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:354)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:327)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:244)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:105)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:367)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:205)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:154)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:149)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:185)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:228)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:257)
> ~[hive-cli-4.0.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.cli.CliDriver.processCmd1(CliDriver.java:201)
> ~[hive-cli-4.0.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:127)
> ~[hive-cli-4.0.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:425)
> ~[hive-cli-4.0.0-SNAPSHOT.jar:?]
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:356)
> ~[hive-cli-4.0.0-SNAPSHOT.jar:?]
> at
> org.apache.hadoop.hive.ql.dataset.QTestDatasetHandler.initDataset(QTestDatasetHandler.java:86)
> ~[hive-it-util-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.dataset.QTestDatasetHandler.beforeTest(QTestDatasetHandler.java:190)
> ~[hive-it-util-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.qoption.QTestOptionDispatcher.beforeTest(QTestOptionDispatcher.java:79)
> ~[hive-it-util-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.QTestUtil.cliInit(QTestUtil.java:607)
> ~[hive-it-util-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:112)
> ~[hive-it-util-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157)
> ~[hive-it-util-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver(TestMiniDruidCliDriver.java:60)
> ~[test-classes/:?]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_261]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
[jira] [Commented] (HIVE-27846) Tests under hive-unit module are not running
[
https://issues.apache.org/jira/browse/HIVE-27846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17783153#comment-17783153
]
Stamatis Zampetakis commented on HIVE-27846:
In the [latest CI run on
mater|http://ci.hive.apache.org/job/hive-precommit/job/master/1945/testReport/]
104,179 tests are running.
In the [PR head with the proposed
changes|http://ci.hive.apache.org/job/hive-precommit/job/PR-4850/3/testReport/]
105,452 tests are running.
The changes under PR#4850 restore 1273 unit tests.
> Tests under hive-unit module are not running
>
>
> Key: HIVE-27846
> URL: https://issues.apache.org/jira/browse/HIVE-27846
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
>
> All the tests under hive-unit module are not running currently in master
> neither locally nor in Jenkins CI.
> {noformat}
> mvn test -pl itests/hive-unit -Pitests
> {noformat}
> {noformat}
> [INFO] — maven-surefire-plugin:3.0.0-M4:test (default-test) @ hive-it-unit —
> [INFO]
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO]
> [INFO] Results:
> [INFO]
> [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 10.444 s
> [INFO] Finished at: 2023-11-02T12:21:51+01:00
> [INFO]
>
> {noformat}
> The problem is caused by HIVE-27757 as it can be seen by comparing the test
> reports in master:
> * http://ci.hive.apache.org/job/hive-precommit/job/master/1907/testReport/
> * http://ci.hive.apache.org/job/hive-precommit/job/master/1906/testReport/
> Observe that when HIVE-27757 was merged the total number of tests dropped
> from 49,200 (1906) to 47,663 (1907) reducing the total number of tests by
> 1537. Comparing the two test reports together it becomes clear that all the
> tests under hive-unit module are now missing.
> The problem seems to be caused by the new junit-jupiter-engine dependency
> that was added in hive-unit module.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (HIVE-27824) Upgrade Ivy to 2.5.2
[ https://issues.apache.org/jira/browse/HIVE-27824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis reopened HIVE-27824: > Upgrade Ivy to 2.5.2 > > > Key: HIVE-27824 > URL: https://issues.apache.org/jira/browse/HIVE-27824 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Devaspati Krishnatri >Assignee: Devaspati Krishnatri >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: mvn_dependency_tree.txt > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27824) Upgrade Ivy to 2.5.2
[ https://issues.apache.org/jira/browse/HIVE-27824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis updated HIVE-27824: --- Fix Version/s: (was: 4.0.0) > Upgrade Ivy to 2.5.2 > > > Key: HIVE-27824 > URL: https://issues.apache.org/jira/browse/HIVE-27824 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Devaspati Krishnatri >Assignee: Devaspati Krishnatri >Priority: Major > Labels: pull-request-available > Attachments: mvn_dependency_tree.txt > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27824) Upgrade Ivy to 2.5.2
[ https://issues.apache.org/jira/browse/HIVE-27824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17783159#comment-17783159 ] Stamatis Zampetakis commented on HIVE-27824: I reverted the ivy ugprade in https://github.com/apache/hive/commit/698194183a2cb4ee6bc528ee71462cbae2c2c7ad since it causes test failures thus I am reopening this ticket. > Upgrade Ivy to 2.5.2 > > > Key: HIVE-27824 > URL: https://issues.apache.org/jira/browse/HIVE-27824 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Devaspati Krishnatri >Assignee: Devaspati Krishnatri >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: mvn_dependency_tree.txt > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27853) Add support for Kerberos authentication in Windows
[ https://issues.apache.org/jira/browse/HIVE-27853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shubham Goel updated HIVE-27853: Description: Kerberos authentication functions differently on Windows and Unix system. In Unix, TGT cache is exposed and can be utilized for authentication purposes. However, Windows does not expose this cache. Therefore, in order to implement Kerberos authentication in a Windows setup, we need to use SSPI (Windows native Kerberos API implementation). Due to this, Kerberos authentication in Hive JDBC driver works in Unix but doesn't work in Windows. > Add support for Kerberos authentication in Windows > -- > > Key: HIVE-27853 > URL: https://issues.apache.org/jira/browse/HIVE-27853 > Project: Hive > Issue Type: New Feature > Components: JDBC >Reporter: Shubham Goel >Priority: Major > > Kerberos authentication functions differently on Windows and Unix system. In > Unix, TGT cache is exposed and can be utilized for authentication purposes. > However, Windows does not expose this cache. Therefore, in order to implement > Kerberos authentication in a Windows setup, we need to use SSPI (Windows > native Kerberos API implementation). > > Due to this, Kerberos authentication in Hive JDBC driver works in Unix but > doesn't work in Windows. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Attachment: image-2023-11-06-18-22-45-812.png > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute addShutdownHook, but now no thread to call > the close method of driver, so it alway stay in ShutdownHookManager's hooks, > causing a memory leak > !image-2023-11-06-18-11-47-832.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Description: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute addShutdownHook, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by add this code : !image-2023-11-06-19-30-14-120.png! and obviously we see no more memory leak was: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute addShutdownHook, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute addShutdownHook, but now no thread to call > the close method of driver, so it alway stay in ShutdownHookManager's hooks, > causing a memory leak > !image-2023-11-06-18-11-47-832.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by add this code : > !image-2023-11-06-19-30-14-120.png! > and obviously we see no more memory leak -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Attachment: image-2023-11-06-19-40-49-488.png > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute addShutdownHook, but now no thread to call > the close method of driver, so it alway stay in ShutdownHookManager's hooks, > causing a memory leak > !image-2023-11-06-18-11-47-832.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by add this code : > !image-2023-11-06-19-30-14-120.png! > and obviously we see no more memory leak -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-27854 started by chenruotao. - > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png, image-2023-11-06-19-45-03-388.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute the addShutdownHook method, but now no > thread to call the close method of driver, so it alway stay in > ShutdownHookManager's hooks, causing a memory leak > !image-2023-11-06-18-11-47-832.png|width=943,height=241! > !image-2023-11-06-19-45-03-388.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver after SQLOperation exexute cancel > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the shutdownHookManager$HookEntry will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by adding this code : > !image-2023-11-06-19-30-14-120.png! > Obviously we see no more memory leak, and shutdownHookManager$HookEntry num > will not increase every time i submit sql > !image-2023-11-06-19-40-49-488.png|width=956,height=140! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27854) Driver compile may causing memory leak
chenruotao created HIVE-27854: - Summary: Driver compile may causing memory leak Key: HIVE-27854 URL: https://issues.apache.org/jira/browse/HIVE-27854 Project: Hive Issue Type: Bug Components: Hive Affects Versions: 4.0.0-beta-1, 3.1.2 Reporter: chenruotao Assignee: chenruotao when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute addShutdownHook, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Attachment: image-2023-11-06-18-13-43-182.png image-2023-11-06-18-13-00-842.png image-2023-11-06-18-11-47-832.png Description: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute addShutdownHook, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! was:when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute addShutdownHook, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute addShutdownHook, but now no thread to call > the close method of driver, so it alway stay in ShutdownHookManager's hooks, > causing a memory leak > !image-2023-11-06-18-11-47-832.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Attachment: image-2023-11-06-18-21-19-056.png > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute addShutdownHook, but now no thread to call > the close method of driver, so it alway stay in ShutdownHookManager's hooks, > causing a memory leak > !image-2023-11-06-18-11-47-832.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Attachment: image-2023-11-06-18-24-41-982.png > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute addShutdownHook, but now no thread to call > the close method of driver, so it alway stay in ShutdownHookManager's hooks, > causing a memory leak > !image-2023-11-06-18-11-47-832.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Description: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute addShutdownHook, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by add this code : !image-2023-11-06-19-30-14-120.png! and obviously we see no more memory leak, ShutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! was: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute addShutdownHook, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by add this code : !image-2023-11-06-19-30-14-120.png! and obviously we see no more memory leak, ShutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png! > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute addShutdownHook, but now no thread to call > the close method of driver, so it alway stay in ShutdownHookManager's hooks, > causing a memory leak > !image-2023-11-06-18-11-47-832.png|width=943,height=241! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by add this code : > !image-2023-11-06-19-30-14-120.png! > and obviously we see no more memory leak, ShutdownHookManager$HookEntry num > will not increase every time i submit sql > !image-2023-11-06-19-40-49-488.png|width=956,height=140! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Description: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute the addShutdownHook method, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! !image-2023-11-06-19-45-03-388.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver after SQLOperation exexute cancel !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the shutdownHookManager$HookEntry will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by adding this code : !image-2023-11-06-19-30-14-120.png! Obviously we see no more memory leak, and shutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! was: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute the addShutdownHook method, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! !image-2023-11-06-19-45-03-388.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver after SQLOperation exexute cancel !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by adding this code : !image-2023-11-06-19-30-14-120.png! Obviously we see no more memory leak, and shutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png, image-2023-11-06-19-45-03-388.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute the addShutdownHook method, but now no > thread to call the close method of driver, so it alway stay in > ShutdownHookManager's hooks, causing a memory leak > !image-2023-11-06-18-11-47-832.png|width=943,height=241! > !image-2023-11-06-19-45-03-388.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver after SQLOperation exexute cancel > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the shutdownHookManager$HookEntry will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by adding this code : > !image-2023-11-06-19-30-14-120.png! > Obviously we see no more memory leak, and shutdownHookManager$HookEntry num > will not increase every time i submit sql > !image-2023-11-06-19-40-49-488.png|width=956,height=140! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Description: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute the addShutdownHook method, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! !image-2023-11-06-19-45-03-388.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver after SQLOperation exexute cancel !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by adding this code : !image-2023-11-06-19-30-14-120.png! Obviously we see no more memory leak, and shutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! was: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute the addShutdownHook method, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! !image-2023-11-06-19-45-03-388.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver after SQLOperation exexute cancel !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by add this code : !image-2023-11-06-19-30-14-120.png! and obviously we see no more memory leak, ShutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png, image-2023-11-06-19-45-03-388.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute the addShutdownHook method, but now no > thread to call the close method of driver, so it alway stay in > ShutdownHookManager's hooks, causing a memory leak > !image-2023-11-06-18-11-47-832.png|width=943,height=241! > !image-2023-11-06-19-45-03-388.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver after SQLOperation exexute cancel > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by adding this code : > !image-2023-11-06-19-30-14-120.png! > Obviously we see no more memory leak, and shutdownHookManager$HookEntry num > will not increase every time i submit sql > !image-2023-11-06-19-40-49-488.png|width=956,height=140! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Description: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute the addShutdownHook method, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! !image-2023-11-06-19-45-03-388.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver after SQLOperation exexute cancel !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! and now the driverState is INITIALIZED, so the lDrvState can not set abort to interrupted the compile !image-2023-11-06-20-26-07-406.png|width=479,height=365! !image-2023-11-06-20-28-08-239.png|width=731,height=97! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png|width=763,height=257! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the shutdownHookManager$HookEntry will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by adding this code : !image-2023-11-06-19-30-14-120.png! Obviously we see no more memory leak, and shutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! was: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute the addShutdownHook method, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! !image-2023-11-06-19-45-03-388.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver after SQLOperation exexute cancel !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! and now the driverState is INITIALIZED, so the lDrvState can not set abort to interrupted the !image-2023-11-06-20-26-07-406.png|width=479,height=365! !image-2023-11-06-20-28-08-239.png|width=731,height=97! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png|width=763,height=257! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the shutdownHookManager$HookEntry will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by adding this code : !image-2023-11-06-19-30-14-120.png! Obviously we see no more memory leak, and shutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png, image-2023-11-06-19-45-03-388.png, > image-2023-11-06-20-26-07-406.png, image-2023-11-06-20-28-08-239.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute the addShutdownHook method, but now no > thread to call the close method of driver, so it alway stay in > ShutdownHookManager's hooks, causing a memory leak > !image-2023-11-06-18-11-47-832.png|width=943,height=241! > !image-2023-11-06-19-45-03-388.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver after SQLOperation exexute cancel > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > and now the driverState is INITIALIZED, so the lDrvState can not set abort to > interrupted the compile > !image-2023-11-06-20-26-07-406.png|width=479,height=365! > !image-2023-11-06-20-28-08-239.png|width=731,height=97! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png|width=763,height=257! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png!
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Attachment: image-2023-11-06-20-28-08-239.png > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png, image-2023-11-06-19-45-03-388.png, > image-2023-11-06-20-26-07-406.png, image-2023-11-06-20-28-08-239.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute the addShutdownHook method, but now no > thread to call the close method of driver, so it alway stay in > ShutdownHookManager's hooks, causing a memory leak > !image-2023-11-06-18-11-47-832.png|width=943,height=241! > !image-2023-11-06-19-45-03-388.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver after SQLOperation exexute cancel > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the shutdownHookManager$HookEntry will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by adding this code : > !image-2023-11-06-19-30-14-120.png! > Obviously we see no more memory leak, and shutdownHookManager$HookEntry num > will not increase every time i submit sql > !image-2023-11-06-19-40-49-488.png|width=956,height=140! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27848) TestCrudCompactorOnTez.secondCompactionShouldBeRefusedBeforeEnqueueing fails
[
https://issues.apache.org/jira/browse/HIVE-27848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17783161#comment-17783161
]
Stamatis Zampetakis commented on HIVE-27848:
I disabled the test in master since it is failing. cc [~dkuzmenko] [~lvegh]
since you reviewed HIVE-27598.
> TestCrudCompactorOnTez.secondCompactionShouldBeRefusedBeforeEnqueueing fails
> -
>
> Key: HIVE-27848
> URL: https://issues.apache.org/jira/browse/HIVE-27848
> Project: Hive
> Issue Type: Bug
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
>
> {noformat}
> cd itest/hive-unit
> mvn test
> -Dtest=TestCrudCompactorOnTez#secondCompactionShouldBeRefusedBeforeEnqueueing
> {noformat}
> {noformat}
> [INFO] ---
> [INFO] Running org.apache.hadoop.hive.ql.txn.compactor.TestCrudCompactorOnTez
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 39.548 s <<< FAILURE! - in
> org.apache.hadoop.hive.ql.txn.compactor.TestCrudCompactorOnTez
> [ERROR]
> org.apache.hadoop.hive.ql.txn.compactor.TestCrudCompactorOnTez.secondCompactionShouldBeRefusedBeforeEnqueueing
> Time elapsed: 39.444 s <<< FAILURE!
> org.junit.ComparisonFailure: expected:<...ault.compaction_test[] is refused,
> details...> but was:<...ault.compaction_test[CompactionId: -1] is refused,
> details...>
> at org.junit.Assert.assertEquals(Assert.java:117)
> at org.junit.Assert.assertEquals(Assert.java:146)
> at
> org.apache.hadoop.hive.ql.txn.compactor.TestCrudCompactorOnTez.secondCompactionShouldBeRefusedBeforeEnqueueing(TestCrudCompactorOnTez.java:638)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138)
> at
> org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451)
> {noformat}
> The problem was discovered while trying to re-enable the tests under
> hive-unit module (HIVE-27846).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-27848) TestCrudCompactorOnTez.secondCompactionShouldBeRefusedBeforeEnqueueing fails
[
https://issues.apache.org/jira/browse/HIVE-27848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis reassigned HIVE-27848:
--
Assignee: (was: Stamatis Zampetakis)
> TestCrudCompactorOnTez.secondCompactionShouldBeRefusedBeforeEnqueueing fails
> -
>
> Key: HIVE-27848
> URL: https://issues.apache.org/jira/browse/HIVE-27848
> Project: Hive
> Issue Type: Bug
>Reporter: Stamatis Zampetakis
>Priority: Major
>
> {noformat}
> cd itest/hive-unit
> mvn test
> -Dtest=TestCrudCompactorOnTez#secondCompactionShouldBeRefusedBeforeEnqueueing
> {noformat}
> {noformat}
> [INFO] ---
> [INFO] Running org.apache.hadoop.hive.ql.txn.compactor.TestCrudCompactorOnTez
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 39.548 s <<< FAILURE! - in
> org.apache.hadoop.hive.ql.txn.compactor.TestCrudCompactorOnTez
> [ERROR]
> org.apache.hadoop.hive.ql.txn.compactor.TestCrudCompactorOnTez.secondCompactionShouldBeRefusedBeforeEnqueueing
> Time elapsed: 39.444 s <<< FAILURE!
> org.junit.ComparisonFailure: expected:<...ault.compaction_test[] is refused,
> details...> but was:<...ault.compaction_test[CompactionId: -1] is refused,
> details...>
> at org.junit.Assert.assertEquals(Assert.java:117)
> at org.junit.Assert.assertEquals(Assert.java:146)
> at
> org.apache.hadoop.hive.ql.txn.compactor.TestCrudCompactorOnTez.secondCompactionShouldBeRefusedBeforeEnqueueing(TestCrudCompactorOnTez.java:638)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138)
> at
> org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451)
> {noformat}
> The problem was discovered while trying to re-enable the tests under
> hive-unit module (HIVE-27846).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-25695) Make spark views authorization in hive configurable.
[
https://issues.apache.org/jira/browse/HIVE-25695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manish Maheshwari updated HIVE-25695:
-
Description:
HIVE-24026 introduced an authorization model where views created from external
sources like spark are not authorized at create time, but when a user does
select on the view. We need to make this authorization configurable.
This Jira introduces a new config to make this auth model configurable.
{code:java}
hive.security.authorization.enabled.on.spark.views=true {code}
This config is turned on by default. If the users wish to turn off this config,
then they can set this config to false, which means that during the select
query, the underlying tables for that view will not be authorized.
The reason for making this auth model configurable is because there can be a
use-case where a user is running workload of create/alter/select views without
HIVE-24026 (with ranger/sentry policies in place where user have select
permissions only on view but not on underlying tables) and when user upgrades
to HIVE-24026, the admin will have to configure ranger/sentry policies on all
the underlying tables for required users. By simply turning off this config,
the user can do workload operations but at the cost of the security hole for
not authorizing the underlying tables.
was:
HIVE-24026 introduced an authorization model where views created from external
sources like spark are not authorized at create time, but when a user does
select on the view. We need to make this authorization configurable.
This Jira introduces a new config to make this auth model configurable.
{code:java}
hive.security.authorization.enabled.on.spark.views=true {code}
This config is turned on by default. If the users wish to turn off this config,
then they can set this config to false, which means that during the select
query, the underlying tables for that view will not be authorized.
The reason for making this auth model configurable is because there can be a
use-case where a user is running workload of create/alter/select views without
HIVE-24026 (with ranger/sentry policies in place where user have select
permissions only on view but not on underlying tables) and when user upgrades
to HIVE-24026, the admin will have to configure ranger/sentry policies on all
the underlying tables for required users. By simply turning off this config,
the user can do workload operations but at the cost of the security hole for
not authorizing the underlying tables.
> Make spark views authorization in hive configurable.
>
>
> Key: HIVE-25695
> URL: https://issues.apache.org/jira/browse/HIVE-25695
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Sai Hemanth Gantasala
>Assignee: Sai Hemanth Gantasala
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> HIVE-24026 introduced an authorization model where views created from
> external sources like spark are not authorized at create time, but when a
> user does select on the view. We need to make this authorization
> configurable.
> This Jira introduces a new config to make this auth model configurable.
> {code:java}
> hive.security.authorization.enabled.on.spark.views=true {code}
> This config is turned on by default. If the users wish to turn off this
> config, then they can set this config to false, which means that during the
> select query, the underlying tables for that view will not be authorized.
> The reason for making this auth model configurable is because there can be a
> use-case where a user is running workload of create/alter/select views
> without HIVE-24026 (with ranger/sentry policies in place where user have
> select permissions only on view but not on underlying tables) and when user
> upgrades to HIVE-24026, the admin will have to configure ranger/sentry
> policies on all the underlying tables for required users. By simply turning
> off this config, the user can do workload operations but at the cost of the
> security hole for not authorizing the underlying tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Description: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute addShutdownHook, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! was: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute addShutdownHook, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute addShutdownHook, but now no thread to call > the close method of driver, so it alway stay in ShutdownHookManager's hooks, > causing a memory leak > !image-2023-11-06-18-11-47-832.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Attachment: image-2023-11-06-19-30-14-120.png > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute addShutdownHook, but now no thread to call > the close method of driver, so it alway stay in ShutdownHookManager's hooks, > causing a memory leak > !image-2023-11-06-18-11-47-832.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-27854: -- Labels: pull-request-available (was: ) > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute addShutdownHook, but now no thread to call > the close method of driver, so it alway stay in ShutdownHookManager's hooks, > causing a memory leak > !image-2023-11-06-18-11-47-832.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by add this code : > !image-2023-11-06-19-30-14-120.png! > and obviously we see no more memory leak -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Description: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute addShutdownHook, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by add this code : !image-2023-11-06-19-30-14-120.png! and obviously we see no more memory leak, ShutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png! was: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute addShutdownHook, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by add this code : !image-2023-11-06-19-30-14-120.png! and obviously we see no more memory leak > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute addShutdownHook, but now no thread to call > the close method of driver, so it alway stay in ShutdownHookManager's hooks, > causing a memory leak > !image-2023-11-06-18-11-47-832.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by add this code : > !image-2023-11-06-19-30-14-120.png! > and obviously we see no more memory leak, ShutdownHookManager$HookEntry num > will not increase every time i submit sql > !image-2023-11-06-19-40-49-488.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Description: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute the addShutdownHook method, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! !image-2023-11-06-19-45-03-388.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver after SQLOperation exexute cancel !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by add this code : !image-2023-11-06-19-30-14-120.png! and obviously we see no more memory leak, ShutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! was: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute the addShutdownHook method, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! !image-2023-11-06-19-45-03-388.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by add this code : !image-2023-11-06-19-30-14-120.png! and obviously we see no more memory leak, ShutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png, image-2023-11-06-19-45-03-388.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute the addShutdownHook method, but now no > thread to call the close method of driver, so it alway stay in > ShutdownHookManager's hooks, causing a memory leak > !image-2023-11-06-18-11-47-832.png|width=943,height=241! > !image-2023-11-06-19-45-03-388.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver after SQLOperation exexute cancel > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by add this code : > !image-2023-11-06-19-30-14-120.png! > and obviously we see no more memory leak, ShutdownHookManager$HookEntry num > will not increase every time i submit sql > !image-2023-11-06-19-40-49-488.png|width=956,height=140! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Description: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute the addShutdownHook method, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! !image-2023-11-06-19-45-03-388.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver after SQLOperation exexute cancel !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! and now the driverState is INITIALIZED, so the lDrvState can not set abort to interrupted the !image-2023-11-06-20-26-07-406.png|width=479,height=365! !image-2023-11-06-20-28-08-239.png|width=731,height=97! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png|width=763,height=257! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the shutdownHookManager$HookEntry will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by adding this code : !image-2023-11-06-19-30-14-120.png! Obviously we see no more memory leak, and shutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! was: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute the addShutdownHook method, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! !image-2023-11-06-19-45-03-388.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver after SQLOperation exexute cancel !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the shutdownHookManager$HookEntry will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by adding this code : !image-2023-11-06-19-30-14-120.png! Obviously we see no more memory leak, and shutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png, image-2023-11-06-19-45-03-388.png, > image-2023-11-06-20-26-07-406.png, image-2023-11-06-20-28-08-239.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute the addShutdownHook method, but now no > thread to call the close method of driver, so it alway stay in > ShutdownHookManager's hooks, causing a memory leak > !image-2023-11-06-18-11-47-832.png|width=943,height=241! > !image-2023-11-06-19-45-03-388.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver after SQLOperation exexute cancel > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > and now the driverState is INITIALIZED, so the lDrvState can not set abort to > interrupted the > !image-2023-11-06-20-26-07-406.png|width=479,height=365! > !image-2023-11-06-20-28-08-239.png|width=731,height=97! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png|width=763,height=257! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the shutdownHookManager$HookEntry will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by adding this code : > !image-2023-11-06-19-30-14-120.png! > Obviously we see
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Attachment: image-2023-11-06-20-26-07-406.png > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png, image-2023-11-06-19-45-03-388.png, > image-2023-11-06-20-26-07-406.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute the addShutdownHook method, but now no > thread to call the close method of driver, so it alway stay in > ShutdownHookManager's hooks, causing a memory leak > !image-2023-11-06-18-11-47-832.png|width=943,height=241! > !image-2023-11-06-19-45-03-388.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver after SQLOperation exexute cancel > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the shutdownHookManager$HookEntry will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by adding this code : > !image-2023-11-06-19-30-14-120.png! > Obviously we see no more memory leak, and shutdownHookManager$HookEntry num > will not increase every time i submit sql > !image-2023-11-06-19-40-49-488.png|width=956,height=140! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27846) Tests under hive-unit module are not running
[
https://issues.apache.org/jira/browse/HIVE-27846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis resolved HIVE-27846.
Fix Version/s: 4.0.0
Resolution: Fixed
Fixed in
https://github.com/apache/hive/commit/698194183a2cb4ee6bc528ee71462cbae2c2c7ad.
Thanks for the review [~ayushsaxena]!
> Tests under hive-unit module are not running
>
>
> Key: HIVE-27846
> URL: https://issues.apache.org/jira/browse/HIVE-27846
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> All the tests under hive-unit module are not running currently in master
> neither locally nor in Jenkins CI.
> {noformat}
> mvn test -pl itests/hive-unit -Pitests
> {noformat}
> {noformat}
> [INFO] — maven-surefire-plugin:3.0.0-M4:test (default-test) @ hive-it-unit —
> [INFO]
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO]
> [INFO] Results:
> [INFO]
> [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 10.444 s
> [INFO] Finished at: 2023-11-02T12:21:51+01:00
> [INFO]
>
> {noformat}
> The problem is caused by HIVE-27757 as it can be seen by comparing the test
> reports in master:
> * http://ci.hive.apache.org/job/hive-precommit/job/master/1907/testReport/
> * http://ci.hive.apache.org/job/hive-precommit/job/master/1906/testReport/
> Observe that when HIVE-27757 was merged the total number of tests dropped
> from 49,200 (1906) to 47,663 (1907) reducing the total number of tests by
> 1537. Comparing the two test reports together it becomes clear that all the
> tests under hive-unit module are now missing.
> The problem seems to be caused by the new junit-jupiter-engine dependency
> that was added in hive-unit module.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Attachment: image-2023-11-06-19-45-03-388.png > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png, image-2023-11-06-19-45-03-388.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute addShutdownHook, but now no thread to call > the close method of driver, so it alway stay in ShutdownHookManager's hooks, > causing a memory leak > !image-2023-11-06-18-11-47-832.png|width=943,height=241! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by add this code : > !image-2023-11-06-19-30-14-120.png! > and obviously we see no more memory leak, ShutdownHookManager$HookEntry num > will not increase every time i submit sql > !image-2023-11-06-19-40-49-488.png|width=956,height=140! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27854) Driver compile may causing memory leak
[ https://issues.apache.org/jira/browse/HIVE-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenruotao updated HIVE-27854: -- Description: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute the addShutdownHook method, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! !image-2023-11-06-19-45-03-388.png! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by add this code : !image-2023-11-06-19-30-14-120.png! and obviously we see no more memory leak, ShutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! was: when submit a sql to hiveserver, if the sql query timeout before compile, it will continue compile and execute addShutdownHook, but now no thread to call the close method of driver, so it alway stay in ShutdownHookManager's hooks, causing a memory leak !image-2023-11-06-18-11-47-832.png|width=943,height=241! the cancel from SQLOperation will set it's driver = null, so it can not call the close method of driver !image-2023-11-06-18-13-43-182.png! !image-2023-11-06-18-13-00-842.png! it will not trigger everytime, but we could do this to trigger it 1、sleep 5s before compile !image-2023-11-06-18-21-19-056.png! 2、set hive.query.timeout.seconds to 4 !image-2023-11-06-18-22-45-812.png! 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one !image-2023-11-06-18-24-41-982.png|width=842,height=134! i solve it in hive 3.1.2 by add this code : !image-2023-11-06-19-30-14-120.png! and obviously we see no more memory leak, ShutdownHookManager$HookEntry num will not increase every time i submit sql !image-2023-11-06-19-40-49-488.png|width=956,height=140! > Driver compile may causing memory leak > -- > > Key: HIVE-27854 > URL: https://issues.apache.org/jira/browse/HIVE-27854 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 4.0.0-beta-1 >Reporter: chenruotao >Assignee: chenruotao >Priority: Major > Labels: pull-request-available > Attachments: image-2023-11-06-18-11-47-832.png, > image-2023-11-06-18-13-00-842.png, image-2023-11-06-18-13-43-182.png, > image-2023-11-06-18-21-19-056.png, image-2023-11-06-18-22-45-812.png, > image-2023-11-06-18-24-41-982.png, image-2023-11-06-19-30-14-120.png, > image-2023-11-06-19-40-49-488.png, image-2023-11-06-19-45-03-388.png > > > when submit a sql to hiveserver, if the sql query timeout before compile, it > will continue compile and execute the addShutdownHook method, but now no > thread to call the close method of driver, so it alway stay in > ShutdownHookManager's hooks, causing a memory leak > !image-2023-11-06-18-11-47-832.png|width=943,height=241! > !image-2023-11-06-19-45-03-388.png! > the cancel from SQLOperation will set it's driver = null, so it can not call > the close method of driver > !image-2023-11-06-18-13-43-182.png! > !image-2023-11-06-18-13-00-842.png! > it will not trigger everytime, but we could do this to trigger it > 1、sleep 5s before compile > !image-2023-11-06-18-21-19-056.png! > 2、set hive.query.timeout.seconds to 4 > !image-2023-11-06-18-22-45-812.png! > 3、now everytime I submit sql , the ShutdownHookManager's hooks will add one > !image-2023-11-06-18-24-41-982.png|width=842,height=134! > i solve it in hive 3.1.2 by add this code : > !image-2023-11-06-19-30-14-120.png! > and obviously we see no more memory leak, ShutdownHookManager$HookEntry num > will not increase every time i submit sql > !image-2023-11-06-19-40-49-488.png|width=956,height=140! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27849) Replication tests using ivy fail after the upgrade to 2.5.2
[
https://issues.apache.org/jira/browse/HIVE-27849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis resolved HIVE-27849.
Fix Version/s: 4.0.0
Resolution: Fixed
> Replication tests using ivy fail after the upgrade to 2.5.2
> ---
>
> Key: HIVE-27849
> URL: https://issues.apache.org/jira/browse/HIVE-27849
> Project: Hive
> Issue Type: Bug
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Fix For: 4.0.0
>
>
> The tests inside TestReplAcrossInstancesWithJsonMessageFormat and
> TestReplicationScenariosAcrossInstances that are making some use of ivy fail
> after the upgrade of the latter to 2.5.2 (HIVE-27824).
> The stacktrace from some failures is outlined below.
> {noformat}
> 2023-11-03T09:18:23,132 ERROR [main] exec.FunctionRegistry: Unable to load
> resources for
> testbootstrapreplloadretryafterfailureforpartitions_1699028290779.testfunctionone:java.text.ParseException:
> failed to load settings from
> jar:file:/home/stamatis/.m2/repository/org/codehaus/groovy/groovy-all/2.4.21/groovy-all-2.4.21.jar!/groovy/grape/defaultGrapeConfig.xml:
> Property 'http://javax.xml.XMLConstants/property/accessExternalSchema' is
> not recognized.
> java.text.ParseException: failed to load settings from
> jar:file:/home/stamatis/.m2/repository/org/codehaus/groovy/groovy-all/2.4.21/groovy-all-2.4.21.jar!/groovy/grape/defaultGrapeConfig.xml:
> Property 'http://javax.xml.XMLConstants/property/accessExternalSchema' is
> not recognized.
> at
> org.apache.ivy.core.settings.XmlSettingsParser.doParse(XmlSettingsParser.java:160)
> ~[ivy-2.5.2.jar:2.5.2]
> at
> org.apache.ivy.core.settings.XmlSettingsParser.parse(XmlSettingsParser.java:147)
> ~[ivy-2.5.2.jar:2.5.2]
> at org.apache.ivy.core.settings.IvySettings.load(IvySettings.java:415)
> ~[ivy-2.5.2.jar:2.5.2]
> at org.apache.ivy.core.settings.IvySettings$load$0.call(Unknown Source)
> ~[?:?]
> at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
> ~[groovy-all-2.4.21.jar:2.4.21]
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
> ~[groovy-all-2.4.21.jar:2.4.21]
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)
> ~[groovy-all-2.4.21.jar:2.4.21]
> at groovy.grape.GrapeIvy.(GrapeIvy.groovy:100)
> ~[groovy-all-2.4.21.jar:2.4.21]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[?:1.8.0_261]
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> ~[?:1.8.0_261]
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> ~[?:1.8.0_261]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> ~[?:1.8.0_261]
> at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_261]
> at groovy.grape.Grape.getInstance(Grape.java:121)
> ~[groovy-all-2.4.21.jar:2.4.21]
> at groovy.grape.Grape.resolve(Grape.java:194)
> ~[groovy-all-2.4.21.jar:2.4.21]
> at groovy.grape.Grape.resolve(Grape.java:188)
> ~[groovy-all-2.4.21.jar:2.4.21]
> at
> org.apache.hadoop.hive.ql.util.DependencyResolver.grab(DependencyResolver.java:176)
> ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.util.DependencyResolver.downloadDependencies(DependencyResolver.java:97)
> ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.util.ResourceDownloader.resolveAndDownloadInternal(ResourceDownloader.java:88)
> ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.util.ResourceDownloader.resolveAndDownload(ResourceDownloader.java:76)
> ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1661)
> ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1613)
> ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1599)
> ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FunctionUtils.addFunctionResources(FunctionUtils.java:85)
> ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.Registry.registerToSessionRegistry(Registry.java:693)
> ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
> at
>
[jira] [Commented] (HIVE-27849) Replication tests using ivy fail after the upgrade to 2.5.2
[
https://issues.apache.org/jira/browse/HIVE-27849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17783157#comment-17783157
]
Stamatis Zampetakis commented on HIVE-27849:
Fixed in
https://github.com/apache/hive/commit/698194183a2cb4ee6bc528ee71462cbae2c2c7ad
by reverting ivy upgrade. Thanks for the review [~ayushtkn]!
> Replication tests using ivy fail after the upgrade to 2.5.2
> ---
>
> Key: HIVE-27849
> URL: https://issues.apache.org/jira/browse/HIVE-27849
> Project: Hive
> Issue Type: Bug
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
>
> The tests inside TestReplAcrossInstancesWithJsonMessageFormat and
> TestReplicationScenariosAcrossInstances that are making some use of ivy fail
> after the upgrade of the latter to 2.5.2 (HIVE-27824).
> The stacktrace from some failures is outlined below.
> {noformat}
> 2023-11-03T09:18:23,132 ERROR [main] exec.FunctionRegistry: Unable to load
> resources for
> testbootstrapreplloadretryafterfailureforpartitions_1699028290779.testfunctionone:java.text.ParseException:
> failed to load settings from
> jar:file:/home/stamatis/.m2/repository/org/codehaus/groovy/groovy-all/2.4.21/groovy-all-2.4.21.jar!/groovy/grape/defaultGrapeConfig.xml:
> Property 'http://javax.xml.XMLConstants/property/accessExternalSchema' is
> not recognized.
> java.text.ParseException: failed to load settings from
> jar:file:/home/stamatis/.m2/repository/org/codehaus/groovy/groovy-all/2.4.21/groovy-all-2.4.21.jar!/groovy/grape/defaultGrapeConfig.xml:
> Property 'http://javax.xml.XMLConstants/property/accessExternalSchema' is
> not recognized.
> at
> org.apache.ivy.core.settings.XmlSettingsParser.doParse(XmlSettingsParser.java:160)
> ~[ivy-2.5.2.jar:2.5.2]
> at
> org.apache.ivy.core.settings.XmlSettingsParser.parse(XmlSettingsParser.java:147)
> ~[ivy-2.5.2.jar:2.5.2]
> at org.apache.ivy.core.settings.IvySettings.load(IvySettings.java:415)
> ~[ivy-2.5.2.jar:2.5.2]
> at org.apache.ivy.core.settings.IvySettings$load$0.call(Unknown Source)
> ~[?:?]
> at
> org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
> ~[groovy-all-2.4.21.jar:2.4.21]
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
> ~[groovy-all-2.4.21.jar:2.4.21]
> at
> org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)
> ~[groovy-all-2.4.21.jar:2.4.21]
> at groovy.grape.GrapeIvy.(GrapeIvy.groovy:100)
> ~[groovy-all-2.4.21.jar:2.4.21]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[?:1.8.0_261]
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> ~[?:1.8.0_261]
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> ~[?:1.8.0_261]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> ~[?:1.8.0_261]
> at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_261]
> at groovy.grape.Grape.getInstance(Grape.java:121)
> ~[groovy-all-2.4.21.jar:2.4.21]
> at groovy.grape.Grape.resolve(Grape.java:194)
> ~[groovy-all-2.4.21.jar:2.4.21]
> at groovy.grape.Grape.resolve(Grape.java:188)
> ~[groovy-all-2.4.21.jar:2.4.21]
> at
> org.apache.hadoop.hive.ql.util.DependencyResolver.grab(DependencyResolver.java:176)
> ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.util.DependencyResolver.downloadDependencies(DependencyResolver.java:97)
> ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.util.ResourceDownloader.resolveAndDownloadInternal(ResourceDownloader.java:88)
> ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.util.ResourceDownloader.resolveAndDownload(ResourceDownloader.java:76)
> ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.session.SessionState.resolveAndDownload(SessionState.java:1661)
> ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1613)
> ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.session.SessionState.add_resources(SessionState.java:1599)
> ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.FunctionUtils.addFunctionResources(FunctionUtils.java:85)
> ~[hive-exec-4.0.0-beta-2-SNAPSHOT.jar:4.0.0-beta-2-SNAPSHOT]
> at
>
[jira] [Created] (HIVE-27853) Add support for Kerberos authentication in Windows
Shubham Goel created HIVE-27853: --- Summary: Add support for Kerberos authentication in Windows Key: HIVE-27853 URL: https://issues.apache.org/jira/browse/HIVE-27853 Project: Hive Issue Type: New Feature Components: JDBC Reporter: Shubham Goel -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27850) Compaction for Iceberg tables
[ https://issues.apache.org/jira/browse/HIVE-27850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-27850: -- Labels: pull-request-available (was: ) > Compaction for Iceberg tables > - > > Key: HIVE-27850 > URL: https://issues.apache.org/jira/browse/HIVE-27850 > Project: Hive > Issue Type: New Feature > Components: Iceberg integration >Reporter: Dmitriy Fingerman >Assignee: Dmitriy Fingerman >Priority: Major > Labels: pull-request-available > > Hive currently doesn't have the table compaction functionality. It would be > highly beneficial for performance to implement this feature because this > would create larger data files and eliminate positional delete files. -- This message was sent by Atlassian Jira (v8.20.10#820010)
