[jira] [Updated] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-22572: - Attachment: HIVE-22572.7.patch > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.1.patch, HIVE-22572.2.patch, > HIVE-22572.3.patch, HIVE-22572.4.patch, HIVE-22572.5.patch, > HIVE-22572.6.patch, HIVE-22572.7.patch, HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22616) Disable PreCommit test org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
[ https://issues.apache.org/jira/browse/HIVE-22616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-22616: - Attachment: HIVE-22616.patch > Disable PreCommit test org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 > - > > Key: HIVE-22616 > URL: https://issues.apache.org/jira/browse/HIVE-22616 > Project: Hive > Issue Type: Test > Components: Hive >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Attachments: HIVE-22616.patch, TestFail.pdf > > > The test is flaky and produces following errors: > {{java.io.FileNotFoundException: Source > '/home/hiveptest/34.69.225.53-hiveptest-2/apache-github-source-source/itests/hive-unit/target/junit-qfile-results/clientpositive/join2.q.out' > does not exist}} > {{ at > org.apache.commons.io.FileUtils.checkFileRequirements(FileUtils.java:1383)}} > {{ at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1060)}} > {{ at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1028)}} > {{ at > org.apache.hadoop.hive.ql.QOutProcessor.maskPatterns(QOutProcessor.java:162)}} > {{ at > org.apache.hadoop.hive.ql.QTestUtil.checkCliDriverResults(QTestUtil.java:932)}} > {{ at > org.apache.hadoop.hive.ql.QTestRunnerUtils.queryListRunnerMultiThreaded(QTestRunnerUtils.java:152)}} > {{ at > org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1(TestMTQueries.java:55)}} > {{ at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}} > {{ at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}} > {{ at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}} > {{ at java.lang.reflect.Method.invoke(Method.java:498)}} > {{ at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)}} > {{ at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)}} > {{ at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)}} > {{ at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)}} > {{ at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)}} > {{ at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)}} > {{ at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)}} > {{ at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)}} > {{ at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)}} > {{ at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)}} > {{ at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)}} > {{ at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)}} > {{ at org.junit.runners.ParentRunner.run(ParentRunner.java:309)}} > {{ at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)}} > {{ at > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)}} > {{ at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)}} > {{ at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)}} > {{ at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)}} > {{ at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)}} > {{ at > org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)}} > {{ at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)}} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22616) Disable PreCommit test org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
[ https://issues.apache.org/jira/browse/HIVE-22616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-22616: - Status: Patch Available (was: Open) Disabled test > Disable PreCommit test org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 > - > > Key: HIVE-22616 > URL: https://issues.apache.org/jira/browse/HIVE-22616 > Project: Hive > Issue Type: Test > Components: Hive >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Attachments: HIVE-22616.patch, TestFail.pdf > > > The test is flaky and produces following errors: > {{java.io.FileNotFoundException: Source > '/home/hiveptest/34.69.225.53-hiveptest-2/apache-github-source-source/itests/hive-unit/target/junit-qfile-results/clientpositive/join2.q.out' > does not exist}} > {{ at > org.apache.commons.io.FileUtils.checkFileRequirements(FileUtils.java:1383)}} > {{ at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1060)}} > {{ at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1028)}} > {{ at > org.apache.hadoop.hive.ql.QOutProcessor.maskPatterns(QOutProcessor.java:162)}} > {{ at > org.apache.hadoop.hive.ql.QTestUtil.checkCliDriverResults(QTestUtil.java:932)}} > {{ at > org.apache.hadoop.hive.ql.QTestRunnerUtils.queryListRunnerMultiThreaded(QTestRunnerUtils.java:152)}} > {{ at > org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1(TestMTQueries.java:55)}} > {{ at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}} > {{ at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}} > {{ at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}} > {{ at java.lang.reflect.Method.invoke(Method.java:498)}} > {{ at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)}} > {{ at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)}} > {{ at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)}} > {{ at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)}} > {{ at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)}} > {{ at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)}} > {{ at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)}} > {{ at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)}} > {{ at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)}} > {{ at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)}} > {{ at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)}} > {{ at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)}} > {{ at org.junit.runners.ParentRunner.run(ParentRunner.java:309)}} > {{ at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)}} > {{ at > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)}} > {{ at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)}} > {{ at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)}} > {{ at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)}} > {{ at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)}} > {{ at > org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)}} > {{ at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)}} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22616) Disable PreCommit test org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
[ https://issues.apache.org/jira/browse/HIVE-22616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-22616: - Attachment: TestFail.pdf > Disable PreCommit test org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 > - > > Key: HIVE-22616 > URL: https://issues.apache.org/jira/browse/HIVE-22616 > Project: Hive > Issue Type: Test > Components: Hive >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Attachments: TestFail.pdf > > > The test is flaky and produces following errors: > {{java.io.FileNotFoundException: Source > '/home/hiveptest/34.69.225.53-hiveptest-2/apache-github-source-source/itests/hive-unit/target/junit-qfile-results/clientpositive/join2.q.out' > does not exist}} > {{ at > org.apache.commons.io.FileUtils.checkFileRequirements(FileUtils.java:1383)}} > {{ at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1060)}} > {{ at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1028)}} > {{ at > org.apache.hadoop.hive.ql.QOutProcessor.maskPatterns(QOutProcessor.java:162)}} > {{ at > org.apache.hadoop.hive.ql.QTestUtil.checkCliDriverResults(QTestUtil.java:932)}} > {{ at > org.apache.hadoop.hive.ql.QTestRunnerUtils.queryListRunnerMultiThreaded(QTestRunnerUtils.java:152)}} > {{ at > org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1(TestMTQueries.java:55)}} > {{ at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}} > {{ at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}} > {{ at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}} > {{ at java.lang.reflect.Method.invoke(Method.java:498)}} > {{ at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)}} > {{ at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)}} > {{ at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)}} > {{ at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)}} > {{ at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)}} > {{ at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)}} > {{ at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)}} > {{ at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)}} > {{ at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)}} > {{ at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)}} > {{ at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)}} > {{ at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)}} > {{ at org.junit.runners.ParentRunner.run(ParentRunner.java:309)}} > {{ at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)}} > {{ at > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)}} > {{ at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)}} > {{ at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)}} > {{ at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)}} > {{ at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)}} > {{ at > org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)}} > {{ at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)}} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22616) Disable PreCommit test org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
[ https://issues.apache.org/jira/browse/HIVE-22616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993134#comment-16993134 ] Oliver Draese commented on HIVE-22616: -- Full test output in attachment TestFail.pdf > Disable PreCommit test org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 > - > > Key: HIVE-22616 > URL: https://issues.apache.org/jira/browse/HIVE-22616 > Project: Hive > Issue Type: Test > Components: Hive >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Attachments: TestFail.pdf > > > The test is flaky and produces following errors: > {{java.io.FileNotFoundException: Source > '/home/hiveptest/34.69.225.53-hiveptest-2/apache-github-source-source/itests/hive-unit/target/junit-qfile-results/clientpositive/join2.q.out' > does not exist}} > {{ at > org.apache.commons.io.FileUtils.checkFileRequirements(FileUtils.java:1383)}} > {{ at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1060)}} > {{ at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1028)}} > {{ at > org.apache.hadoop.hive.ql.QOutProcessor.maskPatterns(QOutProcessor.java:162)}} > {{ at > org.apache.hadoop.hive.ql.QTestUtil.checkCliDriverResults(QTestUtil.java:932)}} > {{ at > org.apache.hadoop.hive.ql.QTestRunnerUtils.queryListRunnerMultiThreaded(QTestRunnerUtils.java:152)}} > {{ at > org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1(TestMTQueries.java:55)}} > {{ at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}} > {{ at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}} > {{ at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}} > {{ at java.lang.reflect.Method.invoke(Method.java:498)}} > {{ at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)}} > {{ at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)}} > {{ at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)}} > {{ at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)}} > {{ at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)}} > {{ at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)}} > {{ at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)}} > {{ at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)}} > {{ at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)}} > {{ at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)}} > {{ at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)}} > {{ at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)}} > {{ at org.junit.runners.ParentRunner.run(ParentRunner.java:309)}} > {{ at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)}} > {{ at > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)}} > {{ at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)}} > {{ at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)}} > {{ at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)}} > {{ at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)}} > {{ at > org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)}} > {{ at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)}} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22616) Disable PreCommit test org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
[ https://issues.apache.org/jira/browse/HIVE-22616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese reassigned HIVE-22616: > Disable PreCommit test org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 > - > > Key: HIVE-22616 > URL: https://issues.apache.org/jira/browse/HIVE-22616 > Project: Hive > Issue Type: Test > Components: Hive >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > > The test is flaky and produces following errors: > {{java.io.FileNotFoundException: Source > '/home/hiveptest/34.69.225.53-hiveptest-2/apache-github-source-source/itests/hive-unit/target/junit-qfile-results/clientpositive/join2.q.out' > does not exist}} > {{ at > org.apache.commons.io.FileUtils.checkFileRequirements(FileUtils.java:1383)}} > {{ at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1060)}} > {{ at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1028)}} > {{ at > org.apache.hadoop.hive.ql.QOutProcessor.maskPatterns(QOutProcessor.java:162)}} > {{ at > org.apache.hadoop.hive.ql.QTestUtil.checkCliDriverResults(QTestUtil.java:932)}} > {{ at > org.apache.hadoop.hive.ql.QTestRunnerUtils.queryListRunnerMultiThreaded(QTestRunnerUtils.java:152)}} > {{ at > org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1(TestMTQueries.java:55)}} > {{ at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}} > {{ at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}} > {{ at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}} > {{ at java.lang.reflect.Method.invoke(Method.java:498)}} > {{ at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)}} > {{ at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)}} > {{ at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)}} > {{ at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)}} > {{ at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)}} > {{ at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)}} > {{ at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)}} > {{ at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)}} > {{ at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)}} > {{ at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)}} > {{ at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)}} > {{ at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)}} > {{ at org.junit.runners.ParentRunner.run(ParentRunner.java:309)}} > {{ at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)}} > {{ at > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)}} > {{ at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)}} > {{ at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)}} > {{ at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)}} > {{ at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)}} > {{ at > org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)}} > {{ at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)}} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-22572: - Attachment: HIVE-22572.6.patch > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.1.patch, HIVE-22572.2.patch, > HIVE-22572.3.patch, HIVE-22572.4.patch, HIVE-22572.5.patch, > HIVE-22572.6.patch, HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-22572: - Attachment: HIVE-22572.5.patch > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.1.patch, HIVE-22572.2.patch, > HIVE-22572.3.patch, HIVE-22572.4.patch, HIVE-22572.5.patch, HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993015#comment-16993015 ] Oliver Draese commented on HIVE-22572: -- Test failures are not related to patch. Re-attached patch to trigger new test run. > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.1.patch, HIVE-22572.2.patch, > HIVE-22572.3.patch, HIVE-22572.4.patch, HIVE-22572.5.patch, HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992894#comment-16992894 ] Oliver Draese commented on HIVE-22572: -- Test failures are not related to patch. Re-attached patch to trigger new test run. > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.1.patch, HIVE-22572.2.patch, > HIVE-22572.3.patch, HIVE-22572.4.patch, HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-22572: - Attachment: HIVE-22572.4.patch > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.1.patch, HIVE-22572.2.patch, > HIVE-22572.3.patch, HIVE-22572.4.patch, HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-22572: - Attachment: HIVE-22572.3.patch > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.1.patch, HIVE-22572.2.patch, > HIVE-22572.3.patch, HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16990193#comment-16990193 ] Oliver Draese commented on HIVE-22572: -- Test failures are not related to patch. Re-attached patch to trigger new test run. > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.1.patch, HIVE-22572.2.patch, > HIVE-22572.3.patch, HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989180#comment-16989180 ] Oliver Draese commented on HIVE-22572: -- Test failures are not related to patch. Re-attached patch to trigger new test run. > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.1.patch, HIVE-22572.2.patch, HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-22572: - Attachment: HIVE-22572.2.patch > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.1.patch, HIVE-22572.2.patch, HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22318) Java.io.exception:Two readers for
[ https://issues.apache.org/jira/browse/HIVE-22318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989163#comment-16989163 ] Oliver Draese commented on HIVE-22318: -- The fix seems to be in HIVE-16812. Minor compaction doesn't filter correctly by bucketID. You end up with multiple delete delta files, containing same tombstone RIDs, which leads to the above exception. > Java.io.exception:Two readers for > - > > Key: HIVE-22318 > URL: https://issues.apache.org/jira/browse/HIVE-22318 > Project: Hive > Issue Type: Bug > Components: Hive, HiveServer2 >Affects Versions: 3.1.0 >Reporter: max_c >Priority: Major > Attachments: hiveserver2 for exception.log > > > I create a ACID table with ORC format: > > {noformat} > CREATE TABLE `some.TableA`( > >) > ROW FORMAT SERDE >'org.apache.hadoop.hive.ql.io.orc.OrcSerde' > STORED AS INPUTFORMAT >'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' > OUTPUTFORMAT >'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' > TBLPROPERTIES ( >'bucketing_version'='2', >'orc.compress'='SNAPPY', >'transactional'='true', >'transactional_properties'='default'){noformat} > After executing merge into operation: > {noformat} > MERGE INTO some.TableA AS a USING (SELECT vend_no FROM some.TableB UNION ALL > SELECT vend_no FROM some.TableC) AS b ON a.vend_no=b.vend_no WHEN MATCHED > THEN DELETE > {noformat} > the problem happend(when selecting the TableA, the exception happens too): > {noformat} > java.io.IOException: java.io.IOException: Two readers for {originalWriteId: > 4, bucket: 536870912(1.0.0), row: 2434, currentWriteId 25}: new > [key={originalWriteId: 4, bucket: 536870912(1.0.0), row: 2434, currentWriteId > 25}, nextRecord={2, 4, 536870912, 2434, 25, null}, reader=Hive ORC > Reader(hdfs://hdpprod/warehouse/tablespace/managed/hive/some.db/tableA/delete_delta_015_026/bucket_1, > 9223372036854775807)], old [key={originalWriteId: 4, bucket: > 536870912(1.0.0), row: 2434, currentWriteId 25}, nextRecord={2, 4, 536870912, > 2434, 25, null}, reader=Hive ORC > Reader(hdfs://hdpprod/warehouse/tablespace/managed/hive/some.db/tableA/delete_delta_015_026/bucket_0{noformat} > Through orc_tools I scan all the > files(bucket_0,bucket_1,bucket_2) under delete_delta and find all > rows of files are the same.I think this will cause the same > key(RecordIdentifer) when scan the bucket_1 after bucket_0 but I > don't know why all the rows are the same in these bucket files. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22584) Flakyness in TestTaskExecutorService.testSetCapacity
[ https://issues.apache.org/jira/browse/HIVE-22584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988947#comment-16988947 ] Oliver Draese commented on HIVE-22584: -- +1 LGTM > Flakyness in TestTaskExecutorService.testSetCapacity > > > Key: HIVE-22584 > URL: https://issues.apache.org/jira/browse/HIVE-22584 > Project: Hive > Issue Type: Test >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-22584.patch > > > Very rarely the test fails: > {code} > java.lang.AssertionError: expected:<0> but was:<1> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.junit.Assert.assertEquals(Assert.java:542) > at > org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testSetCapacity(TestTaskExecutorService.java:515) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) > {code} > See: > https://builds.apache.org/job/PreCommit-HIVE-Build/19739/testReport/org.apache.hadoop.hive.llap.daemon.impl/TestTaskExecutorService/testSetCapacity/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988191#comment-16988191 ] Oliver Draese commented on HIVE-22572: -- UnitTests to be added via https://issues.apache.org/jira/browse/HIVE-22581 > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.1.patch, HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22581) Add tests for dynamic semijoins reduction
[ https://issues.apache.org/jira/browse/HIVE-22581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese reassigned HIVE-22581: > Add tests for dynamic semijoins reduction > - > > Key: HIVE-22581 > URL: https://issues.apache.org/jira/browse/HIVE-22581 > Project: Hive > Issue Type: Test > Components: Query Planning >Reporter: Oliver Draese >Assignee: Jesus Camacho Rodriguez >Priority: Major > > There don't seem to be tests for the TezCompiler. A test suite should be > added and a test for the error scenario of HIVE-22572 should be implemented > there. > > This is a follow-up action for: > https://issues.apache.org/jira/browse/HIVE-22572 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988093#comment-16988093 ] Oliver Draese commented on HIVE-22572: -- Test failures are not related to patch. Re-attached patch to trigger new test run. > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.1.patch, HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-22572: - Attachment: HIVE-22572.1.patch > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.1.patch, HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-22572: - Attachment: HIVE-22572.patch Status: Patch Available (was: In Progress) > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-22572: - Attachment: (was: HIVE-22572.patch) > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-22572: - Status: In Progress (was: Patch Available) > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-22572: - Attachment: HIVE-22572.patch Status: Patch Available (was: Open) > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22572.patch > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22572) NullPointerException when using dynamic semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-22572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese reassigned HIVE-22572: > NullPointerException when using dynamic semijoin reduction > -- > > Key: HIVE-22572 > URL: https://issues.apache.org/jira/browse/HIVE-22572 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > > From HS2 logs > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.TezCompiler.removeSemijoinOptimizationByBenefit(TezCompiler.java:1541) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.semijoinRemovalBasedTransformations(TezCompiler.java:471) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:182) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12487) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:360) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] > at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1869) > ~[hive-exec-3.1.0.3.1.0.142-1.jar:3.1.0.3.1.0.142-1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22113) Prevent LLAP shutdown on AMReporter related RuntimeException
[ https://issues.apache.org/jira/browse/HIVE-22113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16908405#comment-16908405 ] Oliver Draese commented on HIVE-22113: -- Created follow-up cleanup action as HIVE-22117 > Prevent LLAP shutdown on AMReporter related RuntimeException > > > Key: HIVE-22113 > URL: https://issues.apache.org/jira/browse/HIVE-22113 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Attachments: HIVE-22113.1.patch, HIVE-22113.2.patch, HIVE-22113.patch > > > If a task attempt cannot be removed from AMReporter (i.e. task attempt was > not found), the AMReporter throws a RuntimeException. This exception is not > caught and trickles up, causing an LLAP shutdown: > {{2019-08-08T23:34:39,748 ERROR [Wait-Queue-Scheduler-0 ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Thread Thread[Wait-Queue-Scheduler-0,5,main] threw an Exception. Shutting down now...}}{{java.lang.RuntimeException: attempt_1563528877295_18872_3728_01_03_0 was not registered and couldn't be removed}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.AMReporter$AMNodeInfo.removeTaskAttempt(AMReporter.java:524) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.AMReporter.unregisterTask(AMReporter.java:243) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskRunnerCallable.killTask(TaskRunnerCallable.java:384) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.handleScheduleAttemptedRejection(TaskExecutorService.java:739) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.access$1100(TaskExecutorService.java:91) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$WaitQueueWorker.run(TaskExecutorService.java:396) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_161]}}{{ > > at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) [hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]}}{{ > > at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) [hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]}}{{ > > at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) [hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]}}{{ > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]}}{{ > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]}}{{ > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]}} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (HIVE-22117) Clean up RuntimeException code in AMReporter
[ https://issues.apache.org/jira/browse/HIVE-22117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese reassigned HIVE-22117: > Clean up RuntimeException code in AMReporter > > > Key: HIVE-22117 > URL: https://issues.apache.org/jira/browse/HIVE-22117 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > > The AMReporter of LLAP throws RuntimExceptions from within addTaskAttempt and > removeTaskAttempt. These can cause LLAP to come down. > As an interims fix (see HIVE-22113), the RuntimeException of removeTaskAttemp > is caught from within TaskRunnerCallable, preventing LLAP termination if a > killed task is not found in AMReporter. > Ideally, we would just log this on removeTask (a gone task is a gone task) > and have a checked exception in addTaskAttempt. If the checkedException is > caught, we should fail the task attempt (as there is already an attempt with > this ID running). -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22113) Prevent LLAP shutdown on AMReporter related RuntimeException
[ https://issues.apache.org/jira/browse/HIVE-22113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-22113: - Attachment: HIVE-22113.2.patch > Prevent LLAP shutdown on AMReporter related RuntimeException > > > Key: HIVE-22113 > URL: https://issues.apache.org/jira/browse/HIVE-22113 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Attachments: HIVE-22113.1.patch, HIVE-22113.2.patch, HIVE-22113.patch > > > If a task attempt cannot be removed from AMReporter (i.e. task attempt was > not found), the AMReporter throws a RuntimeException. This exception is not > caught and trickles up, causing an LLAP shutdown: > {{2019-08-08T23:34:39,748 ERROR [Wait-Queue-Scheduler-0 ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Thread Thread[Wait-Queue-Scheduler-0,5,main] threw an Exception. Shutting down now...}}{{java.lang.RuntimeException: attempt_1563528877295_18872_3728_01_03_0 was not registered and couldn't be removed}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.AMReporter$AMNodeInfo.removeTaskAttempt(AMReporter.java:524) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.AMReporter.unregisterTask(AMReporter.java:243) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskRunnerCallable.killTask(TaskRunnerCallable.java:384) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.handleScheduleAttemptedRejection(TaskExecutorService.java:739) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.access$1100(TaskExecutorService.java:91) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$WaitQueueWorker.run(TaskExecutorService.java:396) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_161]}}{{ > > at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) [hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]}}{{ > > at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) [hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]}}{{ > > at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) [hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]}}{{ > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]}}{{ > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]}}{{ > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]}} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-22113) Prevent LLAP shutdown on AMReporter related RuntimeException
[ https://issues.apache.org/jira/browse/HIVE-22113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16908202#comment-16908202 ] Oliver Draese commented on HIVE-22113: -- Re-added attachment for repeated test execution. (Test failure in MiniDriver unrelated to fix) > Prevent LLAP shutdown on AMReporter related RuntimeException > > > Key: HIVE-22113 > URL: https://issues.apache.org/jira/browse/HIVE-22113 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Attachments: HIVE-22113.1.patch, HIVE-22113.2.patch, HIVE-22113.patch > > > If a task attempt cannot be removed from AMReporter (i.e. task attempt was > not found), the AMReporter throws a RuntimeException. This exception is not > caught and trickles up, causing an LLAP shutdown: > {{2019-08-08T23:34:39,748 ERROR [Wait-Queue-Scheduler-0 ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Thread Thread[Wait-Queue-Scheduler-0,5,main] threw an Exception. Shutting down now...}}{{java.lang.RuntimeException: attempt_1563528877295_18872_3728_01_03_0 was not registered and couldn't be removed}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.AMReporter$AMNodeInfo.removeTaskAttempt(AMReporter.java:524) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.AMReporter.unregisterTask(AMReporter.java:243) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskRunnerCallable.killTask(TaskRunnerCallable.java:384) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.handleScheduleAttemptedRejection(TaskExecutorService.java:739) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.access$1100(TaskExecutorService.java:91) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$WaitQueueWorker.run(TaskExecutorService.java:396) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_161]}}{{ > > at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) [hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]}}{{ > > at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) [hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]}}{{ > > at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) [hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]}}{{ > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]}}{{ > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]}}{{ > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]}} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22113) Prevent LLAP shutdown on AMReporter related RuntimeException
[ https://issues.apache.org/jira/browse/HIVE-22113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-22113: - Attachment: HIVE-22113.1.patch > Prevent LLAP shutdown on AMReporter related RuntimeException > > > Key: HIVE-22113 > URL: https://issues.apache.org/jira/browse/HIVE-22113 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Attachments: HIVE-22113.1.patch, HIVE-22113.patch > > > If a task attempt cannot be removed from AMReporter (i.e. task attempt was > not found), the AMReporter throws a RuntimeException. This exception is not > caught and trickles up, causing an LLAP shutdown: > {{2019-08-08T23:34:39,748 ERROR [Wait-Queue-Scheduler-0 ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Thread Thread[Wait-Queue-Scheduler-0,5,main] threw an Exception. Shutting down now...}}{{java.lang.RuntimeException: attempt_1563528877295_18872_3728_01_03_0 was not registered and couldn't be removed}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.AMReporter$AMNodeInfo.removeTaskAttempt(AMReporter.java:524) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.AMReporter.unregisterTask(AMReporter.java:243) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskRunnerCallable.killTask(TaskRunnerCallable.java:384) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.handleScheduleAttemptedRejection(TaskExecutorService.java:739) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.access$1100(TaskExecutorService.java:91) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$WaitQueueWorker.run(TaskExecutorService.java:396) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_161]}}{{ > > at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) [hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]}}{{ > > at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) [hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]}}{{ > > at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) [hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]}}{{ > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]}}{{ > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]}}{{ > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]}} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (HIVE-22113) Prevent LLAP shutdown on AMReporter related RuntimeException
[ https://issues.apache.org/jira/browse/HIVE-22113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-22113: - Attachment: HIVE-22113.patch Status: Patch Available (was: In Progress) > Prevent LLAP shutdown on AMReporter related RuntimeException > > > Key: HIVE-22113 > URL: https://issues.apache.org/jira/browse/HIVE-22113 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Attachments: HIVE-22113.patch > > > If a task attempt cannot be removed from AMReporter (i.e. task attempt was > not found), the AMReporter throws a RuntimeException. This exception is not > caught and trickles up, causing an LLAP shutdown: > {{2019-08-08T23:34:39,748 ERROR [Wait-Queue-Scheduler-0 ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Thread Thread[Wait-Queue-Scheduler-0,5,main] threw an Exception. Shutting down now...}}{{java.lang.RuntimeException: attempt_1563528877295_18872_3728_01_03_0 was not registered and couldn't be removed}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.AMReporter$AMNodeInfo.removeTaskAttempt(AMReporter.java:524) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.AMReporter.unregisterTask(AMReporter.java:243) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskRunnerCallable.killTask(TaskRunnerCallable.java:384) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.handleScheduleAttemptedRejection(TaskExecutorService.java:739) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.access$1100(TaskExecutorService.java:91) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$WaitQueueWorker.run(TaskExecutorService.java:396) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_161]}}{{ > > at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) [hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]}}{{ > > at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) [hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]}}{{ > > at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) [hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]}}{{ > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]}}{{ > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]}}{{ > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]}} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work started] (HIVE-22113) Prevent LLAP shutdown on AMReporter related RuntimeException
[ https://issues.apache.org/jira/browse/HIVE-22113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-22113 started by Oliver Draese. > Prevent LLAP shutdown on AMReporter related RuntimeException > > > Key: HIVE-22113 > URL: https://issues.apache.org/jira/browse/HIVE-22113 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Attachments: HIVE-22113.patch > > > If a task attempt cannot be removed from AMReporter (i.e. task attempt was > not found), the AMReporter throws a RuntimeException. This exception is not > caught and trickles up, causing an LLAP shutdown: > {{2019-08-08T23:34:39,748 ERROR [Wait-Queue-Scheduler-0 ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Thread Thread[Wait-Queue-Scheduler-0,5,main] threw an Exception. Shutting down now...}}{{java.lang.RuntimeException: attempt_1563528877295_18872_3728_01_03_0 was not registered and couldn't be removed}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.AMReporter$AMNodeInfo.removeTaskAttempt(AMReporter.java:524) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.AMReporter.unregisterTask(AMReporter.java:243) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskRunnerCallable.killTask(TaskRunnerCallable.java:384) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.handleScheduleAttemptedRejection(TaskExecutorService.java:739) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.access$1100(TaskExecutorService.java:91) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$WaitQueueWorker.run(TaskExecutorService.java:396) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_161]}}{{ > > at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) [hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]}}{{ > > at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) [hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]}}{{ > > at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) [hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]}}{{ > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]}}{{ > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]}}{{ > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]}} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (HIVE-22113) Prevent LLAP shutdown on AMReporter related RuntimeException
[ https://issues.apache.org/jira/browse/HIVE-22113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese reassigned HIVE-22113: > Prevent LLAP shutdown on AMReporter related RuntimeException > > > Key: HIVE-22113 > URL: https://issues.apache.org/jira/browse/HIVE-22113 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > > If a task attempt cannot be removed from AMReporter (i.e. task attempt was > not found), the AMReporter throws a RuntimeException. This exception is not > caught and trickles up, causing an LLAP shutdown: > {{2019-08-08T23:34:39,748 ERROR [Wait-Queue-Scheduler-0 ()] org.apache.hadoop.hive.llap.daemon.impl.LlapDaemon: Thread Thread[Wait-Queue-Scheduler-0,5,main] threw an Exception. Shutting down now...}}{{java.lang.RuntimeException: attempt_1563528877295_18872_3728_01_03_0 was not registered and couldn't be removed}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.AMReporter$AMNodeInfo.removeTaskAttempt(AMReporter.java:524) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.AMReporter.unregisterTask(AMReporter.java:243) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskRunnerCallable.killTask(TaskRunnerCallable.java:384) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.handleScheduleAttemptedRejection(TaskExecutorService.java:739) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService.access$1100(TaskExecutorService.java:91) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at org.apache.hadoop.hive.llap.daemon.impl.TaskExecutorService$WaitQueueWorker.run(TaskExecutorService.java:396) ~[hive-llap-server-3.1.0.3.1.0.103-1.jar:3.1.0.3.1.0.103-1]}}{{ > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_161]}}{{ > > at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) [hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]}}{{ > > at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) [hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]}}{{ > > at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) [hive-exec-3.1.0.3.1.0.103-1.jar:3.1.0-SNAPSHOT]}}{{ > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]}}{{ > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]}}{{ > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]}} -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21988) Do not consider nodes with 0 capacity when calculating host affinity
[ https://issues.apache.org/jira/browse/HIVE-21988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16884005#comment-16884005 ] Oliver Draese commented on HIVE-21988: -- +1 LGTM > Do not consider nodes with 0 capacity when calculating host affinity > > > Key: HIVE-21988 > URL: https://issues.apache.org/jira/browse/HIVE-21988 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21988.patch > > Time Spent: 40m > Remaining Estimate: 0h > > When a node is blacklisted (capacity set to 0) then we should not assign any > more tasks to it -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21908) LlapDaemon node status should be reflected in the metrics
[ https://issues.apache.org/jira/browse/HIVE-21908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16878046#comment-16878046 ] Oliver Draese commented on HIVE-21908: -- LGTM. +1 (see minor comments in PR) > LlapDaemon node status should be reflected in the metrics > - > > Key: HIVE-21908 > URL: https://issues.apache.org/jira/browse/HIVE-21908 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21908.01.patch, HIVE-21908.02.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > When we enable/disable a node it should be reflected in the LlapDaemon > metrics, so the administrator can act upon the disabled nodes. They can > manually check the status and either reenable them by restart, or fix the > existing issues causing the problems -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21949) Revert HIVE-21232 LLAP: Add a cache-miss friendly split affinity provider
[ https://issues.apache.org/jira/browse/HIVE-21949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16878029#comment-16878029 ] Oliver Draese commented on HIVE-21949: -- +1 > Revert HIVE-21232 LLAP: Add a cache-miss friendly split affinity provider > - > > Key: HIVE-21949 > URL: https://issues.apache.org/jira/browse/HIVE-21949 > Project: Hive > Issue Type: Bug >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Major > Attachments: HIVE-21949.01.patch > > > HDFS skew issues become LLAP skew issues and issues of skew need to be fixed > by running "hdfs balancer" instead of getting a uniform distribution via LLAP. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15177) Authentication with hive fails when kerberos auth type is set to fromSubject and principal contains _HOST
[ https://issues.apache.org/jira/browse/HIVE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16872473#comment-16872473 ] Oliver Draese commented on HIVE-15177: -- Trying rerun. Test case failure is unrelated to patch. > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST > - > > Key: HIVE-15177 > URL: https://issues.apache.org/jira/browse/HIVE-15177 > Project: Hive > Issue Type: Bug > Components: Authentication >Reporter: Subrahmanya >Assignee: Oliver Draese >Priority: Major > Fix For: 3.1.1 > > Attachments: HIVE-15177.1.patch, HIVE-15177.2.patch > > > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST. > When auth type is set to fromSubject, _HOST in principal is not resolved to > the actual host name even though the correct host name is available. This > leads to connection failure. If auth type is not set to fromSubject host > resolution is done correctly. > The problem is in getKerberosTransport method of > org.apache.hive.service.auth.KerberosSaslHelper class. When assumeSubject is > true host name in the principal is not resolved. When it is false, host name > is passed on to HadoopThriftAuthBridge, which takes care of resolving the > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-15177) Authentication with hive fails when kerberos auth type is set to fromSubject and principal contains _HOST
[ https://issues.apache.org/jira/browse/HIVE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-15177: - Attachment: HIVE-15177.2.patch > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST > - > > Key: HIVE-15177 > URL: https://issues.apache.org/jira/browse/HIVE-15177 > Project: Hive > Issue Type: Bug > Components: Authentication >Reporter: Subrahmanya >Assignee: Oliver Draese >Priority: Major > Fix For: 3.1.1 > > Attachments: HIVE-15177.1.patch, HIVE-15177.2.patch > > > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST. > When auth type is set to fromSubject, _HOST in principal is not resolved to > the actual host name even though the correct host name is available. This > leads to connection failure. If auth type is not set to fromSubject host > resolution is done correctly. > The problem is in getKerberosTransport method of > org.apache.hive.service.auth.KerberosSaslHelper class. When assumeSubject is > true host name in the principal is not resolved. When it is false, host name > is passed on to HadoopThriftAuthBridge, which takes care of resolving the > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-15177) Authentication with hive fails when kerberos auth type is set to fromSubject and principal contains _HOST
[ https://issues.apache.org/jira/browse/HIVE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-15177: - Attachment: HIVE-15177.1.patch > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST > - > > Key: HIVE-15177 > URL: https://issues.apache.org/jira/browse/HIVE-15177 > Project: Hive > Issue Type: Bug > Components: Authentication >Reporter: Subrahmanya >Assignee: Oliver Draese >Priority: Major > Fix For: 3.1.1 > > Attachments: HIVE-15177.1.patch > > > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST. > When auth type is set to fromSubject, _HOST in principal is not resolved to > the actual host name even though the correct host name is available. This > leads to connection failure. If auth type is not set to fromSubject host > resolution is done correctly. > The problem is in getKerberosTransport method of > org.apache.hive.service.auth.KerberosSaslHelper class. When assumeSubject is > true host name in the principal is not resolved. When it is false, host name > is passed on to HadoopThriftAuthBridge, which takes care of resolving the > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-15177) Authentication with hive fails when kerberos auth type is set to fromSubject and principal contains _HOST
[ https://issues.apache.org/jira/browse/HIVE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-15177: - Attachment: (was: HIVE-15177.patch) > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST > - > > Key: HIVE-15177 > URL: https://issues.apache.org/jira/browse/HIVE-15177 > Project: Hive > Issue Type: Bug > Components: Authentication >Reporter: Subrahmanya >Assignee: Oliver Draese >Priority: Major > Fix For: 3.1.1 > > Attachments: HIVE-15177.1.patch > > > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST. > When auth type is set to fromSubject, _HOST in principal is not resolved to > the actual host name even though the correct host name is available. This > leads to connection failure. If auth type is not set to fromSubject host > resolution is done correctly. > The problem is in getKerberosTransport method of > org.apache.hive.service.auth.KerberosSaslHelper class. When assumeSubject is > true host name in the principal is not resolved. When it is false, host name > is passed on to HadoopThriftAuthBridge, which takes care of resolving the > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15177) Authentication with hive fails when kerberos auth type is set to fromSubject and principal contains _HOST
[ https://issues.apache.org/jira/browse/HIVE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871769#comment-16871769 ] Oliver Draese commented on HIVE-15177: -- [https://github.com/apache/hive/pull/686] > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST > - > > Key: HIVE-15177 > URL: https://issues.apache.org/jira/browse/HIVE-15177 > Project: Hive > Issue Type: Bug > Components: Authentication >Reporter: Subrahmanya >Assignee: Oliver Draese >Priority: Major > Fix For: 3.1.1 > > Attachments: HIVE-15177.patch > > > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST. > When auth type is set to fromSubject, _HOST in principal is not resolved to > the actual host name even though the correct host name is available. This > leads to connection failure. If auth type is not set to fromSubject host > resolution is done correctly. > The problem is in getKerberosTransport method of > org.apache.hive.service.auth.KerberosSaslHelper class. When assumeSubject is > true host name in the principal is not resolved. When it is false, host name > is passed on to HadoopThriftAuthBridge, which takes care of resolving the > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-15177) Authentication with hive fails when kerberos auth type is set to fromSubject and principal contains _HOST
[ https://issues.apache.org/jira/browse/HIVE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-15177: - Fix Version/s: 3.1.1 Status: Patch Available (was: Open) Added resolving of _HOST for the fromSubject kerberos path as well. > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST > - > > Key: HIVE-15177 > URL: https://issues.apache.org/jira/browse/HIVE-15177 > Project: Hive > Issue Type: Bug > Components: Authentication >Reporter: Subrahmanya >Assignee: Oliver Draese >Priority: Major > Fix For: 3.1.1 > > Attachments: HIVE-15177.patch > > > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST. > When auth type is set to fromSubject, _HOST in principal is not resolved to > the actual host name even though the correct host name is available. This > leads to connection failure. If auth type is not set to fromSubject host > resolution is done correctly. > The problem is in getKerberosTransport method of > org.apache.hive.service.auth.KerberosSaslHelper class. When assumeSubject is > true host name in the principal is not resolved. When it is false, host name > is passed on to HadoopThriftAuthBridge, which takes care of resolving the > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-15177) Authentication with hive fails when kerberos auth type is set to fromSubject and principal contains _HOST
[ https://issues.apache.org/jira/browse/HIVE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese reassigned HIVE-15177: Assignee: Oliver Draese > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST > - > > Key: HIVE-15177 > URL: https://issues.apache.org/jira/browse/HIVE-15177 > Project: Hive > Issue Type: Bug > Components: Authentication >Reporter: Subrahmanya >Assignee: Oliver Draese >Priority: Major > Attachments: HIVE-15177.patch > > > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST. > When auth type is set to fromSubject, _HOST in principal is not resolved to > the actual host name even though the correct host name is available. This > leads to connection failure. If auth type is not set to fromSubject host > resolution is done correctly. > The problem is in getKerberosTransport method of > org.apache.hive.service.auth.KerberosSaslHelper class. When assumeSubject is > true host name in the principal is not resolved. When it is false, host name > is passed on to HadoopThriftAuthBridge, which takes care of resolving the > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-15177) Authentication with hive fails when kerberos auth type is set to fromSubject and principal contains _HOST
[ https://issues.apache.org/jira/browse/HIVE-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-15177: - Attachment: HIVE-15177.patch > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST > - > > Key: HIVE-15177 > URL: https://issues.apache.org/jira/browse/HIVE-15177 > Project: Hive > Issue Type: Bug > Components: Authentication >Reporter: Subrahmanya >Assignee: Oliver Draese >Priority: Major > Attachments: HIVE-15177.patch > > > Authentication with hive fails when kerberos auth type is set to fromSubject > and principal contains _HOST. > When auth type is set to fromSubject, _HOST in principal is not resolved to > the actual host name even though the correct host name is available. This > leads to connection failure. If auth type is not set to fromSubject host > resolution is done correctly. > The problem is in getKerberosTransport method of > org.apache.hive.service.auth.KerberosSaslHelper class. When assumeSubject is > true host name in the principal is not resolved. When it is false, host name > is passed on to HadoopThriftAuthBridge, which takes care of resolving the > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21838) Hive Metastore Translation: Add API call to tell client why table has limited access
[ https://issues.apache.org/jira/browse/HIVE-21838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16861645#comment-16861645 ] Oliver Draese commented on HIVE-21838: -- No concerns - this sounds very reasonable. Thanks! > Hive Metastore Translation: Add API call to tell client why table has limited > access > > > Key: HIVE-21838 > URL: https://issues.apache.org/jira/browse/HIVE-21838 > Project: Hive > Issue Type: Sub-task >Reporter: Yongzhi Chen >Assignee: Naveen Gangam >Priority: Major > > When a table access type is Read-only or None, we need a way to tell clients > why. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21785) Add task queue/runtime stats per LLAP daemon to output
[ https://issues.apache.org/jira/browse/HIVE-21785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16856043#comment-16856043 ] Oliver Draese commented on HIVE-21785: -- re - added (same) patch to trigger test run again. test errors were unrelated. > Add task queue/runtime stats per LLAP daemon to output > -- > > Key: HIVE-21785 > URL: https://issues.apache.org/jira/browse/HIVE-21785 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.1.1 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 3.1.1 > > Attachments: HIVE-21785.1.patch, HIVE-21785.2.patch, HIVE-21785.patch > > > There are several scenarios, where we want to investigate if a particular > LLAP daemon is performing faster or slower than the others in the cluster. In > these scenarios, it is specifically important to figure out if tasks spent > significant time, waiting for an available executor (queued) vs. on the > execution itself. Also, a skew in task-to-daemon assignment is interesting. > This patch adds these statistics to the TezCounters and therefore to the job > output on a per LLAP daemon base. Here is an example. > {{INFO : LlapTaskRuntimeAgg by daemon:}} > {{INFO : Count-host-1.example.com: 41}} > {{INFO : Count-host-2.example.com: 39}} > {{INFO : Count-host-3.example.com: 45}} > {{INFO : QueueTime-host-1.example.com: 51437776}} > {{INFO : QueueTime-host-2.example.com: 35758306}} > {{INFO : QueueTime-host-3.example.com: 47168327}} > {{INFO : RunTime-host-1.example.com: 165151539295}} > {{INFO : RunTime-host-2.example.com: 141729193528}} > {{INFO : RunTime-host-3.example.com: 166876988771}} > The "Count-" are simple task counts for the appended host name (LLAP daemon) > The "QueueTime-" values tell, how long tasks waited in the > TaskExecutorService's queue before getting actually executed. > The "RunTime-" values cover the time from execution start to finish (where > finish can either be successful execution or a killed/failed execution). > For the new counts to appear in the output, both - the preexisting > hive.tez.exec.print.summary and the new hive.llap.task.time.print.summary > have to be set to true. > > {{}} > {{ hive.tez.exec.print.summary}} > {{ true}} > {{}} > {{}} > {{ hive.llap.task.time.print.summary}} > {{ true}} > {{}} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21785) Add task queue/runtime stats per LLAP daemon to output
[ https://issues.apache.org/jira/browse/HIVE-21785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21785: - Attachment: HIVE-21785.2.patch > Add task queue/runtime stats per LLAP daemon to output > -- > > Key: HIVE-21785 > URL: https://issues.apache.org/jira/browse/HIVE-21785 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.1.1 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 3.1.1 > > Attachments: HIVE-21785.1.patch, HIVE-21785.2.patch, HIVE-21785.patch > > > There are several scenarios, where we want to investigate if a particular > LLAP daemon is performing faster or slower than the others in the cluster. In > these scenarios, it is specifically important to figure out if tasks spent > significant time, waiting for an available executor (queued) vs. on the > execution itself. Also, a skew in task-to-daemon assignment is interesting. > This patch adds these statistics to the TezCounters and therefore to the job > output on a per LLAP daemon base. Here is an example. > {{INFO : LlapTaskRuntimeAgg by daemon:}} > {{INFO : Count-host-1.example.com: 41}} > {{INFO : Count-host-2.example.com: 39}} > {{INFO : Count-host-3.example.com: 45}} > {{INFO : QueueTime-host-1.example.com: 51437776}} > {{INFO : QueueTime-host-2.example.com: 35758306}} > {{INFO : QueueTime-host-3.example.com: 47168327}} > {{INFO : RunTime-host-1.example.com: 165151539295}} > {{INFO : RunTime-host-2.example.com: 141729193528}} > {{INFO : RunTime-host-3.example.com: 166876988771}} > The "Count-" are simple task counts for the appended host name (LLAP daemon) > The "QueueTime-" values tell, how long tasks waited in the > TaskExecutorService's queue before getting actually executed. > The "RunTime-" values cover the time from execution start to finish (where > finish can either be successful execution or a killed/failed execution). > For the new counts to appear in the output, both - the preexisting > hive.tez.exec.print.summary and the new hive.llap.task.time.print.summary > have to be set to true. > > {{}} > {{ hive.tez.exec.print.summary}} > {{ true}} > {{}} > {{}} > {{ hive.llap.task.time.print.summary}} > {{ true}} > {{}} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21785) Add task queue/runtime stats per LLAP daemon to output
[ https://issues.apache.org/jira/browse/HIVE-21785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21785: - Attachment: HIVE-21785.1.patch > Add task queue/runtime stats per LLAP daemon to output > -- > > Key: HIVE-21785 > URL: https://issues.apache.org/jira/browse/HIVE-21785 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.1.1 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 3.1.1 > > Attachments: HIVE-21785.1.patch, HIVE-21785.patch > > > There are several scenarios, where we want to investigate if a particular > LLAP daemon is performing faster or slower than the others in the cluster. In > these scenarios, it is specifically important to figure out if tasks spent > significant time, waiting for an available executor (queued) vs. on the > execution itself. Also, a skew in task-to-daemon assignment is interesting. > This patch adds these statistics to the TezCounters and therefore to the job > output on a per LLAP daemon base. Here is an example. > {{INFO : LlapTaskRuntimeAgg by daemon:}} > {{INFO : Count-host-1.example.com: 41}} > {{INFO : Count-host-2.example.com: 39}} > {{INFO : Count-host-3.example.com: 45}} > {{INFO : QueueTime-host-1.example.com: 51437776}} > {{INFO : QueueTime-host-2.example.com: 35758306}} > {{INFO : QueueTime-host-3.example.com: 47168327}} > {{INFO : RunTime-host-1.example.com: 165151539295}} > {{INFO : RunTime-host-2.example.com: 141729193528}} > {{INFO : RunTime-host-3.example.com: 166876988771}} > The "Count-" are simple task counts for the appended host name (LLAP daemon) > The "QueueTime-" values tell, how long tasks waited in the > TaskExecutorService's queue before getting actually executed. > The "RunTime-" values cover the time from execution start to finish (where > finish can either be successful execution or a killed/failed execution). > For the new counts to appear in the output, both - the preexisting > hive.tez.exec.print.summary and the new hive.llap.task.time.print.summary > have to be set to true. > > {{}} > {{ hive.tez.exec.print.summary}} > {{ true}} > {{}} > {{}} > {{ hive.llap.task.time.print.summary}} > {{ true}} > {{}} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21785) Add task queue/runtime stats per LLAP daemon to output
[ https://issues.apache.org/jira/browse/HIVE-21785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21785: - Status: Patch Available (was: Open) > Add task queue/runtime stats per LLAP daemon to output > -- > > Key: HIVE-21785 > URL: https://issues.apache.org/jira/browse/HIVE-21785 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.1.1 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 3.1.1 > > Attachments: HIVE-21785.patch > > > There are several scenarios, where we want to investigate if a particular > LLAP daemon is performing faster or slower than the others in the cluster. In > these scenarios, it is specifically important to figure out if tasks spent > significant time, waiting for an available executor (queued) vs. on the > execution itself. Also, a skew in task-to-daemon assignment is interesting. > This patch adds these statistics to the TezCounters and therefore to the job > output on a per LLAP daemon base. Here is an example. > {{INFO : LlapTaskRuntimeAgg by daemon:}} > {{INFO : Count-host-1.example.com: 41}} > {{INFO : Count-host-2.example.com: 39}} > {{INFO : Count-host-3.example.com: 45}} > {{INFO : QueueTime-host-1.example.com: 51437776}} > {{INFO : QueueTime-host-2.example.com: 35758306}} > {{INFO : QueueTime-host-3.example.com: 47168327}} > {{INFO : RunTime-host-1.example.com: 165151539295}} > {{INFO : RunTime-host-2.example.com: 141729193528}} > {{INFO : RunTime-host-3.example.com: 166876988771}} > The "Count-" are simple task counts for the appended host name (LLAP daemon) > The "QueueTime-" values tell, how long tasks waited in the > TaskExecutorService's queue before getting actually executed. > The "RunTime-" values cover the time from execution start to finish (where > finish can either be successful execution or a killed/failed execution). > For the new counts to appear in the output, both - the preexisting > hive.tez.exec.print.summary and the new hive.llap.task.time.print.summary > have to be set to true. > > {{}} > {{ hive.tez.exec.print.summary}} > {{ true}} > {{}} > {{}} > {{ hive.llap.task.time.print.summary}} > {{ true}} > {{}} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21785) Add task queue/runtime stats per LLAP daemon to output
[ https://issues.apache.org/jira/browse/HIVE-21785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21785: - Attachment: HIVE-21785.patch > Add task queue/runtime stats per LLAP daemon to output > -- > > Key: HIVE-21785 > URL: https://issues.apache.org/jira/browse/HIVE-21785 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.1.1 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 3.1.1 > > Attachments: HIVE-21785.patch > > > There are several scenarios, where we want to investigate if a particular > LLAP daemon is performing faster or slower than the others in the cluster. In > these scenarios, it is specifically important to figure out if tasks spent > significant time, waiting for an available executor (queued) vs. on the > execution itself. Also, a skew in task-to-daemon assignment is interesting. > This patch adds these statistics to the TezCounters and therefore to the job > output on a per LLAP daemon base. Here is an example. > {{INFO : LlapTaskRuntimeAgg by daemon:}} > {{INFO : Count-host-1.example.com: 41}} > {{INFO : Count-host-2.example.com: 39}} > {{INFO : Count-host-3.example.com: 45}} > {{INFO : QueueTime-host-1.example.com: 51437776}} > {{INFO : QueueTime-host-2.example.com: 35758306}} > {{INFO : QueueTime-host-3.example.com: 47168327}} > {{INFO : RunTime-host-1.example.com: 165151539295}} > {{INFO : RunTime-host-2.example.com: 141729193528}} > {{INFO : RunTime-host-3.example.com: 166876988771}} > The "Count-" are simple task counts for the appended host name (LLAP daemon) > The "QueueTime-" values tell, how long tasks waited in the > TaskExecutorService's queue before getting actually executed. > The "RunTime-" values cover the time from execution start to finish (where > finish can either be successful execution or a killed/failed execution). > For the new counts to appear in the output, both - the preexisting > hive.tez.exec.print.summary and the new hive.llap.task.time.print.summary > have to be set to true. > > {{}} > {{ hive.tez.exec.print.summary}} > {{ true}} > {{}} > {{}} > {{ hive.llap.task.time.print.summary}} > {{ true}} > {{}} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21785) Add task queue/runtime stats per LLAP daemon to output
[ https://issues.apache.org/jira/browse/HIVE-21785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese reassigned HIVE-21785: > Add task queue/runtime stats per LLAP daemon to output > -- > > Key: HIVE-21785 > URL: https://issues.apache.org/jira/browse/HIVE-21785 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 3.1.1 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 3.1.1 > > > There are several scenarios, where we want to investigate if a particular > LLAP daemon is performing faster or slower than the others in the cluster. In > these scenarios, it is specifically important to figure out if tasks spent > significant time, waiting for an available executor (queued) vs. on the > execution itself. Also, a skew in task-to-daemon assignment is interesting. > This patch adds these statistics to the TezCounters and therefore to the job > output on a per LLAP daemon base. Here is an example. > {{INFO : LlapTaskRuntimeAgg by daemon:}} > {{INFO : Count-host-1.example.com: 41}} > {{INFO : Count-host-2.example.com: 39}} > {{INFO : Count-host-3.example.com: 45}} > {{INFO : QueueTime-host-1.example.com: 51437776}} > {{INFO : QueueTime-host-2.example.com: 35758306}} > {{INFO : QueueTime-host-3.example.com: 47168327}} > {{INFO : RunTime-host-1.example.com: 165151539295}} > {{INFO : RunTime-host-2.example.com: 141729193528}} > {{INFO : RunTime-host-3.example.com: 166876988771}} > The "Count-" are simple task counts for the appended host name (LLAP daemon) > The "QueueTime-" values tell, how long tasks waited in the > TaskExecutorService's queue before getting actually executed. > The "RunTime-" values cover the time from execution start to finish (where > finish can either be successful execution or a killed/failed execution). > For the new counts to appear in the output, both - the preexisting > hive.tez.exec.print.summary and the new hive.llap.task.time.print.summary > have to be set to true. > > {{}} > {{ hive.tez.exec.print.summary}} > {{ true}} > {{}} > {{}} > {{ hive.llap.task.time.print.summary}} > {{ true}} > {{}} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21204) Instrumentation for read/write locks in LLAP
[ https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21204: - Attachment: HIVE-21204.6.patch > Instrumentation for read/write locks in LLAP > > > Key: HIVE-21204 > URL: https://issues.apache.org/jira/browse/HIVE-21204 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Attachments: HIVE-21204.4.patch, HIVE-21204.5.patch, > HIVE-21204.6.patch > > > LLAP has several R/W locks for serialization of updates to query tracker, > file data, > Instrumentation is added to monitor the > * total amount of R/W locks within a particular category > * average + max wait/suspension time to get the R/W lock > A category includes all lock instances for particular areas (i.e. category is > FileData and all R/W locks that are used in FileData instances are accounted > within the one category). > The monitoring/accounting is done via Hadoop Metrics 2, making them > accessible via JMX. In addition, a new "locking" GET endpoint is added to the > LLAP daemon's REST interface. It produces output like the following example: > { > {{ "statsCollection": "enabled",}} > {{ "lockStats": [}} > {{ {}}{{ "type": "R/W Lock Stats",}} > {{ "label": "FileData",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ },}} > {{ { "}}{{type": "R/W Lock Stats",}} > {{ "label": "QueryTracker",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ } }}{{]}} > {{}}} > To avoid the overhead of lock instrumentation, lock metrics collection is > disabled by default and can be enabled via the following configuration > parameter: > {{hive.llap.lockmetrics.collect = true}} > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21204) Instrumentation for read/write locks in LLAP
[ https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21204: - Attachment: (was: HIVE-21204.5.patch) > Instrumentation for read/write locks in LLAP > > > Key: HIVE-21204 > URL: https://issues.apache.org/jira/browse/HIVE-21204 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Attachments: HIVE-21204.4.patch, HIVE-21204.5.patch > > > LLAP has several R/W locks for serialization of updates to query tracker, > file data, > Instrumentation is added to monitor the > * total amount of R/W locks within a particular category > * average + max wait/suspension time to get the R/W lock > A category includes all lock instances for particular areas (i.e. category is > FileData and all R/W locks that are used in FileData instances are accounted > within the one category). > The monitoring/accounting is done via Hadoop Metrics 2, making them > accessible via JMX. In addition, a new "locking" GET endpoint is added to the > LLAP daemon's REST interface. It produces output like the following example: > { > {{ "statsCollection": "enabled",}} > {{ "lockStats": [}} > {{ {}}{{ "type": "R/W Lock Stats",}} > {{ "label": "FileData",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ },}} > {{ { "}}{{type": "R/W Lock Stats",}} > {{ "label": "QueryTracker",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ } }}{{]}} > {{}}} > To avoid the overhead of lock instrumentation, lock metrics collection is > disabled by default and can be enabled via the following configuration > parameter: > {{hive.llap.lockmetrics.collect = true}} > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21204) Instrumentation for read/write locks in LLAP
[ https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21204: - Attachment: HIVE-21204.5.patch > Instrumentation for read/write locks in LLAP > > > Key: HIVE-21204 > URL: https://issues.apache.org/jira/browse/HIVE-21204 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Attachments: HIVE-21204.4.patch, HIVE-21204.5.patch > > > LLAP has several R/W locks for serialization of updates to query tracker, > file data, > Instrumentation is added to monitor the > * total amount of R/W locks within a particular category > * average + max wait/suspension time to get the R/W lock > A category includes all lock instances for particular areas (i.e. category is > FileData and all R/W locks that are used in FileData instances are accounted > within the one category). > The monitoring/accounting is done via Hadoop Metrics 2, making them > accessible via JMX. In addition, a new "locking" GET endpoint is added to the > LLAP daemon's REST interface. It produces output like the following example: > { > {{ "statsCollection": "enabled",}} > {{ "lockStats": [}} > {{ {}}{{ "type": "R/W Lock Stats",}} > {{ "label": "FileData",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ },}} > {{ { "}}{{type": "R/W Lock Stats",}} > {{ "label": "QueryTracker",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ } }}{{]}} > {{}}} > To avoid the overhead of lock instrumentation, lock metrics collection is > disabled by default and can be enabled via the following configuration > parameter: > {{hive.llap.lockmetrics.collect = true}} > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21204) Instrumentation for read/write locks in LLAP
[ https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21204: - Attachment: (was: HIVE-21204.5.patch) > Instrumentation for read/write locks in LLAP > > > Key: HIVE-21204 > URL: https://issues.apache.org/jira/browse/HIVE-21204 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Attachments: HIVE-21204.4.patch, HIVE-21204.5.patch > > > LLAP has several R/W locks for serialization of updates to query tracker, > file data, > Instrumentation is added to monitor the > * total amount of R/W locks within a particular category > * average + max wait/suspension time to get the R/W lock > A category includes all lock instances for particular areas (i.e. category is > FileData and all R/W locks that are used in FileData instances are accounted > within the one category). > The monitoring/accounting is done via Hadoop Metrics 2, making them > accessible via JMX. In addition, a new "locking" GET endpoint is added to the > LLAP daemon's REST interface. It produces output like the following example: > { > {{ "statsCollection": "enabled",}} > {{ "lockStats": [}} > {{ {}}{{ "type": "R/W Lock Stats",}} > {{ "label": "FileData",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ },}} > {{ { "}}{{type": "R/W Lock Stats",}} > {{ "label": "QueryTracker",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ } }}{{]}} > {{}}} > To avoid the overhead of lock instrumentation, lock metrics collection is > disabled by default and can be enabled via the following configuration > parameter: > {{hive.llap.lockmetrics.collect = true}} > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21204) Instrumentation for read/write locks in LLAP
[ https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21204: - Attachment: HIVE-21204.5.patch > Instrumentation for read/write locks in LLAP > > > Key: HIVE-21204 > URL: https://issues.apache.org/jira/browse/HIVE-21204 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Attachments: HIVE-21204.4.patch, HIVE-21204.5.patch > > > LLAP has several R/W locks for serialization of updates to query tracker, > file data, > Instrumentation is added to monitor the > * total amount of R/W locks within a particular category > * average + max wait/suspension time to get the R/W lock > A category includes all lock instances for particular areas (i.e. category is > FileData and all R/W locks that are used in FileData instances are accounted > within the one category). > The monitoring/accounting is done via Hadoop Metrics 2, making them > accessible via JMX. In addition, a new "locking" GET endpoint is added to the > LLAP daemon's REST interface. It produces output like the following example: > { > {{ "statsCollection": "enabled",}} > {{ "lockStats": [}} > {{ {}}{{ "type": "R/W Lock Stats",}} > {{ "label": "FileData",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ },}} > {{ { "}}{{type": "R/W Lock Stats",}} > {{ "label": "QueryTracker",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ } }}{{]}} > {{}}} > To avoid the overhead of lock instrumentation, lock metrics collection is > disabled by default and can be enabled via the following configuration > parameter: > {{hive.llap.lockmetrics.collect = true}} > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21183) Interrupt wait time for FileCacheCleanupThread
[ https://issues.apache.org/jira/browse/HIVE-21183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21183: - Attachment: HIVE-21183.2.patch > Interrupt wait time for FileCacheCleanupThread > -- > > Key: HIVE-21183 > URL: https://issues.apache.org/jira/browse/HIVE-21183 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Minor > Attachments: HIVE-21183.1.patch, HIVE-21183.2.patch, HIVE-21183.patch > > > The FileCacheCleanupThread is waiting unnecessarily long for eviction counts > to increment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21183) Interrupt wait time for FileCacheCleanupThread
[ https://issues.apache.org/jira/browse/HIVE-21183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21183: - Attachment: (was: HIVE-21183.2.patch) > Interrupt wait time for FileCacheCleanupThread > -- > > Key: HIVE-21183 > URL: https://issues.apache.org/jira/browse/HIVE-21183 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Minor > Attachments: HIVE-21183.1.patch, HIVE-21183.2.patch, HIVE-21183.patch > > > The FileCacheCleanupThread is waiting unnecessarily long for eviction counts > to increment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21493) BuddyAllocator - Metrics count for allocated arenas wrong if preallocation is done
[ https://issues.apache.org/jira/browse/HIVE-21493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21493: - Status: Patch Available (was: Open) > BuddyAllocator - Metrics count for allocated arenas wrong if preallocation is > done > -- > > Key: HIVE-21493 > URL: https://issues.apache.org/jira/browse/HIVE-21493 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Trivial > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21493.patch > > > The (Hadoop/JMX) metrics are not correctly initialized if arena preallocation > is done and the arena count is greater than 1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21493) BuddyAllocator - Metrics count for allocated arenas wrong if preallocation is done
[ https://issues.apache.org/jira/browse/HIVE-21493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21493: - Attachment: HIVE-21493.patch > BuddyAllocator - Metrics count for allocated arenas wrong if preallocation is > done > -- > > Key: HIVE-21493 > URL: https://issues.apache.org/jira/browse/HIVE-21493 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Trivial > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21493.patch > > > The (Hadoop/JMX) metrics are not correctly initialized if arena preallocation > is done and the arena count is greater than 1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21493) BuddyAllocator - Metrics count for allocated arenas wrong if preallocation is done
[ https://issues.apache.org/jira/browse/HIVE-21493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese reassigned HIVE-21493: > BuddyAllocator - Metrics count for allocated arenas wrong if preallocation is > done > -- > > Key: HIVE-21493 > URL: https://issues.apache.org/jira/browse/HIVE-21493 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.1 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Trivial > Labels: llap > Fix For: 4.0.0 > > > The (Hadoop/JMX) metrics are not correctly initialized if arena preallocation > is done and the arena count is greater than 1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21422: - Attachment: (was: HIVE-21422.2.patch) > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21422.1.patch, HIVE-21422.2.patch, HIVE-21422.patch > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21422: - Attachment: HIVE-21422.2.patch > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21422.1.patch, HIVE-21422.2.patch, HIVE-21422.patch > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21422: - Attachment: (was: HIVE-21422.2.patch) > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21422.1.patch, HIVE-21422.2.patch, HIVE-21422.patch > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21183) Interrupt wait time for FileCacheCleanupThread
[ https://issues.apache.org/jira/browse/HIVE-21183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21183: - Attachment: (was: HIVE-21183.2.patch) > Interrupt wait time for FileCacheCleanupThread > -- > > Key: HIVE-21183 > URL: https://issues.apache.org/jira/browse/HIVE-21183 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Minor > Attachments: HIVE-21183.1.patch, HIVE-21183.2.patch, HIVE-21183.patch > > > The FileCacheCleanupThread is waiting unnecessarily long for eviction counts > to increment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21183) Interrupt wait time for FileCacheCleanupThread
[ https://issues.apache.org/jira/browse/HIVE-21183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21183: - Attachment: (was: HIVE-21183.2.patch) > Interrupt wait time for FileCacheCleanupThread > -- > > Key: HIVE-21183 > URL: https://issues.apache.org/jira/browse/HIVE-21183 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Minor > Attachments: HIVE-21183.1.patch, HIVE-21183.2.patch, HIVE-21183.patch > > > The FileCacheCleanupThread is waiting unnecessarily long for eviction counts > to increment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21183) Interrupt wait time for FileCacheCleanupThread
[ https://issues.apache.org/jira/browse/HIVE-21183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21183: - Attachment: HIVE-21183.2.patch > Interrupt wait time for FileCacheCleanupThread > -- > > Key: HIVE-21183 > URL: https://issues.apache.org/jira/browse/HIVE-21183 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Minor > Attachments: HIVE-21183.1.patch, HIVE-21183.2.patch, HIVE-21183.patch > > > The FileCacheCleanupThread is waiting unnecessarily long for eviction counts > to increment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21204) Instrumentation for read/write locks in LLAP
[ https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21204: - Attachment: HIVE-21204.5.patch > Instrumentation for read/write locks in LLAP > > > Key: HIVE-21204 > URL: https://issues.apache.org/jira/browse/HIVE-21204 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Attachments: HIVE-21204.4.patch, HIVE-21204.5.patch > > > LLAP has several R/W locks for serialization of updates to query tracker, > file data, > Instrumentation is added to monitor the > * total amount of R/W locks within a particular category > * average + max wait/suspension time to get the R/W lock > A category includes all lock instances for particular areas (i.e. category is > FileData and all R/W locks that are used in FileData instances are accounted > within the one category). > The monitoring/accounting is done via Hadoop Metrics 2, making them > accessible via JMX. In addition, a new "locking" GET endpoint is added to the > LLAP daemon's REST interface. It produces output like the following example: > { > {{ "statsCollection": "enabled",}} > {{ "lockStats": [}} > {{ {}}{{ "type": "R/W Lock Stats",}} > {{ "label": "FileData",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ },}} > {{ { "}}{{type": "R/W Lock Stats",}} > {{ "label": "QueryTracker",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ } }}{{]}} > {{}}} > To avoid the overhead of lock instrumentation, lock metrics collection is > disabled by default and can be enabled via the following configuration > parameter: > {{hive.llap.lockmetrics.collect = true}} > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21204) Instrumentation for read/write locks in LLAP
[ https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21204: - Attachment: (was: HIVE-21204.5.patch) > Instrumentation for read/write locks in LLAP > > > Key: HIVE-21204 > URL: https://issues.apache.org/jira/browse/HIVE-21204 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Attachments: HIVE-21204.4.patch > > > LLAP has several R/W locks for serialization of updates to query tracker, > file data, > Instrumentation is added to monitor the > * total amount of R/W locks within a particular category > * average + max wait/suspension time to get the R/W lock > A category includes all lock instances for particular areas (i.e. category is > FileData and all R/W locks that are used in FileData instances are accounted > within the one category). > The monitoring/accounting is done via Hadoop Metrics 2, making them > accessible via JMX. In addition, a new "locking" GET endpoint is added to the > LLAP daemon's REST interface. It produces output like the following example: > { > {{ "statsCollection": "enabled",}} > {{ "lockStats": [}} > {{ {}}{{ "type": "R/W Lock Stats",}} > {{ "label": "FileData",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ },}} > {{ { "}}{{type": "R/W Lock Stats",}} > {{ "label": "QueryTracker",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ } }}{{]}} > {{}}} > To avoid the overhead of lock instrumentation, lock metrics collection is > disabled by default and can be enabled via the following configuration > parameter: > {{hive.llap.lockmetrics.collect = true}} > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21204) Instrumentation for read/write locks in LLAP
[ https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21204: - Attachment: HIVE-21204.5.patch > Instrumentation for read/write locks in LLAP > > > Key: HIVE-21204 > URL: https://issues.apache.org/jira/browse/HIVE-21204 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Attachments: HIVE-21204.4.patch, HIVE-21204.5.patch > > > LLAP has several R/W locks for serialization of updates to query tracker, > file data, > Instrumentation is added to monitor the > * total amount of R/W locks within a particular category > * average + max wait/suspension time to get the R/W lock > A category includes all lock instances for particular areas (i.e. category is > FileData and all R/W locks that are used in FileData instances are accounted > within the one category). > The monitoring/accounting is done via Hadoop Metrics 2, making them > accessible via JMX. In addition, a new "locking" GET endpoint is added to the > LLAP daemon's REST interface. It produces output like the following example: > { > {{ "statsCollection": "enabled",}} > {{ "lockStats": [}} > {{ {}}{{ "type": "R/W Lock Stats",}} > {{ "label": "FileData",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ },}} > {{ { "}}{{type": "R/W Lock Stats",}} > {{ "label": "QueryTracker",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ } }}{{]}} > {{}}} > To avoid the overhead of lock instrumentation, lock metrics collection is > disabled by default and can be enabled via the following configuration > parameter: > {{hive.llap.lockmetrics.collect = true}} > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21204) Instrumentation for read/write locks in LLAP
[ https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21204: - Attachment: (was: HIVE-21204.2.patch) > Instrumentation for read/write locks in LLAP > > > Key: HIVE-21204 > URL: https://issues.apache.org/jira/browse/HIVE-21204 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Attachments: HIVE-21204.4.patch > > > LLAP has several R/W locks for serialization of updates to query tracker, > file data, > Instrumentation is added to monitor the > * total amount of R/W locks within a particular category > * average + max wait/suspension time to get the R/W lock > A category includes all lock instances for particular areas (i.e. category is > FileData and all R/W locks that are used in FileData instances are accounted > within the one category). > The monitoring/accounting is done via Hadoop Metrics 2, making them > accessible via JMX. In addition, a new "locking" GET endpoint is added to the > LLAP daemon's REST interface. It produces output like the following example: > { > {{ "statsCollection": "enabled",}} > {{ "lockStats": [}} > {{ {}}{{ "type": "R/W Lock Stats",}} > {{ "label": "FileData",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ },}} > {{ { "}}{{type": "R/W Lock Stats",}} > {{ "label": "QueryTracker",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ } }}{{]}} > {{}}} > To avoid the overhead of lock instrumentation, lock metrics collection is > disabled by default and can be enabled via the following configuration > parameter: > {{hive.llap.lockmetrics.collect = true}} > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21204) Instrumentation for read/write locks in LLAP
[ https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21204: - Attachment: (was: HIVE-21204.1.patch) > Instrumentation for read/write locks in LLAP > > > Key: HIVE-21204 > URL: https://issues.apache.org/jira/browse/HIVE-21204 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Attachments: HIVE-21204.4.patch > > > LLAP has several R/W locks for serialization of updates to query tracker, > file data, > Instrumentation is added to monitor the > * total amount of R/W locks within a particular category > * average + max wait/suspension time to get the R/W lock > A category includes all lock instances for particular areas (i.e. category is > FileData and all R/W locks that are used in FileData instances are accounted > within the one category). > The monitoring/accounting is done via Hadoop Metrics 2, making them > accessible via JMX. In addition, a new "locking" GET endpoint is added to the > LLAP daemon's REST interface. It produces output like the following example: > { > {{ "statsCollection": "enabled",}} > {{ "lockStats": [}} > {{ {}}{{ "type": "R/W Lock Stats",}} > {{ "label": "FileData",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ },}} > {{ { "}}{{type": "R/W Lock Stats",}} > {{ "label": "QueryTracker",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ } }}{{]}} > {{}}} > To avoid the overhead of lock instrumentation, lock metrics collection is > disabled by default and can be enabled via the following configuration > parameter: > {{hive.llap.lockmetrics.collect = true}} > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21204) Instrumentation for read/write locks in LLAP
[ https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21204: - Attachment: (was: HIVE-21204.patch) > Instrumentation for read/write locks in LLAP > > > Key: HIVE-21204 > URL: https://issues.apache.org/jira/browse/HIVE-21204 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Attachments: HIVE-21204.4.patch > > > LLAP has several R/W locks for serialization of updates to query tracker, > file data, > Instrumentation is added to monitor the > * total amount of R/W locks within a particular category > * average + max wait/suspension time to get the R/W lock > A category includes all lock instances for particular areas (i.e. category is > FileData and all R/W locks that are used in FileData instances are accounted > within the one category). > The monitoring/accounting is done via Hadoop Metrics 2, making them > accessible via JMX. In addition, a new "locking" GET endpoint is added to the > LLAP daemon's REST interface. It produces output like the following example: > { > {{ "statsCollection": "enabled",}} > {{ "lockStats": [}} > {{ {}}{{ "type": "R/W Lock Stats",}} > {{ "label": "FileData",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ },}} > {{ { "}}{{type": "R/W Lock Stats",}} > {{ "label": "QueryTracker",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ } }}{{]}} > {{}}} > To avoid the overhead of lock instrumentation, lock metrics collection is > disabled by default and can be enabled via the following configuration > parameter: > {{hive.llap.lockmetrics.collect = true}} > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21204) Instrumentation for read/write locks in LLAP
[ https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21204: - Attachment: (was: HIVE-21204.3.patch) > Instrumentation for read/write locks in LLAP > > > Key: HIVE-21204 > URL: https://issues.apache.org/jira/browse/HIVE-21204 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Attachments: HIVE-21204.4.patch > > > LLAP has several R/W locks for serialization of updates to query tracker, > file data, > Instrumentation is added to monitor the > * total amount of R/W locks within a particular category > * average + max wait/suspension time to get the R/W lock > A category includes all lock instances for particular areas (i.e. category is > FileData and all R/W locks that are used in FileData instances are accounted > within the one category). > The monitoring/accounting is done via Hadoop Metrics 2, making them > accessible via JMX. In addition, a new "locking" GET endpoint is added to the > LLAP daemon's REST interface. It produces output like the following example: > { > {{ "statsCollection": "enabled",}} > {{ "lockStats": [}} > {{ {}}{{ "type": "R/W Lock Stats",}} > {{ "label": "FileData",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ },}} > {{ { "}}{{type": "R/W Lock Stats",}} > {{ "label": "QueryTracker",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ } }}{{]}} > {{}}} > To avoid the overhead of lock instrumentation, lock metrics collection is > disabled by default and can be enabled via the following configuration > parameter: > {{hive.llap.lockmetrics.collect = true}} > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21204) Instrumentation for read/write locks in LLAP
[ https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21204: - Attachment: HIVE-21204.4.patch > Instrumentation for read/write locks in LLAP > > > Key: HIVE-21204 > URL: https://issues.apache.org/jira/browse/HIVE-21204 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Attachments: HIVE-21204.4.patch > > > LLAP has several R/W locks for serialization of updates to query tracker, > file data, > Instrumentation is added to monitor the > * total amount of R/W locks within a particular category > * average + max wait/suspension time to get the R/W lock > A category includes all lock instances for particular areas (i.e. category is > FileData and all R/W locks that are used in FileData instances are accounted > within the one category). > The monitoring/accounting is done via Hadoop Metrics 2, making them > accessible via JMX. In addition, a new "locking" GET endpoint is added to the > LLAP daemon's REST interface. It produces output like the following example: > { > {{ "statsCollection": "enabled",}} > {{ "lockStats": [}} > {{ {}}{{ "type": "R/W Lock Stats",}} > {{ "label": "FileData",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ },}} > {{ { "}}{{type": "R/W Lock Stats",}} > {{ "label": "QueryTracker",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ } }}{{]}} > {{}}} > To avoid the overhead of lock instrumentation, lock metrics collection is > disabled by default and can be enabled via the following configuration > parameter: > {{hive.llap.lockmetrics.collect = true}} > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21183) Interrupt wait time for FileCacheCleanupThread
[ https://issues.apache.org/jira/browse/HIVE-21183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21183: - Attachment: HIVE-21183.2.patch > Interrupt wait time for FileCacheCleanupThread > -- > > Key: HIVE-21183 > URL: https://issues.apache.org/jira/browse/HIVE-21183 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Minor > Attachments: HIVE-21183.1.patch, HIVE-21183.2.patch, > HIVE-21183.2.patch, HIVE-21183.patch > > > The FileCacheCleanupThread is waiting unnecessarily long for eviction counts > to increment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21204) Instrumentation for read/write locks in LLAP
[ https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21204: - Attachment: HIVE-21204.3.patch > Instrumentation for read/write locks in LLAP > > > Key: HIVE-21204 > URL: https://issues.apache.org/jira/browse/HIVE-21204 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Attachments: HIVE-21204.1.patch, HIVE-21204.2.patch, > HIVE-21204.3.patch, HIVE-21204.patch > > > LLAP has several R/W locks for serialization of updates to query tracker, > file data, > Instrumentation is added to monitor the > * total amount of R/W locks within a particular category > * average + max wait/suspension time to get the R/W lock > A category includes all lock instances for particular areas (i.e. category is > FileData and all R/W locks that are used in FileData instances are accounted > within the one category). > The monitoring/accounting is done via Hadoop Metrics 2, making them > accessible via JMX. In addition, a new "locking" GET endpoint is added to the > LLAP daemon's REST interface. It produces output like the following example: > { > {{ "statsCollection": "enabled",}} > {{ "lockStats": [}} > {{ {}}{{ "type": "R/W Lock Stats",}} > {{ "label": "FileData",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ },}} > {{ { "}}{{type": "R/W Lock Stats",}} > {{ "label": "QueryTracker",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ } }}{{]}} > {{}}} > To avoid the overhead of lock instrumentation, lock metrics collection is > disabled by default and can be enabled via the following configuration > parameter: > {{hive.llap.lockmetrics.collect = true}} > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21422: - Attachment: HIVE-21422.2.patch > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21422.1.patch, HIVE-21422.2.patch, > HIVE-21422.2.patch, HIVE-21422.patch > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16796256#comment-16796256 ] Oliver Draese commented on HIVE-21422: -- Added a patch version (2) which removes all checkstyle warnings. > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21422.1.patch, HIVE-21422.2.patch, HIVE-21422.patch > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21422: - Attachment: HIVE-21422.2.patch > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21422.1.patch, HIVE-21422.2.patch, HIVE-21422.patch > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795399#comment-16795399 ] Oliver Draese commented on HIVE-21422: -- Added pull request: https://github.com/apache/hive/pull/574 > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21422.1.patch, HIVE-21422.patch > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21422: - Comment: was deleted (was: Added pull request for this patch: https://github.com/apache/hive/pull/563/files) > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21422.1.patch, HIVE-21422.patch > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795387#comment-16795387 ] Oliver Draese commented on HIVE-21422: -- Added new counter output also to LLAP server's iomem web endpoint > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21422.1.patch, HIVE-21422.patch > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21422: - Attachment: HIVE-21422.1.patch > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21422.1.patch, HIVE-21422.patch > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16789572#comment-16789572 ] Oliver Draese commented on HIVE-21422: -- Added pull request for this patch: https://github.com/apache/hive/pull/563/files > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21422.patch > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21422: - Labels: llap (was: ) Attachment: HIVE-21422.patch Tags: llap Status: Patch Available (was: Open) > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21422.patch > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese reassigned HIVE-21422: > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Fix For: 4.0.0 > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21221) Make HS2 and LLAP consistent - Bring up LLAP WebUI in test mode if WebUI port is configured
[ https://issues.apache.org/jira/browse/HIVE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21221: - Attachment: HIVE-21221.1.patch > Make HS2 and LLAP consistent - Bring up LLAP WebUI in test mode if WebUI port > is configured > --- > > Key: HIVE-21221 > URL: https://issues.apache.org/jira/browse/HIVE-21221 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Trivial > Labels: llap > Attachments: HIVE-21221.1.patch, HIVE-21221.patch > > > When HiveServer2 comes up, it skips the start of the WebUI if > 1) hive.in.test is set to true > AND > 2) the WebUI port is not specified or default (hive.server2.webui.port) > > Right now, on LLAP daemon start, it is only checked if hive is in test > (condition 1) above. > > The LLAP Daemon start up code (to skip WebUI creation) should be consistent > with HS2, therefore if a port is specified (other than the default), the > WebUI should also be started in test mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21204) Instrumentation for read/write locks in LLAP
[ https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21204: - Attachment: HIVE-21204.2.patch > Instrumentation for read/write locks in LLAP > > > Key: HIVE-21204 > URL: https://issues.apache.org/jira/browse/HIVE-21204 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Attachments: HIVE-21204.1.patch, HIVE-21204.2.patch, HIVE-21204.patch > > > LLAP has several R/W locks for serialization of updates to query tracker, > file data, > Instrumentation is added to monitor the > * total amount of R/W locks within a particular category > * average + max wait/suspension time to get the R/W lock > A category includes all lock instances for particular areas (i.e. category is > FileData and all R/W locks that are used in FileData instances are accounted > within the one category). > The monitoring/accounting is done via Hadoop Metrics 2, making them > accessible via JMX. In addition, a new "locking" GET endpoint is added to the > LLAP daemon's REST interface. It produces output like the following example: > { > {{ "statsCollection": "enabled",}} > {{ "lockStats": [}} > {{ {}}{{ "type": "R/W Lock Stats",}} > {{ "label": "FileData",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ },}} > {{ { "}}{{type": "R/W Lock Stats",}} > {{ "label": "QueryTracker",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ } }}{{]}} > {{}}} > To avoid the overhead of lock instrumentation, lock metrics collection is > disabled by default and can be enabled via the following configuration > parameter: > {{hive.llap.lockmetrics.collect = true}} > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21221) Make HS2 and LLAP consistent - Bring up LLAP WebUI in test mode if WebUI port is configured
[ https://issues.apache.org/jira/browse/HIVE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21221: - Attachment: HIVE-21221.patch Status: Patch Available (was: Open) > Make HS2 and LLAP consistent - Bring up LLAP WebUI in test mode if WebUI port > is configured > --- > > Key: HIVE-21221 > URL: https://issues.apache.org/jira/browse/HIVE-21221 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Trivial > Labels: llap > Attachments: HIVE-21221.patch > > > When HiveServer2 comes up, it skips the start of the WebUI if > 1) hive.in.test is set to true > AND > 2) the WebUI port is not specified or default (hive.server2.webui.port) > > Right now, on LLAP daemon start, it is only checked if hive is in test > (condition 1) above. > > The LLAP Daemon start up code (to skip WebUI creation) should be consistent > with HS2, therefore if a port is specified (other than the default), the > WebUI should also be started in test mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21221) Make HS2 and LLAP consistent - Bring up LLAP WebUI in test mode if WebUI port is configured
[ https://issues.apache.org/jira/browse/HIVE-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese reassigned HIVE-21221: > Make HS2 and LLAP consistent - Bring up LLAP WebUI in test mode if WebUI port > is configured > --- > > Key: HIVE-21221 > URL: https://issues.apache.org/jira/browse/HIVE-21221 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Trivial > Labels: llap > > When HiveServer2 comes up, it skips the start of the WebUI if > 1) hive.in.test is set to true > AND > 2) the WebUI port is not specified or default (hive.server2.webui.port) > > Right now, on LLAP daemon start, it is only checked if hive is in test > (condition 1) above. > > The LLAP Daemon start up code (to skip WebUI creation) should be consistent > with HS2, therefore if a port is specified (other than the default), the > WebUI should also be started in test mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21183) Interrupt wait time for FileCacheCleanupThread
[ https://issues.apache.org/jira/browse/HIVE-21183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21183: - Attachment: HIVE-21183.2.patch > Interrupt wait time for FileCacheCleanupThread > -- > > Key: HIVE-21183 > URL: https://issues.apache.org/jira/browse/HIVE-21183 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Minor > Attachments: HIVE-21183.1.patch, HIVE-21183.2.patch, HIVE-21183.patch > > > The FileCacheCleanupThread is waiting unnecessarily long for eviction counts > to increment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21204) Instrumentation for read/write locks in LLAP
[ https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21204: - Attachment: HIVE-21204.1.patch > Instrumentation for read/write locks in LLAP > > > Key: HIVE-21204 > URL: https://issues.apache.org/jira/browse/HIVE-21204 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Attachments: HIVE-21204.1.patch, HIVE-21204.patch > > > LLAP has several R/W locks for serialization of updates to query tracker, > file data, > Instrumentation is added to monitor the > * total amount of R/W locks within a particular category > * average + max wait/suspension time to get the R/W lock > A category includes all lock instances for particular areas (i.e. category is > FileData and all R/W locks that are used in FileData instances are accounted > within the one category). > The monitoring/accounting is done via Hadoop Metrics 2, making them > accessible via JMX. In addition, a new "locking" GET endpoint is added to the > LLAP daemon's REST interface. It produces output like the following example: > { > {{ "statsCollection": "enabled",}} > {{ "lockStats": [}} > {{ {}}{{ "type": "R/W Lock Stats",}} > {{ "label": "FileData",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ },}} > {{ { "}}{{type": "R/W Lock Stats",}} > {{ "label": "QueryTracker",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > } > {{ } }}{{]}} > {{}}} > To avoid the overhead of lock instrumentation, lock metrics collection is > disabled by default and can be enabled via the following configuration > parameter: > {{hive.llap.lockmetrics.collect = true}} > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21204) Instrumentation for read/write locks in LLAP
[ https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21204: - Description: LLAP has several R/W locks for serialization of updates to query tracker, file data, Instrumentation is added to monitor the * total amount of R/W locks within a particular category * average + max wait/suspension time to get the R/W lock A category includes all lock instances for particular areas (i.e. category is FileData and all R/W locks that are used in FileData instances are accounted within the one category). The monitoring/accounting is done via Hadoop Metrics 2, making them accessible via JMX. In addition, a new "locking" GET endpoint is added to the LLAP daemon's REST interface. It produces output like the following example: { {{ "statsCollection": "enabled",}} {{ "lockStats": [}} {{ {}}{{ "type": "R/W Lock Stats",}} {{ "label": "FileData",}} {{ "totalLockWaitTimeMillis": 0,}} {{ "readLock": {}} {{ "count": 0,}} {{ "avgWaitTimeNanos": 0,}} {{ "maxWaitTimeNanos": 0}} {{ },}} {{ "writeLock": {}} {{ "count": 0,}} {{ "avgWaitTimeNanos": 0,}} {{ "maxWaitTimeNanos": 0}} } {{ },}} {{ { "}}{{type": "R/W Lock Stats",}} {{ "label": "QueryTracker",}} {{ "totalLockWaitTimeMillis": 0,}} {{ "readLock": {}} {{ "count": 0,}} {{ "avgWaitTimeNanos": 0,}} {{ "maxWaitTimeNanos": 0}} {{ },}} {{ "writeLock": {}} {{ "count": 0,}} {{ "avgWaitTimeNanos": 0,}} {{ "maxWaitTimeNanos": 0}} } {{ } }}{{]}} {{}}} To avoid the overhead of lock instrumentation, lock metrics collection is disabled by default and can be enabled via the following configuration parameter: {{hive.llap.lockmetrics.collect = true}} was: LLAP has several R/W locks for serialization of updates to query tracker, file data, Instrumentation is added to monitor the * total amount of R/W locks within a particular category * average + max wait/suspension time to get the R/W lock A category includes all lock instances for particular areas (i.e. category is FileData and all R/W locks that are used in FileData instances are accounted within the one category). The monitoring/accounting is done via Hadoop Metrics 2, making them accessible via JMX. In addition, a new "locking" GET endpoint is added to the LLAP daemon's REST interface. It produces output like the following example: {{{}} {{ "statsCollection": "enabled",}} {{ "lockStats": [}} {{ {}}{{ "type": "R/W Lock Stats",}} {{ "label": "FileData",}} {{ "totalLockWaitTimeMillis": 0,}} {{ "readLock": {}} {{ "count": 0,}} {{ "avgWaitTimeNanos": 0,}} {{ "maxWaitTimeNanos": 0}} {{ },}} {{ "writeLock": {}} {{ "count": 0,}} {{ "avgWaitTimeNanos": 0,}} {{ "maxWaitTimeNanos": 0}} {{ }}} {{ },}} {{ { "}}{{type": "R/W Lock Stats",}} {{ "label": "QueryTracker",}} {{ "totalLockWaitTimeMillis": 0,}} {{ "readLock": {}} {{ "count": 0,}} {{ "avgWaitTimeNanos": 0,}} {{ "maxWaitTimeNanos": 0}} {{ },}} {{ "writeLock": {}} {{ "count": 0,}} {{ "avgWaitTimeNanos": 0,}} {{ "maxWaitTimeNanos": 0}} {{ }}} {{ } }}{{]}} {{ }}} {{}}} To avoid the overhead of lock instrumentation, lock metrics collection is disabled by default and can be enabled via the following configuration parameter: {{hive.llap.lockmetrics.collect = true}} > Instrumentation for read/write locks in LLAP > > > Key: HIVE-21204 > URL: https://issues.apache.org/jira/browse/HIVE-21204 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Attachments: HIVE-21204.patch > > > LLAP has several R/W locks for serialization of updates to query tracker, > file data, > Instrumentation is added to monitor the > * total amount of R/W locks within a particular category > * average + max wait/suspension time to get the R/W lock > A category includes all lock instances for particular areas (i.e. category is > FileData and all R/W locks that are used in FileData instances are accounted > within the one category). > The monitoring/accounting is done via Hadoop Metrics 2, making them > accessible via JMX. In addition, a new "locking" GET endpoint is added to the > LLAP daemon's REST interface. It produces output like the following example: > { > {{ "statsCollection": "enabled",}} > {{ "lockStats": [}} > {{ {}}{{ "type": "R/W Lock Stats",}} > {{ "label": "FileData",}} > {{ "totalLockWaitT
[jira] [Updated] (HIVE-21204) Instrumentation for read/write locks in LLAP
[ https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21204: - Attachment: HIVE-21204.patch Status: Patch Available (was: Open) > Instrumentation for read/write locks in LLAP > > > Key: HIVE-21204 > URL: https://issues.apache.org/jira/browse/HIVE-21204 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Attachments: HIVE-21204.patch > > > LLAP has several R/W locks for serialization of updates to query tracker, > file data, > Instrumentation is added to monitor the > * total amount of R/W locks within a particular category > * average + max wait/suspension time to get the R/W lock > A category includes all lock instances for particular areas (i.e. category is > FileData and all R/W locks that are used in FileData instances are accounted > within the one category). > The monitoring/accounting is done via Hadoop Metrics 2, making them > accessible via JMX. In addition, a new "locking" GET endpoint is added to the > LLAP daemon's REST interface. It produces output like the following example: > {{{}} > {{ "statsCollection": "enabled",}} > {{ "lockStats": [}} > {{ {}}{{ "type": "R/W Lock Stats",}} > {{ "label": "FileData",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ }}} > {{ },}} > {{ { "}}{{type": "R/W Lock Stats",}} > {{ "label": "QueryTracker",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ }}} > {{ } }}{{]}} > {{ }}} > {{}}} > To avoid the overhead of lock instrumentation, lock metrics collection is > disabled by default and can be enabled via the following configuration > parameter: > {{hive.llap.lockmetrics.collect = true}} > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21204) Instrumentation for read/write locks in LLAP
[ https://issues.apache.org/jira/browse/HIVE-21204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese reassigned HIVE-21204: > Instrumentation for read/write locks in LLAP > > > Key: HIVE-21204 > URL: https://issues.apache.org/jira/browse/HIVE-21204 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > > LLAP has several R/W locks for serialization of updates to query tracker, > file data, > Instrumentation is added to monitor the > * total amount of R/W locks within a particular category > * average + max wait/suspension time to get the R/W lock > A category includes all lock instances for particular areas (i.e. category is > FileData and all R/W locks that are used in FileData instances are accounted > within the one category). > The monitoring/accounting is done via Hadoop Metrics 2, making them > accessible via JMX. In addition, a new "locking" GET endpoint is added to the > LLAP daemon's REST interface. It produces output like the following example: > {{{}} > {{ "statsCollection": "enabled",}} > {{ "lockStats": [}} > {{ {}}{{ "type": "R/W Lock Stats",}} > {{ "label": "FileData",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ }}} > {{ },}} > {{ { "}}{{type": "R/W Lock Stats",}} > {{ "label": "QueryTracker",}} > {{ "totalLockWaitTimeMillis": 0,}} > {{ "readLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ },}} > {{ "writeLock": {}} > {{ "count": 0,}} > {{ "avgWaitTimeNanos": 0,}} > {{ "maxWaitTimeNanos": 0}} > {{ }}} > {{ } }}{{]}} > {{ }}} > {{}}} > To avoid the overhead of lock instrumentation, lock metrics collection is > disabled by default and can be enabled via the following configuration > parameter: > {{hive.llap.lockmetrics.collect = true}} > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21183) Interrupt wait time for FileCacheCleanupThread
[ https://issues.apache.org/jira/browse/HIVE-21183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21183: - Attachment: HIVE-21183.1.patch > Interrupt wait time for FileCacheCleanupThread > -- > > Key: HIVE-21183 > URL: https://issues.apache.org/jira/browse/HIVE-21183 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Minor > Attachments: HIVE-21183.1.patch, HIVE-21183.patch > > > The FileCacheCleanupThread is waiting unnecessarily long for eviction counts > to increment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21183) Interrupt wait time for FileCacheCleanupThread
[ https://issues.apache.org/jira/browse/HIVE-21183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21183: - Status: Patch Available (was: Open) > Interrupt wait time for FileCacheCleanupThread > -- > > Key: HIVE-21183 > URL: https://issues.apache.org/jira/browse/HIVE-21183 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Minor > > The FileCacheCleanupThread is waiting unnecessarily long for eviction counts > to increment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21183) Interrupt wait time for FileCacheCleanupThread
[ https://issues.apache.org/jira/browse/HIVE-21183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21183: - Attachment: HIVE-21183.patch > Interrupt wait time for FileCacheCleanupThread > -- > > Key: HIVE-21183 > URL: https://issues.apache.org/jira/browse/HIVE-21183 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Minor > Attachments: HIVE-21183.patch > > > The FileCacheCleanupThread is waiting unnecessarily long for eviction counts > to increment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)