[jira] [Commented] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732372#comment-16732372 ] Aihua Xu commented on HIVE-21033: - [~szehon] The patch looks good to me. +1. > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.2.patch, HIVE-21033.3.patch, > HIVE-21033.4.patch, HIVE-21033.5.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729749#comment-16729749 ] Hive QA commented on HIVE-21033: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12953155/HIVE-21033.5.patch {color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 15759 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15440/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15440/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15440/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12953155 - PreCommit-HIVE-Build > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.2.patch, HIVE-21033.3.patch, > HIVE-21033.4.patch, HIVE-21033.5.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729741#comment-16729741 ] Hive QA commented on HIVE-21033: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 0s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 31s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 39s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 58s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 31s{color} | {color:blue} common in master has 65 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 42s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 36s{color} | {color:blue} service in master has 48 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 26s{color} | {color:blue} cli in master has 13 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 34s{color} | {color:blue} hcatalog/core in master has 30 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 25s{color} | {color:blue} hcatalog/hcatalog-pig-adapter in master has 2 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 50s{color} | {color:blue} itests/util in master has 48 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 38s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 34s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 12s{color} | {color:red} common: The patch generated 2 new + 1 unchanged - 2 fixed = 3 total (was 3) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 36s{color} | {color:red} ql: The patch generated 3 new + 58 unchanged - 3 fixed = 61 total (was 61) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 12s{color} | {color:red} service: The patch generated 1 new + 19 unchanged - 2 fixed = 20 total (was 21) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 36s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 47m 17s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-15440/dev-support/hive-personality.sh | | git revision | master / 926c1e8 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle |
[jira] [Commented] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729667#comment-16729667 ] Hive QA commented on HIVE-21033: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12953150/HIVE-21033.4.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15439/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15439/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15439/ Messages: {noformat} This message was trimmed, see log for full details [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/Closeable.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/lang/AutoCloseable.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/io/Flushable.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(javax/xml/bind/annotation/XmlRootElement.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/commons/commons-exec/1.1/commons-exec-1.1.jar(org/apache/commons/exec/ExecuteException.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/security/PrivilegedExceptionAction.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/ExecutionException.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/TimeoutException.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/fs/FileSystem.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/HadoopShimsSecure.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/ShimLoader.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/HadoopShims.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/apache-github-source-source/shims/common/target/hive-shims-common-4.0.0-SNAPSHOT.jar(org/apache/hadoop/hive/shims/HadoopShims$WebHCatJTShim.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/util/ToolRunner.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/CancellationException.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/RejectedExecutionException.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/SynchronousQueue.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/ThreadPoolExecutor.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/TimeUnit.class)]] [loading ZipFileIndexFileObject[/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar(java/util/concurrent/Future.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/conf/Configured.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/io/NullWritable.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-common/3.1.0/hadoop-common-3.1.0.jar(org/apache/hadoop/io/Text.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.1.0/hadoop-mapreduce-client-core-3.1.0.jar(org/apache/hadoop/mapred/JobClient.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.1.0/hadoop-mapreduce-client-core-3.1.0.jar(org/apache/hadoop/mapred/JobConf.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.1.0/hadoop-mapreduce-client-core-3.1.0.jar(org/apache/hadoop/mapreduce/Job.class)]] [loading ZipFileIndexFileObject[/data/hiveptest/working/maven/org/apache/hadoop/hadoop-mapreduce-client-core/3.1.0/hadoop-mapreduce-client-core-3.1.0.jar(org/apache/hadoop/mapreduce/JobID.class)]] [loading
[jira] [Commented] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729655#comment-16729655 ] Szehon Ho commented on HIVE-21033: -- Fix checkstyle > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.2.patch, HIVE-21033.3.patch, > HIVE-21033.4.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729603#comment-16729603 ] Szehon Ho commented on HIVE-21033: -- Patch creates new kind of PrintStream for SessionState, which I call 'SessionStream' to be able to handle not closing of System.out and System.err. As consequence, a lot of test classes need to change a little bit. > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.2.patch, HIVE-21033.3.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729601#comment-16729601 ] Szehon Ho commented on HIVE-21033: -- [~aihuaxu] Do you have time to look at this patch? > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.2.patch, HIVE-21033.3.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729543#comment-16729543 ] Hive QA commented on HIVE-21033: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12953132/HIVE-21033.3.patch {color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 15759 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15438/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15438/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15438/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12953132 - PreCommit-HIVE-Build > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.2.patch, HIVE-21033.3.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729533#comment-16729533 ] Hive QA commented on HIVE-21033: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 48s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 34s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 37s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 54s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 29s{color} | {color:blue} common in master has 65 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 39s{color} | {color:blue} ql in master has 2311 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 38s{color} | {color:blue} service in master has 48 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 26s{color} | {color:blue} cli in master has 13 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 34s{color} | {color:blue} hcatalog/core in master has 30 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 23s{color} | {color:blue} hcatalog/hcatalog-pig-adapter in master has 2 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 43s{color} | {color:blue} itests/util in master has 48 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 41s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 37s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 10s{color} | {color:red} common: The patch generated 5 new + 1 unchanged - 2 fixed = 6 total (was 3) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 38s{color} | {color:red} ql: The patch generated 4 new + 58 unchanged - 3 fixed = 62 total (was 61) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 12s{color} | {color:red} service: The patch generated 2 new + 19 unchanged - 2 fixed = 21 total (was 21) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s{color} | {color:red} hcatalog/core: The patch generated 1 new + 21 unchanged - 0 fixed = 22 total (was 21) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 12s{color} | {color:red} itests/util: The patch generated 1 new + 53 unchanged - 0 fixed = 54 total (was 53) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 40s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | |
[jira] [Commented] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729105#comment-16729105 ] Hive QA commented on HIVE-21033: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12953077/HIVE-21033.2.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15434/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15434/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15434/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2018-12-26 17:03:03.046 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-15434/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2018-12-26 17:03:03.049 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 926c1e8 HIVE-20936 Allow the Worker thread in the metastore to run outside of it (Jaume Marhuenda via Eugene Koifman) + git clean -f -d Removing standalone-metastore/metastore-server/src/gen/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 926c1e8 HIVE-20936 Allow the Worker thread in the metastore to run outside of it (Jaume Marhuenda via Eugene Koifman) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2018-12-26 17:03:03.759 + rm -rf ../yetus_PreCommit-HIVE-Build-15434 + mkdir ../yetus_PreCommit-HIVE-Build-15434 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-15434 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15434/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: a/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java: does not exist in index error: a/cli/src/test/org/apache/hadoop/hive/cli/TestCliDriverMethods.java: does not exist in index error: a/common/src/java/org/apache/hadoop/hive/common/io/CachingPrintStream.java: does not exist in index error: a/common/src/java/org/apache/hadoop/hive/common/io/FetchConverter.java: does not exist in index error: a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java: does not exist in index error: a/itests/hive-unit/src/test/java/org/apache/hive/jdbc/authorization/TestCLIAuthzSessionContext.java: does not exist in index error: a/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java: does not exist in index error: a/ql/src/test/org/apache/hadoop/hive/ql/processors/TestSetProcessor.java: does not exist in index error: a/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java: does not exist in index error: a/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java: does not exist in index Going to apply patch with: git apply -p1 + [[ maven == \m\a\v\e\n ]] + rm -rf /data/hiveptest/working/maven/org/apache/hive + mvn -B clean install -DskipTests -T 4 -q -Dmaven.repo.local=/data/hiveptest/working/maven protoc-jar: executing: [/tmp/protoc5620331673436136816.exe, --version] libprotoc 2.5.0 protoc-jar: executing: [/tmp/protoc5620331673436136816.exe, -I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore, --java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
[jira] [Commented] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729097#comment-16729097 ] Szehon Ho commented on HIVE-21033: -- Rebasing > Forgetting to close operation cuts off any more HiveServer2 output > -- > > Key: HIVE-21033 > URL: https://issues.apache.org/jira/browse/HIVE-21033 > Project: Hive > Issue Type: Bug >Reporter: Szehon Ho >Assignee: Szehon Ho >Priority: Major > Attachments: HIVE-21033.2.patch, HIVE-21033.patch > > > We had a custom client that did not handle closing the operations, until the > end of the session. it is a mistake in the client, but it reveals kind of a > vulnerability in HiveServer2 > This happens if you have a session with (1) HiveCommandOperation and (2) > SQLOperation and don't close them right after. For example a session that > does the operations (set a=b; select * from foobar; ). > When SQLOperation runs , it set SessionState.out and err to be System.out and > System.err . Ref: > [SQLOperation#setupSessionIO|https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L139] > Then the client closes the session, or disconnects which triggers > closeSession() on the Thrift side. In this case, the closeSession closes all > the operations, starting with HiveCommandOperation. This closes all the > streams, which is System.out and System.err as set by SQLOperation earlier. > Ref: > [HiveCommandOperation#tearDownSessionIO|https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java#L101] > > After this, no more HiveServer2 output appears as System.out and System.err > are closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21033) Forgetting to close operation cuts off any more HiveServer2 output
[ https://issues.apache.org/jira/browse/HIVE-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16729067#comment-16729067 ] Hive QA commented on HIVE-21033: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12953076/HIVE-21033.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15433/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15433/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15433/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2018-12-26 15:24:44.328 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-15433/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2018-12-26 15:24:44.332 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 926c1e8 HIVE-20936 Allow the Worker thread in the metastore to run outside of it (Jaume Marhuenda via Eugene Koifman) + git clean -f -d Removing standalone-metastore/metastore-server/src/gen/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 926c1e8 HIVE-20936 Allow the Worker thread in the metastore to run outside of it (Jaume Marhuenda via Eugene Koifman) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2018-12-26 15:24:45.497 + rm -rf ../yetus_PreCommit-HIVE-Build-15433 + mkdir ../yetus_PreCommit-HIVE-Build-15433 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-15433 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15433/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: a/common/src/java/org/apache/hadoop/hive/common/io/CachingPrintStream.java: does not exist in index error: a/common/src/java/org/apache/hadoop/hive/common/io/FetchConverter.java: does not exist in index error: a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java: does not exist in index error: a/itests/hive-unit/src/test/java/org/apache/hive/jdbc/authorization/TestCLIAuthzSessionContext.java: does not exist in index error: a/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java: does not exist in index error: a/ql/src/test/org/apache/hadoop/hive/ql/processors/TestSetProcessor.java: does not exist in index error: a/service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java: does not exist in index error: a/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java: does not exist in index Going to apply patch with: git apply -p1 + [[ maven == \m\a\v\e\n ]] + rm -rf /data/hiveptest/working/maven/org/apache/hive + mvn -B clean install -DskipTests -T 4 -q -Dmaven.repo.local=/data/hiveptest/working/maven protoc-jar: executing: [/tmp/protoc117750883458147622.exe, --version] libprotoc 2.5.0 protoc-jar: executing: [/tmp/protoc117750883458147622.exe, -I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore, --java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources, /data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto] ANTLR Parser Generator Version 3.5.2 protoc-jar: executing: [/tmp/protoc6030905574379381376.exe, --version] libprotoc 2.5.0