[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14969193#comment-14969193 ] Aihua Xu commented on HIVE-11710: - Xuefu, can you help submit the patch? > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.2.patch, HIVE-11710.3.patch, > HIVE-11710.4.patch, HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963478#comment-14963478 ] Aihua Xu commented on HIVE-11710: - [~xuefuz] can you help review the new patch? > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.2.patch, HIVE-11710.3.patch, > HIVE-11710.4.patch, HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963592#comment-14963592 ] Xuefu Zhang commented on HIVE-11710: +1 > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.2.patch, HIVE-11710.3.patch, > HIVE-11710.4.patch, HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14961636#comment-14961636 ] Hive QA commented on HIVE-11710: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12766875/HIVE-11710.4.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 9698 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_explode org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udtf_explode org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation org.apache.hive.jdbc.TestSSL.testSSLVersion {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5682/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5682/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5682/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12766875 - PreCommit-HIVE-TRUNK-Build > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.2.patch, HIVE-11710.3.patch, > HIVE-11710.4.patch, HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959103#comment-14959103 ] Aihua Xu commented on HIVE-11710: - Seems HIVE-11579 fixed one related issue, but the current incorrect behavior still exists. 1. There could be temp file handler leaking from HIveCommandOption class if somehow there are exception. 2. We still need to reset SessionState.out/info/err to System.out/err in SQLOperation, otherwise for embedded mode, the beeline client output will be redirected to the files, not to the console. [~Ferd] You worked on HIVE-11579 recently and probably know that code well. There is a possibility that the temp file handlers are not closed if there is an exception during the following code, correct? And also we need to flush the output rather than closing the stream if the stream points to System.out/.err. right? {noformat} sessionState.out = new PrintStream(new FileOutputStream(sessionState.getTmpOutputFile()), true, CharEncoding.UTF_8); sessionState.err = new PrintStream(new FileOutputStream(sessionState.getTmpErrOutputFile()), true,CharEncoding.UTF_8); {noformat} [~xuefuz] Sorry. Didn't get time to work on that. MyPrintStream class will be cleaner, but it's not easy to differentiate if it's file based or not from the stream itself since System.out or System.err can also point to file based stream as well. So it's tight to the class HiveCommandOperation class themselves and we may need to pass a flag "flushOnClose" to the MyPrintStream class. Let me look into that. > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.2.patch, HIVE-11710.3.patch, HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959601#comment-14959601 ] Aihua Xu commented on HIVE-11710: - Attached the new patch which will close the streams if there is an exception in HIveCommandOperation. Reset the streams for SQLOperation to standard out/err when initializing. > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.2.patch, HIVE-11710.3.patch, > HIVE-11710.4.patch, HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959359#comment-14959359 ] Aihua Xu commented on HIVE-11710: - OK. Seems we don't need flush the string manually since autoFlush is set to true in PrintStream {{PrintStream(OutputStream out, boolean autoFlush, String encoding) }}. > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.2.patch, HIVE-11710.3.patch, HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959364#comment-14959364 ] Aihua Xu commented on HIVE-11710: - Thanks [~Ferd] I will try to add one unit test to cover that. > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.2.patch, HIVE-11710.3.patch, HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959358#comment-14959358 ] Aihua Xu commented on HIVE-11710: - OK. Seems we don't need flush the string manually since autoFlush is set to true in PrintStream {{PrintStream(OutputStream out, boolean autoFlush, String encoding) }}. > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.2.patch, HIVE-11710.3.patch, HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14907772#comment-14907772 ] Hive QA commented on HIVE-11710: {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12761965/HIVE-11710.3.patch {color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 9568 tests executed *Failed tests:* {noformat} TestMiniTezCliDriver-vector_grouping_sets.q-scriptfile1.q-union2.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_groupby_reduce org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_groupby_reduce org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5408/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5408/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5408/ Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12761965 - PreCommit-HIVE-TRUNK-Build > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.2.patch, HIVE-11710.3.patch, HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904502#comment-14904502 ] Aihua Xu commented on HIVE-11710: - Yeah. The second patch will leak "PrintStream", so I think the first patch is reasonable actually in which will close the stream to the temp file before deleting the temp file. Let me check if the close() function will be called when an exception is thrown. > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.2.patch, HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904628#comment-14904628 ] Aihua Xu commented on HIVE-11710: - [~xuefuz] I double checked. The first patch is good to handle closing the file handler when the current session is getting closing. The second patch will have the file handler leaking. Whether an exception occurs or not, hive will close the session, in which it will call {{HiveCommandOperation:close()}}. The first patch will see if the stream is a file stream or standard stream. If it's a file stream, then we should close it at last. Xuefu, how do you think? Should we proceed with the first patch? > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.2.patch, HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904882#comment-14904882 ] Xuefu Zhang commented on HIVE-11710: [~aihuaxu], looking at the two patches, I seem to see that either patch has a possibility of leaking resources, even if we consider in a single thread. The possible leaks are: 1. If new new FileOutputStream() is successful but new PrintStream() is not, then the FileOutputStream instance is never closed. 2. In SQLOperation class, we never close these streams. I think the only change we need to make in HiveCommandOperation is to fix #1. The original tearDownSessionIO() seems fine with the changes in SQLOperation. > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.2.patch, HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905037#comment-14905037 ] Aihua Xu commented on HIVE-11710: - [~xuefuz] Regarding #2, in SQLOperation class, the streams are using System.out and System.err. We shouldn't close them, right? If we did that, then later we wouldn't be able to write to System.out or System.err (actually that is the exact same issue as this issue). For #1, that is a possible leak. I will fix that. Let me know if I understand correctly on #2. > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.2.patch, HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905070#comment-14905070 ] Aihua Xu commented on HIVE-11710: - For SQLOperation, the stream are System.out and System.err. We can't/shouldn't close them. That is how issue HIVE-11710 arises that we closed System.out in HiveCommandOperation, then later we can't write to System.out anymore for any sql. > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.2.patch, HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905041#comment-14905041 ] Xuefu Zhang commented on HIVE-11710: For #2, I meant that we need something similar to HiveCommanOperation.tearDownSessionIO() in which the streams are closed. Each SQLOperation will initialize the streams and close them when done. Isn't this expected? > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.2.patch, HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905112#comment-14905112 ] Aihua Xu commented on HIVE-11710: - Indeed that we have misunderstanding. :) Now I know what you mean. Actually when you close e.g., System.err or System.out in one place, later you will never reopen it again. So {{sessionState.out = new PrintStream(System.out, true, CharEncoding.UTF_8);}} will create the stream object but the internal "out" will be still in "closed" state. That's why I have to detect if it's a file based stream or not. If it is, then we close the stream; if not, we should leave it open. > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.2.patch, HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905087#comment-14905087 ] Xuefu Zhang commented on HIVE-11710: {quote} For SQLOperation, the stream are System.out and System.err. We can't/shouldn't close them. {quote} The might be some misunderstanding. We are opening the streams in the constructor (changes you made), and closing them when done (changes to be made). The same will happen to the subsequent queries. I don't understand why this still has the original problem. > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.2.patch, HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905073#comment-14905073 ] Aihua Xu commented on HIVE-11710: - We will close File based stream, not standard System.out and System.err. > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.2.patch, HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904121#comment-14904121 ] Hive QA commented on HIVE-11710: {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12761666/HIVE-11710.2.patch {color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 9575 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_groupby_reduce org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_groupby_reduce org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5381/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5381/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5381/ Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12761666 - PreCommit-HIVE-TRUNK-Build > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.2.patch, HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905382#comment-14905382 ] Xuefu Zhang commented on HIVE-11710: Does the following pseudo code helps a little bit? {code} public class MyPrintStream extends PrintStream { private boolean isFileIO = false; public MyPrintStream(...) { } public void close() { super.flush(); if( file out stream) { super.close(); } else { super.flush(); } } } {code} > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.2.patch, HIVE-11710.3.patch, HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905158#comment-14905158 ] Aihua Xu commented on HIVE-11710: - PrintStream.close() will do "flushing the stream and then closing the underlying output stream" from the doc. Seems I need to add "flushing the stream" while keeping the underlying output stream open". > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.2.patch, HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905128#comment-14905128 ] Xuefu Zhang commented on HIVE-11710: I see. If we don't close the PrintStream instances, won't there be leaks? Looks like we need to close these wrapper streams while keeping sys.out, sys.err, etc. open. > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.2.patch, HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903471#comment-14903471 ] Xuefu Zhang commented on HIVE-11710: [~aihuaxu], your new patch looks good. One thing I'm not 100% sure is what happens when deleting file w/o closing it first. Secondly, we could leak either "new PrintStream" or "new FileOutputStream" or both if there is any exception or if we don't close ss.out. This seems minor but reliable code is ideal. Thus, I suggest that we make sure that these objects are closed properly whether there is an exception or not. As you can see that session state is complete thread-unsafe (ref. HIVE-11402). However, this has nothing to do the problem you're addressing. > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.2.patch, HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14902821#comment-14902821 ] Aihua Xu commented on HIVE-11710: - [~xuefuz] Attached the new patch which doesn't close the ss.out or ss.err. Can you help take a look the new patch? > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.2.patch, HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14875868#comment-14875868 ] Xuefu Zhang commented on HIVE-11710: Do you have any thoughts on this? {quote} It looks like that we don't need to close either ss.out or ss.err. Otherwise, if one operation gets closed for any reason (such as timeout), then other operations in the same user session will get ss.out and ss.err that's closed. {quote} > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14745483#comment-14745483 ] Aihua Xu commented on HIVE-11710: - OK. Seems it's fine since from the close() call shown below, we are deleting that temp file any way at that moment. So at that moment when close() is called, we can't write to that tmp file anyway since the file is getting deleted ({{cleanTmpFile()}}). {noformat} public void close() throws HiveSQLException { setState(OperationState.CLOSED); tearDownSessionIO(); cleanTmpFile(); cleanupOperationLog(); } {noformat} > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14745465#comment-14745465 ] Aihua Xu commented on HIVE-11710: - Yeah. You are right. It really depends on how ss.out is used. Originally somehow I thought it's only used by this file, but actually it's not. I will remove that as well. > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743611#comment-14743611 ] Aihua Xu commented on HIVE-11710: - The failures are not caused by the patch. > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744393#comment-14744393 ] Xuefu Zhang commented on HIVE-11710: In HiveCommandOperation.tearDownSessionIO(), should we just close the result of this: new FileOutputStream(sessionState.getTmpOutputFile())? It looks like that we don't need to close either ss.out or ss.err. Otherwise, if one operation gets closed for any reason (such as timeout), then other operations in the same user session will get ss.out and ss.err that's closed. It's my understanding that session state is shared among operations in a hive session. > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741703#comment-14741703 ] Hive QA commented on HIVE-11710: {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12755398/HIVE-11710.patch {color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 9422 tests executed *Failed tests:* {noformat} org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler org.apache.hive.hcatalog.streaming.TestStreaming.testTransactionBatchEmptyCommit {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5242/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5242/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5242/ Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12755398 - PreCommit-HIVE-TRUNK-Build > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738780#comment-14738780 ] Aihua Xu commented on HIVE-11710: - I didn't create code review for this since it's a simple change. Submit the first patch to see if it would break anything. > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739348#comment-14739348 ] Xuefu Zhang commented on HIVE-11710: Could we also check what Hive CLI is doing w.r.t this? > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739409#comment-14739409 ] Aihua Xu commented on HIVE-11710: - CLI sets as follows in CLIDriver.java. {noformat} ss.out = new PrintStream(System.out, true, "UTF-8"); ss.info = new PrintStream(System.err, true, "UTF-8"); ss.err = new CachingPrintStream(System.err, true, "UTF-8"); {noformat} > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11710) Beeline embedded mode doesn't output query progress after setting any session property
[ https://issues.apache.org/jira/browse/HIVE-11710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739441#comment-14739441 ] Aihua Xu commented on HIVE-11710: - Actually after {{IOUtils.cleanup(LOG, parentSession.getSessionState().err);}}, System.err will always be in a closed state although the object is there. Seems We shouldn't close System.err stream. > Beeline embedded mode doesn't output query progress after setting any session > property > -- > > Key: HIVE-11710 > URL: https://issues.apache.org/jira/browse/HIVE-11710 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-11710.patch > > > Connect to beeline embedded mode {{beeline -u jdbc:hive2://}}. Then set > anything in the session like {{set aa=true;}}. > After that, any query like {{select count(*) from src;}} will only output > result but no query progress. -- This message was sent by Atlassian JIRA (v6.3.4#6332)