[jira] [Commented] (HIVE-14342) Beeline output is garbled when executed from a remote shell
[ https://issues.apache.org/jira/browse/HIVE-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16385980#comment-16385980 ] George Smith commented on HIVE-14342: - [~ngangam] Thank you for your response! Are you able to reproduce my problem on your cluster configuration as well? > Beeline output is garbled when executed from a remote shell > --- > > Key: HIVE-14342 > URL: https://issues.apache.org/jira/browse/HIVE-14342 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Fix For: 2.1.1, 2.2.0 > > Attachments: HIVE-14342.2.patch, HIVE-14342.patch, HIVE-14342.patch > > > {code} > use default; > create table clitest (key int, name String, value String); > insert into table clitest values > (1,"TRUE","1"),(2,"TRUE","1"),(3,"TRUE","1"),(4,"TRUE","1"),(5,"FALSE","0"),(6,"FALSE","0"),(7,"FALSE","0"); > {code} > then run a select query > {code} > # cat /tmp/select.sql > set hive.execution.engine=mr; > select key,name,value > from clitest > where value="1" limit 1; > {code} > Then run beeline via a remote shell, for example > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/select.sql" > root@'s password: > 16/07/12 14:59:22 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > nullkey,name,value > 1,TRUE,1 > null > $ > {code} > In older releases that the output is as follows > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/run.sql" > Are you sure you want to continue connecting (yes/no)? yes > root@'s password: > 16/07/12 14:57:55 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > key,name,value > 1,TRUE,1 > $ > {code} > The output contains nulls instead of blank lines. This is due to the use of > -Djline.terminal=jline.UnsupportedTerminal introduced in HIVE-6758 to be able > to run beeline as a background process. But this is the unfortunate side > effect of that fix. > Running beeline in background also produces garbled output. > {code} > # beeline -u "jdbc:hive2://localhost:1" -n hive -p hive --silent=true > --outputformat=csv2 --showHeader=false -f /tmp/run.sql 2>&1 > > /tmp/beeline.txt & > # cat /tmp/beeline.txt > null1,TRUE,1 > # > {code} > So I think the use of jline.UnsupportedTerminal should be documented but not > used automatically by beeline under the covers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-14342) Beeline output is garbled when executed from a remote shell
[ https://issues.apache.org/jira/browse/HIVE-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383719#comment-16383719 ] Naveen Gangam commented on HIVE-14342: -- [~cabot] Your testcase is slightly different than the issue in this jira. You are running beeline as a background process that has to use the {{-Djline.terminal=jline.UnsupportedTerminal}} property. Otherwise there is contention for STDOUT/STDERR streams between the JLine Terminal and Unix background process. As a result, the backgrounded beeline process is SUSPENDED by the OS (similar to CNTRL Z). So without using this property, beeline wouldnt even run as a background process. But unfortunately, it appears to have this side effect. I can look at the jline code again to see if there have been any fixes. Does that make sense? > Beeline output is garbled when executed from a remote shell > --- > > Key: HIVE-14342 > URL: https://issues.apache.org/jira/browse/HIVE-14342 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Fix For: 2.1.1, 2.2.0 > > Attachments: HIVE-14342.2.patch, HIVE-14342.patch, HIVE-14342.patch > > > {code} > use default; > create table clitest (key int, name String, value String); > insert into table clitest values > (1,"TRUE","1"),(2,"TRUE","1"),(3,"TRUE","1"),(4,"TRUE","1"),(5,"FALSE","0"),(6,"FALSE","0"),(7,"FALSE","0"); > {code} > then run a select query > {code} > # cat /tmp/select.sql > set hive.execution.engine=mr; > select key,name,value > from clitest > where value="1" limit 1; > {code} > Then run beeline via a remote shell, for example > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/select.sql" > root@'s password: > 16/07/12 14:59:22 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > nullkey,name,value > 1,TRUE,1 > null > $ > {code} > In older releases that the output is as follows > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/run.sql" > Are you sure you want to continue connecting (yes/no)? yes > root@'s password: > 16/07/12 14:57:55 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > key,name,value > 1,TRUE,1 > $ > {code} > The output contains nulls instead of blank lines. This is due to the use of > -Djline.terminal=jline.UnsupportedTerminal introduced in HIVE-6758 to be able > to run beeline as a background process. But this is the unfortunate side > effect of that fix. > Running beeline in background also produces garbled output. > {code} > # beeline -u "jdbc:hive2://localhost:1" -n hive -p hive --silent=true > --outputformat=csv2 --showHeader=false -f /tmp/run.sql 2>&1 > > /tmp/beeline.txt & > # cat /tmp/beeline.txt > null1,TRUE,1 > # > {code} > So I think the use of jline.UnsupportedTerminal should be documented but not > used automatically by beeline under the covers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-14342) Beeline output is garbled when executed from a remote shell
[ https://issues.apache.org/jira/browse/HIVE-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16378643#comment-16378643 ] George Smith commented on HIVE-14342: - Anyone can confirm my issue? > Beeline output is garbled when executed from a remote shell > --- > > Key: HIVE-14342 > URL: https://issues.apache.org/jira/browse/HIVE-14342 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Fix For: 2.1.1, 2.2.0 > > Attachments: HIVE-14342.2.patch, HIVE-14342.patch, HIVE-14342.patch > > > {code} > use default; > create table clitest (key int, name String, value String); > insert into table clitest values > (1,"TRUE","1"),(2,"TRUE","1"),(3,"TRUE","1"),(4,"TRUE","1"),(5,"FALSE","0"),(6,"FALSE","0"),(7,"FALSE","0"); > {code} > then run a select query > {code} > # cat /tmp/select.sql > set hive.execution.engine=mr; > select key,name,value > from clitest > where value="1" limit 1; > {code} > Then run beeline via a remote shell, for example > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/select.sql" > root@'s password: > 16/07/12 14:59:22 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > nullkey,name,value > 1,TRUE,1 > null > $ > {code} > In older releases that the output is as follows > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/run.sql" > Are you sure you want to continue connecting (yes/no)? yes > root@'s password: > 16/07/12 14:57:55 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > key,name,value > 1,TRUE,1 > $ > {code} > The output contains nulls instead of blank lines. This is due to the use of > -Djline.terminal=jline.UnsupportedTerminal introduced in HIVE-6758 to be able > to run beeline as a background process. But this is the unfortunate side > effect of that fix. > Running beeline in background also produces garbled output. > {code} > # beeline -u "jdbc:hive2://localhost:1" -n hive -p hive --silent=true > --outputformat=csv2 --showHeader=false -f /tmp/run.sql 2>&1 > > /tmp/beeline.txt & > # cat /tmp/beeline.txt > null1,TRUE,1 > # > {code} > So I think the use of jline.UnsupportedTerminal should be documented but not > used automatically by beeline under the covers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-14342) Beeline output is garbled when executed from a remote shell
[ https://issues.apache.org/jira/browse/HIVE-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16374205#comment-16374205 ] George Smith commented on HIVE-14342: - Are you sure this has been fixed? We are using Cloudera distribution with HIVE-14342 bundled and I can confirm that beeline file contains check for *Test case:* _*test.hql content:*_ {code} SELECT 1; {code} {code} $ beeline -u jdbc:hive2://xxx:1 --silent=true --showWarnings=false --showNestedErrs=false --showHeader=false --outputformat=tsv2 -f ~/test.hql & wait [1] 3076126 null1 {code} I also tried to add this line before calling beeline: {code} export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Djline.terminal=jline.UnsupportedTerminal" {code} Nothing has changed. > Beeline output is garbled when executed from a remote shell > --- > > Key: HIVE-14342 > URL: https://issues.apache.org/jira/browse/HIVE-14342 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Fix For: 2.1.1, 2.2.0 > > Attachments: HIVE-14342.2.patch, HIVE-14342.patch, HIVE-14342.patch > > > {code} > use default; > create table clitest (key int, name String, value String); > insert into table clitest values > (1,"TRUE","1"),(2,"TRUE","1"),(3,"TRUE","1"),(4,"TRUE","1"),(5,"FALSE","0"),(6,"FALSE","0"),(7,"FALSE","0"); > {code} > then run a select query > {code} > # cat /tmp/select.sql > set hive.execution.engine=mr; > select key,name,value > from clitest > where value="1" limit 1; > {code} > Then run beeline via a remote shell, for example > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/select.sql" > root@'s password: > 16/07/12 14:59:22 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > nullkey,name,value > 1,TRUE,1 > null > $ > {code} > In older releases that the output is as follows > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/run.sql" > Are you sure you want to continue connecting (yes/no)? yes > root@'s password: > 16/07/12 14:57:55 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > key,name,value > 1,TRUE,1 > $ > {code} > The output contains nulls instead of blank lines. This is due to the use of > -Djline.terminal=jline.UnsupportedTerminal introduced in HIVE-6758 to be able > to run beeline as a background process. But this is the unfortunate side > effect of that fix. > Running beeline in background also produces garbled output. > {code} > # beeline -u "jdbc:hive2://localhost:1" -n hive -p hive --silent=true > --outputformat=csv2 --showHeader=false -f /tmp/run.sql 2>&1 > > /tmp/beeline.txt & > # cat /tmp/beeline.txt > null1,TRUE,1 > # > {code} > So I think the use of jline.UnsupportedTerminal should be documented but not > used automatically by beeline under the covers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-14342) Beeline output is garbled when executed from a remote shell
[ https://issues.apache.org/jira/browse/HIVE-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417553#comment-15417553 ] Mohit Sabharwal commented on HIVE-14342: Thanks, [~ngangam], LGTM +1 > Beeline output is garbled when executed from a remote shell > --- > > Key: HIVE-14342 > URL: https://issues.apache.org/jira/browse/HIVE-14342 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-14342.2.patch, HIVE-14342.patch, HIVE-14342.patch > > > {code} > use default; > create table clitest (key int, name String, value String); > insert into table clitest values > (1,"TRUE","1"),(2,"TRUE","1"),(3,"TRUE","1"),(4,"TRUE","1"),(5,"FALSE","0"),(6,"FALSE","0"),(7,"FALSE","0"); > {code} > then run a select query > {code} > # cat /tmp/select.sql > set hive.execution.engine=mr; > select key,name,value > from clitest > where value="1" limit 1; > {code} > Then run beeline via a remote shell, for example > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/select.sql" > root@'s password: > 16/07/12 14:59:22 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > nullkey,name,value > 1,TRUE,1 > null > $ > {code} > In older releases that the output is as follows > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/run.sql" > Are you sure you want to continue connecting (yes/no)? yes > root@'s password: > 16/07/12 14:57:55 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > key,name,value > 1,TRUE,1 > $ > {code} > The output contains nulls instead of blank lines. This is due to the use of > -Djline.terminal=jline.UnsupportedTerminal introduced in HIVE-6758 to be able > to run beeline as a background process. But this is the unfortunate side > effect of that fix. > Running beeline in background also produces garbled output. > {code} > # beeline -u "jdbc:hive2://localhost:1" -n hive -p hive --silent=true > --outputformat=csv2 --showHeader=false -f /tmp/run.sql 2>&1 > > /tmp/beeline.txt & > # cat /tmp/beeline.txt > null1,TRUE,1 > # > {code} > So I think the use of jline.UnsupportedTerminal should be documented but not > used automatically by beeline under the covers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14342) Beeline output is garbled when executed from a remote shell
[ https://issues.apache.org/jira/browse/HIVE-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15410524#comment-15410524 ] Hive QA commented on HIVE-14342: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12822344/HIVE-14342.2.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10440 tests executed *Failed tests:* {noformat} TestMsgBusConnection - did not produce a TEST-*.xml file TestQueryLifeTimeHook - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_llap_counters org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/793/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/793/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-793/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12822344 - PreCommit-HIVE-MASTER-Build > Beeline output is garbled when executed from a remote shell > --- > > Key: HIVE-14342 > URL: https://issues.apache.org/jira/browse/HIVE-14342 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-14342.2.patch, HIVE-14342.patch, HIVE-14342.patch > > > {code} > use default; > create table clitest (key int, name String, value String); > insert into table clitest values > (1,"TRUE","1"),(2,"TRUE","1"),(3,"TRUE","1"),(4,"TRUE","1"),(5,"FALSE","0"),(6,"FALSE","0"),(7,"FALSE","0"); > {code} > then run a select query > {code} > # cat /tmp/select.sql > set hive.execution.engine=mr; > select key,name,value > from clitest > where value="1" limit 1; > {code} > Then run beeline via a remote shell, for example > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/select.sql" > root@'s password: > 16/07/12 14:59:22 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > nullkey,name,value > 1,TRUE,1 > null > $ > {code} > In older releases that the output is as follows > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/run.sql" > Are you sure you want to continue connecting (yes/no)? yes > root@'s password: > 16/07/12 14:57:55 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > key,name,value > 1,TRUE,1 > $ > {code} > The output contains nulls instead of blank lines. This is due to the use of > -Djline.terminal=jline.UnsupportedTerminal introduced in HIVE-6758 to be able > to run beeline as a background process. But this is the unfortunate side > effect of that fix. > Running beeline in background also produces garbled output. > {code} > # beeline -u "jdbc:hive2://localhost:1" -n hive -p hive --silent=true > --outputformat=csv2 --showHeader=false -f /tmp/run.sql 2>&1 > > /tmp/beeline.txt & > # cat /tmp/beeline.txt > null1,TRUE,1 > # > {code} > So I think the use of jline.UnsupportedTerminal should be documented but not > used automatically by beeline under the covers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14342) Beeline output is garbled when executed from a remote shell
[ https://issues.apache.org/jira/browse/HIVE-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15410183#comment-15410183 ] Naveen Gangam commented on HIVE-14342: -- Thanks Mohit. I ran the following test on bash 4.1.2 (tried both conditions, the second condition for Bash 4.2+ does not seem to work as expected). The prompt variable does not seem to be set in any scenario. I ran the test both in background and foreground and the result is the same. Running from a remote node results in the same. {code} # cat test.sh #!/usr/bin/env bash if [ -z $PS1 ] # no prompt? #if [ -v PS1 ] # On Bash 4.2+ ... then echo "non-interactive3" else echo "interactive3" fi # ./test.sh non-interactive3 # ./test.sh & [1] 5715 # non-interactive3 [1]+ Done./test.sh {code} >From a remote node {code} ngangam-MBP-2:~ ngangam$ ssh -l root "/root/test.sh" root@'s password: non-interactive3 {code} In the link above, further down on the page, there is a sample using exactly what the patch is, "-p /dev/stdin". Do you know any other means that might work? Thanks > Beeline output is garbled when executed from a remote shell > --- > > Key: HIVE-14342 > URL: https://issues.apache.org/jira/browse/HIVE-14342 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-14342.2.patch, HIVE-14342.patch, HIVE-14342.patch > > > {code} > use default; > create table clitest (key int, name String, value String); > insert into table clitest values > (1,"TRUE","1"),(2,"TRUE","1"),(3,"TRUE","1"),(4,"TRUE","1"),(5,"FALSE","0"),(6,"FALSE","0"),(7,"FALSE","0"); > {code} > then run a select query > {code} > # cat /tmp/select.sql > set hive.execution.engine=mr; > select key,name,value > from clitest > where value="1" limit 1; > {code} > Then run beeline via a remote shell, for example > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/select.sql" > root@'s password: > 16/07/12 14:59:22 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > nullkey,name,value > 1,TRUE,1 > null > $ > {code} > In older releases that the output is as follows > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/run.sql" > Are you sure you want to continue connecting (yes/no)? yes > root@'s password: > 16/07/12 14:57:55 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > key,name,value > 1,TRUE,1 > $ > {code} > The output contains nulls instead of blank lines. This is due to the use of > -Djline.terminal=jline.UnsupportedTerminal introduced in HIVE-6758 to be able > to run beeline as a background process. But this is the unfortunate side > effect of that fix. > Running beeline in background also produces garbled output. > {code} > # beeline -u "jdbc:hive2://localhost:1" -n hive -p hive --silent=true > --outputformat=csv2 --showHeader=false -f /tmp/run.sql 2>&1 > > /tmp/beeline.txt & > # cat /tmp/beeline.txt > null1,TRUE,1 > # > {code} > So I think the use of jline.UnsupportedTerminal should be documented but not > used automatically by beeline under the covers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14342) Beeline output is garbled when executed from a remote shell
[ https://issues.apache.org/jira/browse/HIVE-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15410064#comment-15410064 ] Mohit Sabharwal commented on HIVE-14342: Thanks, [~ngangam] Wondering if you tried simply detecting if the prompt variable is set ? Seems like that is a more intuitive solution: http://tldp.org/LDP/abs/html/intandnonint.html > Beeline output is garbled when executed from a remote shell > --- > > Key: HIVE-14342 > URL: https://issues.apache.org/jira/browse/HIVE-14342 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-14342.2.patch, HIVE-14342.patch, HIVE-14342.patch > > > {code} > use default; > create table clitest (key int, name String, value String); > insert into table clitest values > (1,"TRUE","1"),(2,"TRUE","1"),(3,"TRUE","1"),(4,"TRUE","1"),(5,"FALSE","0"),(6,"FALSE","0"),(7,"FALSE","0"); > {code} > then run a select query > {code} > # cat /tmp/select.sql > set hive.execution.engine=mr; > select key,name,value > from clitest > where value="1" limit 1; > {code} > Then run beeline via a remote shell, for example > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/select.sql" > root@'s password: > 16/07/12 14:59:22 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > nullkey,name,value > 1,TRUE,1 > null > $ > {code} > In older releases that the output is as follows > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/run.sql" > Are you sure you want to continue connecting (yes/no)? yes > root@'s password: > 16/07/12 14:57:55 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > key,name,value > 1,TRUE,1 > $ > {code} > The output contains nulls instead of blank lines. This is due to the use of > -Djline.terminal=jline.UnsupportedTerminal introduced in HIVE-6758 to be able > to run beeline as a background process. But this is the unfortunate side > effect of that fix. > Running beeline in background also produces garbled output. > {code} > # beeline -u "jdbc:hive2://localhost:1" -n hive -p hive --silent=true > --outputformat=csv2 --showHeader=false -f /tmp/run.sql 2>&1 > > /tmp/beeline.txt & > # cat /tmp/beeline.txt > null1,TRUE,1 > # > {code} > So I think the use of jline.UnsupportedTerminal should be documented but not > used automatically by beeline under the covers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14342) Beeline output is garbled when executed from a remote shell
[ https://issues.apache.org/jira/browse/HIVE-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15409377#comment-15409377 ] Naveen Gangam commented on HIVE-14342: -- [~mohitsabharwal] [~aihuaxu] Could you please review the fix? The proposed fix adds an additional condition to only use the UnsupportedTerminal for beeline running as background process when run locally. The condition detects that beeline is being invoked from a remote shell by checking if stdin for the script is a pipe. > Beeline output is garbled when executed from a remote shell > --- > > Key: HIVE-14342 > URL: https://issues.apache.org/jira/browse/HIVE-14342 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-14342.patch, HIVE-14342.patch > > > {code} > use default; > create table clitest (key int, name String, value String); > insert into table clitest values > (1,"TRUE","1"),(2,"TRUE","1"),(3,"TRUE","1"),(4,"TRUE","1"),(5,"FALSE","0"),(6,"FALSE","0"),(7,"FALSE","0"); > {code} > then run a select query > {code} > # cat /tmp/select.sql > set hive.execution.engine=mr; > select key,name,value > from clitest > where value="1" limit 1; > {code} > Then run beeline via a remote shell, for example > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/select.sql" > root@'s password: > 16/07/12 14:59:22 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > nullkey,name,value > 1,TRUE,1 > null > $ > {code} > In older releases that the output is as follows > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/run.sql" > Are you sure you want to continue connecting (yes/no)? yes > root@'s password: > 16/07/12 14:57:55 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > key,name,value > 1,TRUE,1 > $ > {code} > The output contains nulls instead of blank lines. This is due to the use of > -Djline.terminal=jline.UnsupportedTerminal introduced in HIVE-6758 to be able > to run beeline as a background process. But this is the unfortunate side > effect of that fix. > Running beeline in background also produces garbled output. > {code} > # beeline -u "jdbc:hive2://localhost:1" -n hive -p hive --silent=true > --outputformat=csv2 --showHeader=false -f /tmp/run.sql 2>&1 > > /tmp/beeline.txt & > # cat /tmp/beeline.txt > null1,TRUE,1 > # > {code} > So I think the use of jline.UnsupportedTerminal should be documented but not > used automatically by beeline under the covers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14342) Beeline output is garbled when executed from a remote shell
[ https://issues.apache.org/jira/browse/HIVE-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408918#comment-15408918 ] Hive QA commented on HIVE-14342: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12822205/HIVE-14342.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10440 tests executed *Failed tests:* {noformat} TestMsgBusConnection - did not produce a TEST-*.xml file TestQueryLifeTimeHook - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_orc_llap_counters org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testRevokeTimedOutWorkers {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/781/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/781/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-781/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12822205 - PreCommit-HIVE-MASTER-Build > Beeline output is garbled when executed from a remote shell > --- > > Key: HIVE-14342 > URL: https://issues.apache.org/jira/browse/HIVE-14342 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-14342.patch, HIVE-14342.patch > > > {code} > use default; > create table clitest (key int, name String, value String); > insert into table clitest values > (1,"TRUE","1"),(2,"TRUE","1"),(3,"TRUE","1"),(4,"TRUE","1"),(5,"FALSE","0"),(6,"FALSE","0"),(7,"FALSE","0"); > {code} > then run a select query > {code} > # cat /tmp/select.sql > set hive.execution.engine=mr; > select key,name,value > from clitest > where value="1" limit 1; > {code} > Then run beeline via a remote shell, for example > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/select.sql" > root@'s password: > 16/07/12 14:59:22 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > nullkey,name,value > 1,TRUE,1 > null > $ > {code} > In older releases that the output is as follows > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/run.sql" > Are you sure you want to continue connecting (yes/no)? yes > root@'s password: > 16/07/12 14:57:55 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > key,name,value > 1,TRUE,1 > $ > {code} > The output contains nulls instead of blank lines. This is due to the use of > -Djline.terminal=jline.UnsupportedTerminal introduced in HIVE-6758 to be able > to run beeline as a background process. But this is the unfortunate side > effect of that fix. > Running beeline in background also produces garbled output. > {code} > # beeline -u "jdbc:hive2://localhost:1" -n hive -p hive --silent=true > --outputformat=csv2 --showHeader=false -f /tmp/run.sql 2>&1 > > /tmp/beeline.txt & > # cat /tmp/beeline.txt > null1,TRUE,1 > # > {code} > So I think the use of jline.UnsupportedTerminal should be documented but not > used automatically by beeline under the covers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14342) Beeline output is garbled when executed from a remote shell
[ https://issues.apache.org/jira/browse/HIVE-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408628#comment-15408628 ] Hive QA commented on HIVE-14342: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12822163/HIVE-14342.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/777/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/777/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-777/ Messages: {noformat} This message was trimmed, see log for full details [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/spark-client/target/warehouse [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf [copy] Copying 15 files to /data/hive-ptest/working/apache-github-source-source/spark-client/target/tmp/conf [INFO] Executed tasks [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ spark-client --- [INFO] Compiling 5 source files to /data/hive-ptest/working/apache-github-source-source/spark-client/target/test-classes [INFO] [INFO] --- maven-dependency-plugin:2.8:copy (copy-guava-14) @ spark-client --- [INFO] Configured Artifact: com.google.guava:guava:14.0.1:jar [INFO] Copying guava-14.0.1.jar to /data/hive-ptest/working/apache-github-source-source/spark-client/target/dependency/guava-14.0.1.jar [INFO] [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ spark-client --- [INFO] Tests are skipped. [INFO] [INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ spark-client --- [INFO] Building jar: /data/hive-ptest/working/apache-github-source-source/spark-client/target/spark-client-2.2.0-SNAPSHOT.jar [INFO] [INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ spark-client --- [INFO] [INFO] --- maven-install-plugin:2.4:install (default-install) @ spark-client --- [INFO] Installing /data/hive-ptest/working/apache-github-source-source/spark-client/target/spark-client-2.2.0-SNAPSHOT.jar to /data/hive-ptest/working/maven/org/apache/hive/spark-client/2.2.0-SNAPSHOT/spark-client-2.2.0-SNAPSHOT.jar [INFO] Installing /data/hive-ptest/working/apache-github-source-source/spark-client/pom.xml to /data/hive-ptest/working/maven/org/apache/hive/spark-client/2.2.0-SNAPSHOT/spark-client-2.2.0-SNAPSHOT.pom [INFO] [INFO] [INFO] Building Hive Query Language 2.2.0-SNAPSHOT [INFO] [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-exec --- [INFO] Deleting /data/hive-ptest/working/apache-github-source-source/ql/target [INFO] Deleting /data/hive-ptest/working/apache-github-source-source/ql (includes = [datanucleus.log, derby.log], excludes = []) [INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @ hive-exec --- [INFO] [INFO] --- maven-antrun-plugin:1.7:run (generate-sources) @ hive-exec --- [INFO] Executing tasks main: [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/aggregates/gen [mkdir] Created dir: /data/hive-ptest/working/apache-github-source-source/ql/target/generated-test-sources/java/org/apache/hadoop/hive/ql/exec/vector/expressions/gen Generating vector expression code Generating vector expression test code [INFO] Executed tasks [INFO] [INFO] --- build-helper-maven-plugin:1.8:add-source (add-source) @ hive-exec --- [INFO] Source directory: /data/hive-ptest/working/apache-github-source-source/ql/src/gen/thrift/gen-javabean added. [INFO] Source directory: /data/hive-ptest/working/apache-github-source-source/ql/target/generated-sources/java added. [INFO] [INFO] --- antlr3-maven-plugin:3.4:antlr (default) @ hive-exec --- [INFO] ANTLR: Processing source directory /data/hive-ptest/working/apache-github-source-source/ql/src/java ANTLR Parser Generator Version 3.4 org/apache/hadoop/hive/ql/parse/HiveLexer.g org/apache/hadoop/hive/ql/parse/HiveParser.g [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hive-exec --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hive-exec --- [INFO] Using 'UTF-8' encoding to copy filtere
[jira] [Commented] (HIVE-14342) Beeline output is garbled when executed from a remote shell
[ https://issues.apache.org/jira/browse/HIVE-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15408468#comment-15408468 ] Naveen Gangam commented on HIVE-14342: -- After quite a bit of testing, I found that the most reliable means to detect if beeline is being run via a remote shell is to check if the for the process is a pipe. So I have added an additional check to the condition before using jline's UnsupportedTerminal. The use of UnsupportedTerminal should be limited to beeline processes running locally as a background process. The output now is as we expected prior to this change. {code} ssh -l root "sudo -u hive beeline -u jdbc:hive2://localhost:1 -n hive -p hive --silent=true --outputformat=csv2 -f /tmp/run.sql" root@'s password: 16/08/04 13:46:35 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree module jar containing PrefixTreeCodec is not present. Continuing without it. key,name,value 1,TRUE,1 $ {code} > Beeline output is garbled when executed from a remote shell > --- > > Key: HIVE-14342 > URL: https://issues.apache.org/jira/browse/HIVE-14342 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 2.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > > {code} > use default; > create table clitest (key int, name String, value String); > insert into table clitest values > (1,"TRUE","1"),(2,"TRUE","1"),(3,"TRUE","1"),(4,"TRUE","1"),(5,"FALSE","0"),(6,"FALSE","0"),(7,"FALSE","0"); > {code} > then run a select query > {code} > # cat /tmp/select.sql > set hive.execution.engine=mr; > select key,name,value > from clitest > where value="1" limit 1; > {code} > Then run beeline via a remote shell, for example > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/select.sql" > root@'s password: > 16/07/12 14:59:22 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > nullkey,name,value > 1,TRUE,1 > null > $ > {code} > In older releases that the output is as follows > {code} > $ ssh -l root "sudo -u hive beeline -u > jdbc:hive2://localhost:1 -n hive -p hive --silent=true > --outputformat=csv2 -f /tmp/run.sql" > Are you sure you want to continue connecting (yes/no)? yes > root@'s password: > 16/07/12 14:57:55 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree > module jar containing PrefixTreeCodec is not present. Continuing without it. > key,name,value > 1,TRUE,1 > $ > {code} > The output contains nulls instead of blank lines. This is due to the use of > -Djline.terminal=jline.UnsupportedTerminal introduced in HIVE-6758 to be able > to run beeline as a background process. But this is the unfortunate side > effect of that fix. > Running beeline in background also produces garbled output. > {code} > # beeline -u "jdbc:hive2://localhost:1" -n hive -p hive --silent=true > --outputformat=csv2 --showHeader=false -f /tmp/run.sql 2>&1 > > /tmp/beeline.txt & > # cat /tmp/beeline.txt > null1,TRUE,1 > # > {code} > So I think the use of jline.UnsupportedTerminal should be documented but not > used automatically by beeline under the covers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)