[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[ https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15362914#comment-15362914 ] Prasanth Jayachandran commented on HIVE-10100: -- The test failures look related. I will look at it shortly. > Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0 > -- > > Key: HIVE-10100 > URL: https://issues.apache.org/jira/browse/HIVE-10100 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.2.0 >Reporter: Gunther Hagleitner >Assignee: Prasanth Jayachandran >Priority: Critical > Attachments: HIVE-10100.1.patch, HIVE-10100.2.patch, > HIVE-10100.3.patch, yarn_bin.patch > > > HADOOP-11257 adds a warning to stdout > {noformat} > WARNING: Use "yarn jar" to launch YARN applications. > {noformat} > which will cause issues if untreated with folks that programatically parse > stdout for query results (i.e.: CLI, silent mode, etc). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[ https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15360201#comment-15360201 ] Hive QA commented on HIVE-10100: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12815803/HIVE-10100.3.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 28 failed/errored test(s), 10287 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_stats_orc org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_excludeHadoop20 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_8 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_fetch_aggregation org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nonmr_fetch org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_analyze org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample10 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook_use_metadata org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_parquet org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver_dboutput org.apache.hadoop.hive.cli.TestContribNegativeCliDriver.testNegativeCliDriver_case_with_row_sequence org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_hash org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_join_hash org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapPlan1 org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapPlan2 org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan1 org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan2 org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan3 org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan4 org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan5 org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan6 {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/349/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/349/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-349/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 28 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12815803 - PreCommit-HIVE-MASTER-Build > Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0 > -- > > Key: HIVE-10100 > URL: https://issues.apache.org/jira/browse/HIVE-10100 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.2.0 >Reporter: Gunther Hagleitner >Assignee: Prasanth Jayachandran >Priority: Critical > Attachments: HIVE-10100.1.patch, HIVE-10100.2.patch, > HIVE-10100.3.patch, yarn_bin.patch > > > HADOOP-11257 adds a warning to stdout > {noformat} > WARNING: Use "yarn jar" to launch YARN applications. > {noformat} > which will cause issues if untreated with folks that programatically parse > stdout for query results (i.e.: CLI, silent mode, etc). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[ https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15353371#comment-15353371 ] Gunther Hagleitner commented on HIVE-10100: --- [~sseth] we can ask, but the warning has been around for a long time. My guess eventually the idea is to get rid of it. So might as well change these places. > Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0 > -- > > Key: HIVE-10100 > URL: https://issues.apache.org/jira/browse/HIVE-10100 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.2.0 >Reporter: Gunther Hagleitner >Assignee: Prasanth Jayachandran >Priority: Critical > Attachments: HIVE-10100.1.patch, HIVE-10100.2.patch, yarn_bin.patch > > > HADOOP-11257 adds a warning to stdout > {noformat} > WARNING: Use "yarn jar" to launch YARN applications. > {noformat} > which will cause issues if untreated with folks that programatically parse > stdout for query results (i.e.: CLI, silent mode, etc). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[ https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352322#comment-15352322 ] Siddharth Seth commented on HIVE-10100: --- Why is this critical? - due to the confusion caused by the error message printed. It may be worth asking for a way to suppress this specific message rather than trying to change Hive code all over the place. 'yarn jar metastore' <- what does this mean ? > Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0 > -- > > Key: HIVE-10100 > URL: https://issues.apache.org/jira/browse/HIVE-10100 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.2.0 >Reporter: Gunther Hagleitner >Assignee: Prasanth Jayachandran >Priority: Critical > Attachments: HIVE-10100.1.patch, HIVE-10100.2.patch, yarn_bin.patch > > > HADOOP-11257 adds a warning to stdout > {noformat} > WARNING: Use "yarn jar" to launch YARN applications. > {noformat} > which will cause issues if untreated with folks that programatically parse > stdout for query results (i.e.: CLI, silent mode, etc). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[ https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352319#comment-15352319 ] Siddharth Seth commented on HIVE-10100: --- hadoop jar from within Java code (instead of using the programatic API) ? - I'm sure there's a good reason to do this :) > Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0 > -- > > Key: HIVE-10100 > URL: https://issues.apache.org/jira/browse/HIVE-10100 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.2.0 >Reporter: Gunther Hagleitner >Assignee: Prasanth Jayachandran >Priority: Critical > Attachments: HIVE-10100.1.patch, HIVE-10100.2.patch, yarn_bin.patch > > > HADOOP-11257 adds a warning to stdout > {noformat} > WARNING: Use "yarn jar" to launch YARN applications. > {noformat} > which will cause issues if untreated with folks that programatically parse > stdout for query results (i.e.: CLI, silent mode, etc). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[ https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352141#comment-15352141 ] Siddharth Seth commented on HIVE-10100: --- YARN_USER_CLASSPATH needs to be set in a couple more scripts - beeline, hplsql, hwi. Other than that, looks good to me. Don't really know why hadoop jar was deprecated in favor of yarn jar, and what the difference between the two is. > Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0 > -- > > Key: HIVE-10100 > URL: https://issues.apache.org/jira/browse/HIVE-10100 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.2.0 >Reporter: Gunther Hagleitner >Assignee: Prasanth Jayachandran >Priority: Critical > Attachments: HIVE-10100.1.patch, HIVE-10100.2.patch > > > HADOOP-11257 adds a warning to stdout > {noformat} > WARNING: Use "yarn jar" to launch YARN applications. > {noformat} > which will cause issues if untreated with folks that programatically parse > stdout for query results (i.e.: CLI, silent mode, etc). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[ https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351931#comment-15351931 ] Prasanth Jayachandran commented on HIVE-10100: -- bq.For HADOOP_CLASSPATH_FIRST and YARN_CLASSPATH_FIRST - should we use the value set in HADOOP_CLASSPATH_FIRST or the one used in YARN_CLASSPATH_FIRST ? I think we need this to be always true for proper initialization of log4j2 and jline. > Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0 > -- > > Key: HIVE-10100 > URL: https://issues.apache.org/jira/browse/HIVE-10100 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.2.0 >Reporter: Gunther Hagleitner >Assignee: Prasanth Jayachandran >Priority: Critical > Attachments: HIVE-10100.1.patch > > > HADOOP-11257 adds a warning to stdout > {noformat} > WARNING: Use "yarn jar" to launch YARN applications. > {noformat} > which will cause issues if untreated with folks that programatically parse > stdout for query results (i.e.: CLI, silent mode, etc). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[ https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15351929#comment-15351929 ] Prasanth Jayachandran commented on HIVE-10100: -- bq. Why is this snippet used in some files, but not others ? Some scripts explicitly modifies the variables after hive script sets it up to append service specific info. Example: beeline seems to be doing some ordering of classpath and metastore appends its own java opts which might not be required otherwise. I can't find YARN_CLASSPATH in yarn script. I think after some version YARN_CLASSPATH is renamed to YARN_USER_CLASSPATH. So I added HADOOP_CLASSPATH to YARN_USER_CLASSPATH in the next patch. Yes. Hive script is slow for 2 reasons 1) Automatic initialization of logging (when trying to run hadoop or hbase commands entire classpath is searched to look for log4j2.properties file) 2) hbase mapredcp takes time There is some info here HIVE-12497. Regardless we should bring down the time. > Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0 > -- > > Key: HIVE-10100 > URL: https://issues.apache.org/jira/browse/HIVE-10100 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.2.0 >Reporter: Gunther Hagleitner >Assignee: Prasanth Jayachandran >Priority: Critical > Attachments: HIVE-10100.1.patch > > > HADOOP-11257 adds a warning to stdout > {noformat} > WARNING: Use "yarn jar" to launch YARN applications. > {noformat} > which will cause issues if untreated with folks that programatically parse > stdout for query results (i.e.: CLI, silent mode, etc). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[ https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15347520#comment-15347520 ] Prasanth Jayachandran commented on HIVE-10100: -- [~gopalv]/[~sseth] Can someone please review this patch? > Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0 > -- > > Key: HIVE-10100 > URL: https://issues.apache.org/jira/browse/HIVE-10100 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.2.0 >Reporter: Gunther Hagleitner >Assignee: Prasanth Jayachandran >Priority: Critical > Attachments: HIVE-10100.1.patch > > > HADOOP-11257 adds a warning to stdout > {noformat} > WARNING: Use "yarn jar" to launch YARN applications. > {noformat} > which will cause issues if untreated with folks that programatically parse > stdout for query results (i.e.: CLI, silent mode, etc). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[ https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384348#comment-14384348 ] Gunther Hagleitner commented on HIVE-10100: --- Thank you [~cnauroth]! Lowering priority. Will keep open to switch from "hadoop jar" to "yarn jar" to get rid of the warning over time. > Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0 > -- > > Key: HIVE-10100 > URL: https://issues.apache.org/jira/browse/HIVE-10100 > Project: Hive > Issue Type: Bug >Reporter: Gunther Hagleitner >Priority: Critical > > HADOOP-11257 adds a warning to stdout > {noformat} > WARNING: Use "yarn jar" to launch YARN applications. > {noformat} > which will cause issues if untreated with folks that programatically parse > stdout for query results (i.e.: CLI, silent mode, etc). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10100) Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0
[ https://issues.apache.org/jira/browse/HIVE-10100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383330#comment-14383330 ] Chris Nauroth commented on HIVE-10100: -- I committed an addendum patch for HADOOP-11257 to redirect this warning to stderr so that it won't get misinterpreted as part of Hive query results. > Warning "yarn jar" instead of "hadoop jar" in hadoop 2.7.0 > -- > > Key: HIVE-10100 > URL: https://issues.apache.org/jira/browse/HIVE-10100 > Project: Hive > Issue Type: Bug >Reporter: Gunther Hagleitner >Priority: Blocker > > HADOOP-11257 adds a warning to stdout > {noformat} > WARNING: Use "yarn jar" to launch YARN applications. > {noformat} > which will cause issues if untreated with folks that programatically parse > stdout for query results (i.e.: CLI, silent mode, etc). -- This message was sent by Atlassian JIRA (v6.3.4#6332)