[jira] [Commented] (HIVE-13643) Various enhancements / fixes to llap cli tools
[ https://issues.apache.org/jira/browse/HIVE-13643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16104507#comment-16104507 ] Lefty Leverenz commented on HIVE-13643: --- Doc note: The findAppTimeout option is documented in the LLAP Status section of the LLAP wikidoc. * [LLAP -- LLAP Status | https://cwiki.apache.org/confluence/display/Hive/LLAP#LLAP-LLAPStatus] > Various enhancements / fixes to llap cli tools > -- > > Key: HIVE-13643 > URL: https://issues.apache.org/jira/browse/HIVE-13643 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.1.0 > > Attachments: HIVE-13643.01.patch, HIVE-13643.02.patch > > > - BUG - llapstatus does not always produce output (PrintWriter.close) > - llapstatus should parse hiveconf > - package llap-cli-log4j2.template > - Log at the start and end of the script to include parameters > - Generate logs to a file different from hive.log > - hive --service llap --name does not propagate the name to the daemons -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-13643) Various enhancements / fixes to llap cli tools
[ https://issues.apache.org/jira/browse/HIVE-13643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288099#comment-15288099 ] Lefty Leverenz commented on HIVE-13643: --- Okay, thanks. > Various enhancements / fixes to llap cli tools > -- > > Key: HIVE-13643 > URL: https://issues.apache.org/jira/browse/HIVE-13643 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.1.0 > > Attachments: HIVE-13643.01.patch, HIVE-13643.02.patch > > > - BUG - llapstatus does not always produce output (PrintWriter.close) > - llapstatus should parse hiveconf > - package llap-cli-log4j2.template > - Log at the start and end of the script to include parameters > - Generate logs to a file different from hive.log > - hive --service llap --name does not propagate the name to the daemons -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13643) Various enhancements / fixes to llap cli tools
[ https://issues.apache.org/jira/browse/HIVE-13643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15287904#comment-15287904 ] Siddharth Seth commented on HIVE-13643: --- This doesn't require additional documentation, mostly bug fixes and minor enhancements. > Various enhancements / fixes to llap cli tools > -- > > Key: HIVE-13643 > URL: https://issues.apache.org/jira/browse/HIVE-13643 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.1.0 > > Attachments: HIVE-13643.01.patch, HIVE-13643.02.patch > > > - BUG - llapstatus does not always produce output (PrintWriter.close) > - llapstatus should parse hiveconf > - package llap-cli-log4j2.template > - Log at the start and end of the script to include parameters > - Generate logs to a file different from hive.log > - hive --service llap --name does not propagate the name to the daemons -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13643) Various enhancements / fixes to llap cli tools
[ https://issues.apache.org/jira/browse/HIVE-13643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268845#comment-15268845 ] Siddharth Seth commented on HIVE-13643: --- Oops. I mistook the following as a +1. bq. slider apps can still use hardcoded registry, can't they? Otherwise looks good hive --service llap and hive --service llapstatus don't really work for non slider apps. Let me know if something needs to be fixed, or this needs to be reverted. I saw you've created a new jira - HIVE-13668. > Various enhancements / fixes to llap cli tools > -- > > Key: HIVE-13643 > URL: https://issues.apache.org/jira/browse/HIVE-13643 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.1.0 > > Attachments: HIVE-13643.01.patch, HIVE-13643.02.patch > > > - BUG - llapstatus does not always produce output (PrintWriter.close) > - llapstatus should parse hiveconf > - package llap-cli-log4j2.template > - Log at the start and end of the script to include parameters > - Generate logs to a file different from hive.log > - hive --service llap --name does not propagate the name to the daemons -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13643) Various enhancements / fixes to llap cli tools
[ https://issues.apache.org/jira/browse/HIVE-13643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15267124#comment-15267124 ] Sergey Shelukhin commented on HIVE-13643: - I didn't actually +1 this :) > Various enhancements / fixes to llap cli tools > -- > > Key: HIVE-13643 > URL: https://issues.apache.org/jira/browse/HIVE-13643 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.1.0 > > Attachments: HIVE-13643.01.patch, HIVE-13643.02.patch > > > - BUG - llapstatus does not always produce output (PrintWriter.close) > - llapstatus should parse hiveconf > - package llap-cli-log4j2.template > - Log at the start and end of the script to include parameters > - Generate logs to a file different from hive.log > - hive --service llap --name does not propagate the name to the daemons -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13643) Various enhancements / fixes to llap cli tools
[
https://issues.apache.org/jira/browse/HIVE-13643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265581#comment-15265581
]
Hive QA commented on HIVE-13643:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12801410/HIVE-13643.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 58 failed/errored test(s), 9944 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_decimal_2.q-explainuser_1.q-explainuser_3.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorization_16.q-vector_decimal_round.q-orc_merge6.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorized_parquet.q-vector_decimal_aggregate.q-tez_self_join.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby1_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_nomore_ambiguous_table_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_regexp_extract
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern3
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_clustern4
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_nonkey_groupby
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_selectDistinctStarNeg_2
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_subquery_shared_alias
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udtf_not_supported1
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.metastore.hbase.TestHBaseImport.org.apache.hadoop.hive.metastore.hbase.TestHBaseImport
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
[jira] [Commented] (HIVE-13643) Various enhancements / fixes to llap cli tools
[ https://issues.apache.org/jira/browse/HIVE-13643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265192#comment-15265192 ] Siddharth Seth commented on HIVE-13643: --- No. That won't work. We should add some validation around this when trying to use hive --service llap. Sparate jira though. llapstatus will also not work for such clusters. There's some checks and log messages around that. > Various enhancements / fixes to llap cli tools > -- > > Key: HIVE-13643 > URL: https://issues.apache.org/jira/browse/HIVE-13643 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13643.01.patch, HIVE-13643.02.patch > > > - BUG - llapstatus does not always produce output (PrintWriter.close) > - llapstatus should parse hiveconf > - package llap-cli-log4j2.template > - Log at the start and end of the script to include parameters > - Generate logs to a file different from hive.log > - hive --service llap --name does not propagate the name to the daemons -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13643) Various enhancements / fixes to llap cli tools
[ https://issues.apache.org/jira/browse/HIVE-13643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264525#comment-15264525 ] Sergey Shelukhin commented on HIVE-13643: - What I mean is, if I try to start a cluster w/o a name on command line, with hardcoded registry, the cluster name may become something like "mynode1.foo.bar,mynode2.foo.bar,mynode3.foo.bar" etc. 1) Will this even work? 2) Is this what we want? > Various enhancements / fixes to llap cli tools > -- > > Key: HIVE-13643 > URL: https://issues.apache.org/jira/browse/HIVE-13643 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13643.01.patch, HIVE-13643.02.patch > > > - BUG - llapstatus does not always produce output (PrintWriter.close) > - llapstatus should parse hiveconf > - package llap-cli-log4j2.template > - Log at the start and end of the script to include parameters > - Generate logs to a file different from hive.log > - hive --service llap --name does not propagate the name to the daemons -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13643) Various enhancements / fixes to llap cli tools
[ https://issues.apache.org/jira/browse/HIVE-13643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264517#comment-15264517 ] Siddharth Seth commented on HIVE-13643: --- bq. slider apps can still use hardcoded registry, can't they? the patch doesn't change anything related to this. > Various enhancements / fixes to llap cli tools > -- > > Key: HIVE-13643 > URL: https://issues.apache.org/jira/browse/HIVE-13643 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13643.01.patch, HIVE-13643.02.patch > > > - BUG - llapstatus does not always produce output (PrintWriter.close) > - llapstatus should parse hiveconf > - package llap-cli-log4j2.template > - Log at the start and end of the script to include parameters > - Generate logs to a file different from hive.log > - hive --service llap --name does not propagate the name to the daemons -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13643) Various enhancements / fixes to llap cli tools
[ https://issues.apache.org/jira/browse/HIVE-13643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264375#comment-15264375 ] Sergey Shelukhin commented on HIVE-13643: - slider apps can still use hardcoded registry, can't they? Otherwise looks good > Various enhancements / fixes to llap cli tools > -- > > Key: HIVE-13643 > URL: https://issues.apache.org/jira/browse/HIVE-13643 > Project: Hive > Issue Type: Improvement >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-13643.01.patch, HIVE-13643.02.patch > > > - BUG - llapstatus does not always produce output (PrintWriter.close) > - llapstatus should parse hiveconf > - package llap-cli-log4j2.template > - Log at the start and end of the script to include parameters > - Generate logs to a file different from hive.log > - hive --service llap --name does not propagate the name to the daemons -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13643) Various enhancements / fixes to llap cli tools
[
https://issues.apache.org/jira/browse/HIVE-13643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15262734#comment-15262734
]
Sergey Shelukhin commented on HIVE-13643:
-
Nit: is it possible to capitalize LLAP in log lines?
{noformat}
if (StringUtils.isEmpty(appName)) {
+appName = HiveConf.getVar(conf,
HiveConf.ConfVars.LLAP_DAEMON_SERVICE_HOSTS);
+if (appName.startsWith("@") && appName.length() > 1) {
+ appName = appName.substring(1);
+}
+ }
+ if (StringUtils.isEmpty(appName) || (appName.startsWith("@") ||
appName.length() == 1)) {
+String message = "Invalid app name. This must be setup via config or
passed in as a parameter";
+LOG.info(message);
+return ExitCode.INCORRECT_USAGE.getInt();
+ }
{noformat}
Would an app name being a list of hosts with dots, commas and stuff actually
work? And make sense?
Also why cannot the length be 1
> Various enhancements / fixes to llap cli tools
> --
>
> Key: HIVE-13643
> URL: https://issues.apache.org/jira/browse/HIVE-13643
> Project: Hive
> Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-13643.01.patch
>
>
> - BUG - llapstatus does not always produce output (PrintWriter.close)
> - llapstatus should parse hiveconf
> - package llap-cli-log4j2.template
> - Log at the start and end of the script to include parameters
> - Generate logs to a file different from hive.log
> - hive --service llap --name does not propagate the name to the daemons
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
