[jira] [Commented] (HIVE-13817) Allow DNS CNAME ALIAS Resolution from apache hive beeline JDBC URL to allow for failover
[
https://issues.apache.org/jira/browse/HIVE-13817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295939#comment-15295939
]
Vijay Singh commented on HIVE-13817:
Also verified that hive2 action invoked from oozie workflow with CNAME alias
string and useCanonicalName=true successfully obtains delegationtokens and
executes the workflow. Please see invocation and verfication log excerpts.
{code}
[root@vjd-1 hive2]# cat hive2job.properties
nameNode=hdfs://vjd-1.gce.cloudera.com:8020
jobTracker=yarnRM
queueName=default
jdbcURL=jdbc:hive2://vjlb.gce.cloudera.com:1/default;useCanonicalName=true;
jdbcPrincipal=hive/_h...@gce.cloudera.com
examplesRoot=/testproject/code/examplesoozie
oozie.use.system.libpath=true
oozie.wf.application.path=${nameNode}${examplesRoot}/hive2
[root@vjd-1 hive2]# cat workflow.xml
hive2.server.principal
${jdbcPrincipal}
hive2.jdbc.url
${jdbcURL}
${jobTracker}
${nameNode}
mapred.job.queue.name
${queueName}
${jdbcURL}
script.q
INPUT=${examplesRoot}/input-data/table
OUTPUT=${examplesRoot}/output-data/hive2
Hive2 (Beeline) action failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]
[root@vjd-1 hive2]# oozie job -oozie
http://vjtest-local.gce.cloudera.com:11000/oozie -config hive2job.properties
-run
job: 041-16052507746-oozie-oozi-W
[root@vjd-1 hive2]# oozie job -oozie
http://vjtest-local.gce.cloudera.com:11000/oozie -info
041-16052507746-oozie-oozi-W
Job ID : 041-16052507746-oozie-oozi-W
Workflow Name : hive2-wf
App Path :
hdfs://vjd-1.gce.cloudera.com:8020/testproject/code/examplesoozie/hive2
Status: SUCCEEDED
Run : 0
User : vijay
Group : -
Created : 2016-05-23 05:38 GMT
Started : 2016-05-23 05:38 GMT
Last Modified : 2016-05-23 05:38 GMT
Ended : 2016-05-23 05:38 GMT
CoordAction ID: -
Actions
ID
StatusExt ID Ext Status Err Code
041-16052507746-oozie-oozi-W@:start:
OK- OK -
041-16052507746-oozie-oozi-W@hive2-node
OKjob_1463802572747_0018 SUCCEEDED -
041-16052507746-oozie-oozi-W@end
OK- OK -
[root@vjd-1 hive2]# cat script.q
DROP TABLE IF EXISTS test;
SELECT * FROM sample_07;
[root@vjd-1 hive2]#
{code}
> Allow DNS CNAME ALIAS Resolution from apache hive beeline JDBC URL to allow
> for failover
>
>
> Key: HIVE-13817
> URL: https://issues.apache.org/jira/browse/HIVE-13817
> Project: Hive
> Issue Type: New Feature
> Components: Beeline
>Affects Versions: 1.2.1
>Reporter: Vijay Singh
> Attachments: HIVE-13817.1.patch, HIVE-13817.2.patch,
> HIVE-13817.3.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Currently, in case of BDR clusters, DNS CNAME alias based connections fail.
> As _HOST resolves to exact endpoint specified in connection string and that
> may not be intended SPN for kerberos based on reverse DNS lookup.
> Consequently this JIRA proposes that client specific setting be used to
> resolv _HOST from CNAME DNS alias to A record entry on the fly in beeline.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295936#comment-15295936 ] Ashutosh Chauhan commented on HIVE-13068: - +1 assuming failures are either unrelated or just update of golden files. > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch, > HIVE-13068.02.patch, HIVE-13068.03.patch, HIVE-13068.04.patch, > HIVE-13068.04.patch, HIVE-13068.05.patch, HIVE-13068.06.patch, > HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[
https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295926#comment-15295926
]
Hive QA commented on HIVE-13068:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12805543/HIVE-13068.06.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 72 failed/errored test(s), 10009 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-constprog_dpp.q-dynamic_partition_pruning.q-vectorization_10.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-explainuser_4.q-update_after_multiple_inserts.q-mapreduce2.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-schema_evol_text_nonvec_mapwork_table.q-vector_decimal_trailing.q-subquery_in.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-script_pipe.q-vector_decimal_aggregate.q-vector_data_types.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join4.q-groupby_cube1.q-auto_join20.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-multi_insert.q-join5.q-groupby6.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-parallel_join1.q-escape_distributeby1.q-auto_sortmerge_join_7.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-ptf_rcfile.q-bucketmapjoin_negative.q-bucket_map_join_spark2.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_lineage2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_outer_join_ppr
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_udaf_percentile_approx_23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropWhen
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constprog2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_decimal_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_stale_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_table_access_keys_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_round_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_mapjoin_reduce
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_null_projection
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_remote_script
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_root_dir_external_table
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_partition_column_names_with_leading_and_trailing_spaces
[jira] [Updated] (HIVE-13409) Fix JDK8 test failures related to COLUMN_STATS_ACCURATE
[ https://issues.apache.org/jira/browse/HIVE-13409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-13409: --- Attachment: HIVE-13409-java8.patch > Fix JDK8 test failures related to COLUMN_STATS_ACCURATE > --- > > Key: HIVE-13409 > URL: https://issues.apache.org/jira/browse/HIVE-13409 > Project: Hive > Issue Type: Sub-task > Components: Tests >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-13409-java8.patch, HIVE-13409-java8.patch, > HIVE-13409-java8.patch > > > 126 failures have crept into JDK8 tests since we resolved HIVE-8607 > http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/ > Majority relate to the ordering of a "COLUMN_STATS_ACCURATE" partition > property. > Looks like a simple fix, use ordered map in > HiveStringUtils.getPropertiesExplain() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13409) Fix JDK8 test failures related to COLUMN_STATS_ACCURATE
[
https://issues.apache.org/jira/browse/HIVE-13409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295870#comment-15295870
]
Hive QA commented on HIVE-13409:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12805549/HIVE-13409-java8.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 107 failed/errored test(s), 9848 tests
executed
*Failed tests:*
{noformat}
TestCliDriver-gen_udf_example_add10.q-ppd_join4.q-union27.q-and-12-more - did
not produce a TEST-*.xml file
TestCliDriver-partition_timestamp.q-ppd_random.q-vector_outer_join5.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-ptf_general_queries.q-unionDistinct_1.q-groupby1_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestJdbcWithMiniHA - did not produce a TEST-*.xml file
TestJdbcWithMiniMr - did not produce a TEST-*.xml file
TestMiniLlapCliDriver - did not produce a TEST-*.xml file
TestMiniTezCliDriver-enforce_order.q-vector_partition_diff_num_cols.q-unionDistinct_1.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-insert_values_non_partitioned.q-join1.q-schema_evol_orc_nonvec_mapwork_part.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-script_pipe.q-vector_decimal_aggregate.q-acid_globallimit.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-smb_cache.q-transform_ppr2.q-vector_outer_join0.q-and-3-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorized_parquet.q-orc_merge4.q-orc_merge_incompat1.q-and-12-more
- did not produce a TEST-*.xml file
TestOperationLoggingAPIWithTez - did not produce a TEST-*.xml file
TestSparkCliDriver-bucketsortoptimize_insert_7.q-smb_mapjoin_15.q-mapreduce1.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-skewjoinopt3.q-union27.q-multigroupby_singlemr.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_json_serde1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_collect_set_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_sort_array
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_udf1
org.apache.hadoop.hive.cli.TestContribNegativeCliDriver.testNegativeCliDriver_case_with_row_sequence
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query85
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query89
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query91
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby1_noskew
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_multi_insert_common_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_input_part2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join0
[jira] [Commented] (HIVE-13797) Provide a connection string example in beeline
[
https://issues.apache.org/jira/browse/HIVE-13797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295761#comment-15295761
]
Hive QA commented on HIVE-13797:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12805339/HIVE-13797.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 66 failed/errored test(s), 10028 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-dynpart_sort_optimization2.q-tez_dynpart_hashjoin_3.q-orc_vectorization_ppd.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-smb_cache.q-transform_ppr2.q-vector_outer_join0.q-and-5-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby10.q-groupby4_noskew.q-union5.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-mapreduce2.q-groupby7_noskew.q-vectorization_5.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-table_access_keys_stats.q-bucketsortoptimize_insert_4.q-runtime_skewjoin_mapjoin_spark.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join21
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join28
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby6
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby7_map
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join5
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_12
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vectorization_4
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionQueueComparator
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs
org.apache.hadoop.hive.metastore.TestHiveMetaStoreStatsMerge.testStatsMerge
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
[jira] [Updated] (HIVE-13409) Fix JDK8 test failures related to COLUMN_STATS_ACCURATE
[ https://issues.apache.org/jira/browse/HIVE-13409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-13409: --- Attachment: HIVE-13409-java8.patch > Fix JDK8 test failures related to COLUMN_STATS_ACCURATE > --- > > Key: HIVE-13409 > URL: https://issues.apache.org/jira/browse/HIVE-13409 > Project: Hive > Issue Type: Sub-task > Components: Tests >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-13409-java8.patch, HIVE-13409-java8.patch > > > 126 failures have crept into JDK8 tests since we resolved HIVE-8607 > http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-TRUNK-JAVA8/ > Majority relate to the ordering of a "COLUMN_STATS_ACCURATE" partition > property. > Looks like a simple fix, use ordered map in > HiveStringUtils.getPropertiesExplain() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13810) insert overwrite select from some table fails throwing org.apache.hadoop.security.AccessControlException
[
https://issues.apache.org/jira/browse/HIVE-13810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295711#comment-15295711
]
Hive QA commented on HIVE-13810:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12805322/HIVE-13810.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 56 failed/errored test(s), 9927 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_join30.q-vector_decimal_10_0.q-acid_globallimit.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-auto_sortmerge_join_7.q-orc_merge9.q-tez_union_dynamic_partition.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-groupby2.q-tez_dynpart_hashjoin_1.q-custom_input_output_format.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorization_13.q-auto_sortmerge_join_13.q-tez_bmj_schema_evolution.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-auto_join9.q-groupby_map_ppr.q-ppd_join3.q-and-12-more - did
not produce a TEST-*.xml file
TestSparkCliDriver-auto_join_reordering_values.q-ptf_seqfile.q-auto_join18.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-avro_joins.q-join36.q-join1.q-and-12-more - did not produce
a TEST-*.xml file
TestSparkCliDriver-groupby3_map.q-skewjoinopt8.q-union_remove_1.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-groupby_complex_types.q-groupby_map_ppr_multi_distinct.q-vectorization_16.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-join4.q-groupby_cube1.q-auto_join20.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-load_dyn_part5.q-load_dyn_part2.q-skewjoinopt16.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-multi_insert.q-join5.q-groupby6.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-script_pipe.q-stats12.q-auto_join24.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby7_noskew
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_mapreduce2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_move_tasks_share_dependencies
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_sample9
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
[jira] [Commented] (HIVE-13817) Allow DNS CNAME ALIAS Resolution from apache hive beeline JDBC URL to allow for failover
[
https://issues.apache.org/jira/browse/HIVE-13817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295707#comment-15295707
]
Vijay Singh commented on HIVE-13817:
Also ran the same tests with impala using modified jdbc driver and SSL turned
on successfully.
{code}
[root@vjd-1 ~]# beeline -u
"jdbc:hive2://vjlb.gce.cloudera.com:21050/default;principal=impala/_h...@gce.cloudera.com;ssl=true"
-e "show tables;"
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated
and will likely be removed in a future release
16/05/22 13:02:47 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree
module jar containing PrefixTreeCodec is not present. Continuing without it.
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
scan complete in 2ms
Connecting to
jdbc:hive2://vjlb.gce.cloudera.com:21050/default;principal=impala/_h...@gce.cloudera.com;ssl=true
Error: Could not open client transport with JDBC Uri:
jdbc:hive2://vjlb.gce.cloudera.com:21050/default;principal=impala/_h...@gce.cloudera.com;ssl=true:
null (state=08S01,code=0)
No current connection
Error: Could not open client transport with JDBC Uri:
jdbc:hive2://vjlb.gce.cloudera.com:21050/default;principal=impala/_h...@gce.cloudera.com;ssl=true:
null (state=08S01,code=0)
[root@vjd-1 ~]# beeline -u
"jdbc:hive2://vjtest-local.gce.cloudera.com:21050/default;principal=impala/_h...@gce.cloudera.com;ssl=true"
-e "show tables;"
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated
and will likely be removed in a future release
16/05/22 13:03:16 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree
module jar containing PrefixTreeCodec is not present. Continuing without it.
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
scan complete in 2ms
Connecting to
jdbc:hive2://vjtest-local.gce.cloudera.com:21050/default;principal=impala/_h...@gce.cloudera.com;ssl=true
Connected to: Impala (version 2.5.0-cdh5.7.0)
Driver: Hive JDBC (version 1.1.0-cdh5.7.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
customers
sample_07
sample_08
web_logs
+---+--+
| name |
+---+--+
+---+--+
No rows selected (2.014 seconds)
Beeline version 1.1.0-cdh5.7.0 by Apache Hive
Closing: 0:
jdbc:hive2://vjtest-local.gce.cloudera.com:21050/default;principal=impala/_h...@gce.cloudera.com;ssl=true
[root@vjd-1 ~]# beeline -u
"jdbc:hive2://vjtest-local.gce.cloudera.com:21050/default;principal=impala/_h...@gce.cloudera.com;ssl=true;useCanonicalName=true"
-e "show tables;"
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated
and will likely be removed in a future release
16/05/22 13:03:42 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree
module jar containing PrefixTreeCodec is not present. Continuing without it.
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
scan complete in 2ms
Connecting to
jdbc:hive2://vjtest-local.gce.cloudera.com:21050/default;principal=impala/_h...@gce.cloudera.com;ssl=true;useCanonicalName=true
Connected to: Impala (version 2.5.0-cdh5.7.0)
Driver: Hive JDBC (version 1.1.0-cdh5.7.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
customers
sample_07
sample_08
web_logs
+---+--+
| name |
+---+--+
+---+--+
No rows selected (2.027 seconds)
Beeline version 1.1.0-cdh5.7.0 by Apache Hive
Closing: 0:
jdbc:hive2://vjtest-local.gce.cloudera.com:21050/default;principal=impala/_h...@gce.cloudera.com;ssl=true;useCanonicalName=true
[root@vjd-1 ~]# beeline -u
"jdbc:hive2://vjlb.gce.cloudera.com:21050/default;principal=impala/_h...@gce.cloudera.com;ssl=true;useCanonicalName=true"
-e "show tables;"
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated
and will likely be removed in a future release
16/05/22 13:04:05 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree
module jar containing PrefixTreeCodec is not present. Continuing without it.
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
scan complete in 2ms
Connecting to
jdbc:hive2://vjlb.gce.cloudera.com:21050/default;principal=impala/_h...@gce.cloudera.com;ssl=true;useCanonicalName=true
Connected to: Impala (version 2.5.0-cdh5.7.0)
Driver: Hive JDBC (version 1.1.0-cdh5.7.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
customers
sample_07
sample_08
web_logs
+---+--+
|
[jira] [Updated] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13068: --- Attachment: HIVE-13068.06.patch > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch, > HIVE-13068.02.patch, HIVE-13068.03.patch, HIVE-13068.04.patch, > HIVE-13068.04.patch, HIVE-13068.05.patch, HIVE-13068.06.patch, > HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13068: --- Attachment: (was: HIVE-13068.06.patch) > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch, > HIVE-13068.02.patch, HIVE-13068.03.patch, HIVE-13068.04.patch, > HIVE-13068.04.patch, HIVE-13068.05.patch, HIVE-13068.06.patch, > HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13068: --- Attachment: HIVE-13068.06.patch > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch, > HIVE-13068.02.patch, HIVE-13068.03.patch, HIVE-13068.04.patch, > HIVE-13068.04.patch, HIVE-13068.05.patch, HIVE-13068.06.patch, > HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13068: --- Attachment: (was: HIVE-13068.06.patch) > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch, > HIVE-13068.02.patch, HIVE-13068.03.patch, HIVE-13068.04.patch, > HIVE-13068.04.patch, HIVE-13068.05.patch, HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13628) Support for permanent functions - error handling if no restart
[
https://issues.apache.org/jira/browse/HIVE-13628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295652#comment-15295652
]
Hive QA commented on HIVE-13628:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12805348/HIVE-13628.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 29 failed/errored test(s), 10034 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-script_pipe.q-vector_decimal_aggregate.q-vector_data_types.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorized_parquet.q-insert_values_non_partitioned.q-schema_evol_orc_nonvec_mapwork_part.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-auto_join9.q-groupby_map_ppr.q-ppd_join3.q-and-12-more - did
not produce a TEST-*.xml file
TestSparkCliDriver-groupby3_map.q-skewjoinopt8.q-union_remove_1.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-smb_mapjoin_4.q-groupby8_map.q-groupby4_map.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-table_access_keys_stats.q-bucketsortoptimize_insert_4.q-runtime_skewjoin_mapjoin_spark.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
org.apache.hive.service.cli.session.TestSessionManagerMetrics.testThreadPoolMetrics
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/359/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/359/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-359/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 29 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12805348 - PreCommit-HIVE-MASTER-Build
> Support for permanent functions - error handling if no restart
> --
>
> Key: HIVE-13628
> URL: https://issues.apache.org/jira/browse/HIVE-13628
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.1.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13628.1.patch, HIVE-13628.2.patch,
> HIVE-13628.3.patch, HIVE-13628.4.patch
>
>
> Support for permanent functions - error handling if no restart
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13306) Better Decimal vectorization
[ https://issues.apache.org/jira/browse/HIVE-13306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-13306: -- Status: Patch Available (was: In Progress) > Better Decimal vectorization > > > Key: HIVE-13306 > URL: https://issues.apache.org/jira/browse/HIVE-13306 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Teddy Choi >Priority: Critical > Attachments: HIVE-13306.1.patch, HIVE-13306.2.patch > > > Decimal Vectorization Requirements > • Today, the LongColumnVector, DoubleColumnVector, BytesColumnVector, > TimestampColumnVector classes store the data as primitive Java data types > long, double, or byte arrays for efficiency. > • DecimalColumnVector is different - it has an array of Object references > to HiveDecimal objects. > • The HiveDecimal object uses an internal object BigDecimal for its > implementation. Further, BigDecimal itself uses an internal object > BigInteger for its implementation, and BigInteger uses an int array. 4 > objects total. > • And, HiveDecimal is an immutable object which means arithmetic and > other operations produce new HiveDecimal object with 3 new objects underneath. > • A major reason Vectorization is fast is the ColumnVector classes except > DecimalColumnVector do not have to allocate additional memory per row. This > avoids memory fragmentation and pressure on the Java Garbage Collector that > DecimalColumnVector can generate. It is very significant. > • What can be done with DecimalColumnVector to make it much more > efficient? > o Design several new decimal classes that allow the caller to manage the > decimal storage. > o If it takes N int values to store a decimal (e.g. N=1..5), then a new > DecimalColumnVector would have an int[] of length N*1024 (where 1024 is the > default column vector size). > o Why store a decimal in separate int values? > • Java does not support 128 bit integers. > • Java does not support unsigned integers. > • In order to do multiplication of a decimal represented in a long you > need twice the storage (i.e. 128 bits). So you need to represent parts in 32 > bit integers. > • But really since we do not have unsigned, really you can only do > multiplications on N-1 bits or 31 bits. > • So, 5 ints are needed for decimal storage... of 38 digits. > o It makes sense to have just one algorithm for decimals rather than one > for HiveDecimal and another for DecimalColumnVector. So, make HiveDecimal > store N int values, too. > o A lower level primitive decimal class would accept decimals stored as > int arrays and produces results into int arrays. It would be used by > HiveDecimal and DecimalColumnVector. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13306) Better Decimal vectorization
[ https://issues.apache.org/jira/browse/HIVE-13306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-13306: -- Attachment: HIVE-13306.2.patch This patch is more improved implementation of new decimal vectorization. I wanted to see whether it passes the integration test. However, it still needs to be integrated with the execution engine. I will keep working on this topic. > Better Decimal vectorization > > > Key: HIVE-13306 > URL: https://issues.apache.org/jira/browse/HIVE-13306 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Teddy Choi >Priority: Critical > Attachments: HIVE-13306.1.patch, HIVE-13306.2.patch > > > Decimal Vectorization Requirements > • Today, the LongColumnVector, DoubleColumnVector, BytesColumnVector, > TimestampColumnVector classes store the data as primitive Java data types > long, double, or byte arrays for efficiency. > • DecimalColumnVector is different - it has an array of Object references > to HiveDecimal objects. > • The HiveDecimal object uses an internal object BigDecimal for its > implementation. Further, BigDecimal itself uses an internal object > BigInteger for its implementation, and BigInteger uses an int array. 4 > objects total. > • And, HiveDecimal is an immutable object which means arithmetic and > other operations produce new HiveDecimal object with 3 new objects underneath. > • A major reason Vectorization is fast is the ColumnVector classes except > DecimalColumnVector do not have to allocate additional memory per row. This > avoids memory fragmentation and pressure on the Java Garbage Collector that > DecimalColumnVector can generate. It is very significant. > • What can be done with DecimalColumnVector to make it much more > efficient? > o Design several new decimal classes that allow the caller to manage the > decimal storage. > o If it takes N int values to store a decimal (e.g. N=1..5), then a new > DecimalColumnVector would have an int[] of length N*1024 (where 1024 is the > default column vector size). > o Why store a decimal in separate int values? > • Java does not support 128 bit integers. > • Java does not support unsigned integers. > • In order to do multiplication of a decimal represented in a long you > need twice the storage (i.e. 128 bits). So you need to represent parts in 32 > bit integers. > • But really since we do not have unsigned, really you can only do > multiplications on N-1 bits or 31 bits. > • So, 5 ints are needed for decimal storage... of 38 digits. > o It makes sense to have just one algorithm for decimals rather than one > for HiveDecimal and another for DecimalColumnVector. So, make HiveDecimal > store N int values, too. > o A lower level primitive decimal class would accept decimals stored as > int arrays and produces results into int arrays. It would be used by > HiveDecimal and DecimalColumnVector. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13787) LLAP: bug in recent security patches (wrong argument order; using full user name in id)
[
https://issues.apache.org/jira/browse/HIVE-13787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295593#comment-15295593
]
Hive QA commented on HIVE-13787:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12805294/HIVE-13787.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 41 failed/errored test(s), 9307 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-groupby2.q-tez_dynpart_hashjoin_1.q-custom_input_output_format.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_distinct_2.q-tez_joins_explain.q-cte_mat_1.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_interval_2.q-schema_evol_text_nonvec_mapwork_part_all_primitive.q-tez_fsstat.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorization_13.q-auto_sortmerge_join_13.q-tez_bmj_schema_evolution.q-and-12-more
- did not produce a TEST-*.xml file
TestNegativeCliDriver-udf_invalid.q-nopart_insert.q-insert_into_with_schema.q-and-734-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-stats13.q-stats2.q-ppd_gby_join.q-and-12-more - did not
produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorParallelism
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tez.TestConverters.testTaskSpecToFragmentSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.hbase.TestHBaseImport.org.apache.hadoop.hive.metastore.hbase.TestHBaseImport
org.apache.hadoop.hive.ql.exec.tez.TestDynamicPartitionPruner.testSingleSourceMultipleFiltersOrdering1
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.org.apache.hive.minikdc.TestJdbcWithDBTokenStore
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/358/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/358/console
Test logs:
[jira] [Updated] (HIVE-13425) Fix partition addition in MSCK REPAIR TABLE command
[
https://issues.apache.org/jira/browse/HIVE-13425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shinichi Yamashita updated HIVE-13425:
--
Attachment: HIVE-13425.5.patch
> Fix partition addition in MSCK REPAIR TABLE command
> ---
>
> Key: HIVE-13425
> URL: https://issues.apache.org/jira/browse/HIVE-13425
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Shinichi Yamashita
>Assignee: Shinichi Yamashita
>Priority: Minor
> Attachments: HIVE-13425.1.patch, HIVE-13425.2.patch,
> HIVE-13425.3.patch, HIVE-13425.4.patch, HIVE-13425.5.patch
>
>
> I create a table in following HiveQL.
> {code}
> hive> create table example (name string) partitioned by (id int);
> {code}
> And, I make some directories about this table in HDFS.
> {code}
> [root@hadoop ~]# hdfs dfs -ls -R /user/hive/warehouse/example
> drwxr-xr-x - root hadoop 0 2016-04-05 22:21
> /user/hive/warehouse/example/id=1
> drwxr-xr-x - root hadoop 0 2016-04-05 22:22
> /user/hive/warehouse/example/id=1/id=2
> -rw-r--r-- 1 root hadoop 8 2016-04-05 22:22
> /user/hive/warehouse/example/id=1/id=2/example.txt
> {code}
> Next I executed MSCK REPAIR TABLE command and added a partition. And this
> result became as follows.
> {code}
> [root@hadoop ~]# hive -e 'msck repair table example'
> OK
> Partitions not in metastore: example:id=1/id=2
> Repair: Added partition to metastore example:id=1/id=2
> Time taken: 1.243 seconds, Fetched: 2 row(s)
> [root@hadoop ~]# hive -e 'show partitions example'
> OK
> id=2
> {code}
> "id=1" should be a partition, but "id=2" was added. I will fix this problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13584) HBaseStorageHandler should support table pre-split
[ https://issues.apache.org/jira/browse/HIVE-13584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Svetozar Ivanov updated HIVE-13584: --- Status: Patch Available (was: Open) > HBaseStorageHandler should support table pre-split > -- > > Key: HIVE-13584 > URL: https://issues.apache.org/jira/browse/HIVE-13584 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 2.0.0 >Reporter: Svetozar Ivanov >Assignee: Svetozar Ivanov > Labels: HBase, PreSplit > Attachments: HIVE-13584-branch-2.0.patch, HIVE-13584.1.patch, > HIVE-13584.2.patch, HIVE-13584.patch > > > Currently is not possible creation of HBase table via HBaseStorageHandler > with specified pre-split strategy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13584) HBaseStorageHandler should support table pre-split
[ https://issues.apache.org/jira/browse/HIVE-13584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Svetozar Ivanov updated HIVE-13584: --- Status: Open (was: Patch Available) > HBaseStorageHandler should support table pre-split > -- > > Key: HIVE-13584 > URL: https://issues.apache.org/jira/browse/HIVE-13584 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 2.0.0 >Reporter: Svetozar Ivanov >Assignee: Svetozar Ivanov > Labels: HBase, PreSplit > Attachments: HIVE-13584-branch-2.0.patch, HIVE-13584.1.patch, > HIVE-13584.2.patch, HIVE-13584.patch > > > Currently is not possible creation of HBase table via HBaseStorageHandler > with specified pre-split strategy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13584) HBaseStorageHandler should support table pre-split
[ https://issues.apache.org/jira/browse/HIVE-13584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Svetozar Ivanov updated HIVE-13584: --- Attachment: HIVE-13584.2.patch > HBaseStorageHandler should support table pre-split > -- > > Key: HIVE-13584 > URL: https://issues.apache.org/jira/browse/HIVE-13584 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 2.0.0 >Reporter: Svetozar Ivanov >Assignee: Svetozar Ivanov > Labels: HBase, PreSplit > Attachments: HIVE-13584-branch-2.0.patch, HIVE-13584.1.patch, > HIVE-13584.2.patch, HIVE-13584.patch > > > Currently is not possible creation of HBase table via HBaseStorageHandler > with specified pre-split strategy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11338) Method getAutocommit() allways returns true
[
https://issues.apache.org/jira/browse/HIVE-11338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295546#comment-15295546
]
Zoltan Haindrich commented on HIVE-11338:
-
i would like to recommend to close this issue as INVALID
> Method getAutocommit() allways returns true
> ---
>
> Key: HIVE-11338
> URL: https://issues.apache.org/jira/browse/HIVE-11338
> Project: Hive
> Issue Type: Bug
> Components: API
>Reporter: cristian cadar
>Priority: Blocker
>
> Method {{getAutocommit()}} always returns {{true}}.
> This means your connection is transacted. Please change it to {{false}},
> since HiveConnection does not support transactions.
> {{org.apache.hive.jdbc.HiveConnection}}
> class {{HiveConnection}}
> {code}
> @Override
> public boolean getAutoCommit() throws SQLException {
> return true;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13566) Auto-gather column stats - phase 1
[
https://issues.apache.org/jira/browse/HIVE-13566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295540#comment-15295540
]
Hive QA commented on HIVE-13566:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12805279/HIVE-13566.02.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 85 failed/errored test(s), 10010 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-cte_4.q-vector_non_string_partition.q-delete_where_non_partitioned.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-load_dyn_part2.q-selectDistinctStar.q-vector_decimal_5.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-script_pipe.q-vector_decimal_aggregate.q-vector_data_types.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-smb_cache.q-transform_ppr2.q-vector_outer_join0.q-and-5-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_grouping_sets.q-update_all_partitioned.q-cte_5.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorization_13.q-auto_sortmerge_join_13.q-tez_bmj_schema_evolution.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-smb_mapjoin_4.q-groupby8_map.q-groupby4_map.q-and-12-more -
did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constantPropagateForSubQuery
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables_compact
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_runtime_skewjoin_mapjoin_spark
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin_noskew
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_skewjoin_onesideskew
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join31
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_transform
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_subquery_exists
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_union30
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_elt
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf
[jira] [Commented] (HIVE-13539) HiveHFileOutputFormat searching the wrong directory for HFiles
[
https://issues.apache.org/jira/browse/HIVE-13539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295483#comment-15295483
]
Tim Robertson commented on HIVE-13539:
--
Thanks [~mmccline].
I will but it won't be for a couple of weeks at least.
> HiveHFileOutputFormat searching the wrong directory for HFiles
> --
>
> Key: HIVE-13539
> URL: https://issues.apache.org/jira/browse/HIVE-13539
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Affects Versions: 1.1.0
> Environment: Built into CDH 5.4.7
>Reporter: Tim Robertson
>Assignee: Tim Robertson
>Priority: Blocker
>
> When creating HFiles for a bulkload in HBase I believe it is looking in the
> wrong directory to find the HFiles, resulting in the following exception:
> {code}
> Error: java.lang.RuntimeException: Hive Runtime Error while closing
> operators: java.io.IOException: Multiple family directories found in
> hdfs://c1n1.gbif.org:8020/user/hive/warehouse/tim.db/coords_hbase/_temporary/2/_temporary
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:295)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:453)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: Multiple family directories found in
> hdfs://c1n1.gbif.org:8020/user/hive/warehouse/tim.db/coords_hbase/_temporary/2/_temporary
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.closeWriters(FileSinkOperator.java:188)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:958)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:598)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:287)
> ... 7 more
> Caused by: java.io.IOException: Multiple family directories found in
> hdfs://c1n1.gbif.org:8020/user/hive/warehouse/tim.db/coords_hbase/_temporary/2/_temporary
> at
> org.apache.hadoop.hive.hbase.HiveHFileOutputFormat$1.close(HiveHFileOutputFormat.java:158)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.closeWriters(FileSinkOperator.java:185)
> ... 11 more
> {code}
> The issue is that is looks for the HFiles in
> {{hdfs://c1n1.gbif.org:8020/user/hive/warehouse/tim.db/coords_hbase/_temporary/2/_temporary}}
> when I believe it should be looking in the task attempt subfolder, such as
> {{hdfs://c1n1.gbif.org:8020/user/hive/warehouse/tim.db/coords_hbase/_temporary/2/_temporary/attempt_1461004169450_0002_r_00_1000}}.
> This can be reproduced in any HFile creation such as:
> {code:sql}
> CREATE TABLE coords_hbase(id INT, x DOUBLE, y DOUBLE)
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
> 'hbase.columns.mapping' = ':key,o:x,o:y',
> 'hbase.table.default.storage.type' = 'binary');
> SET hfile.family.path=/tmp/coords_hfiles/o;
> SET hive.hbase.generatehfiles=true;
> INSERT OVERWRITE TABLE coords_hbase
> SELECT id, decimalLongitude, decimalLatitude
> FROM source
> CLUSTER BY id;
> {code}
> Any advice greatly appreciated
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11233) Include Apache Phoenix support in HBaseStorageHandler
[
https://issues.apache.org/jira/browse/HIVE-11233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Svetozar Ivanov updated HIVE-11233:
---
Status: Patch Available (was: Open)
> Include Apache Phoenix support in HBaseStorageHandler
> -
>
> Key: HIVE-11233
> URL: https://issues.apache.org/jira/browse/HIVE-11233
> Project: Hive
> Issue Type: New Feature
> Components: HBase Handler
>Affects Versions: 2.0.0, 1.2.1
>Reporter: Svetozar Ivanov
>Assignee: Svetozar Ivanov
> Labels: Binary, Hbase, Numeric, Phoenix, Sortable
> Attachments: HIVE-11233-branch-1.2.patch,
> HIVE-11233-branch-2.0.patch, HIVE-11233.1.patch, HIVE-11233.2.patch,
> HIVE-11233.3.patch, HIVE-11233.patch
>
>
> Currently HBaseStorageHandler doesn't provide mechanism for storage of binary
> sortable key and values. It is necessary when given HBase table is used for
> persistence by Apache Hive and Apache Phoenix. In that way all byte arrays
> read or written by Hive will be compatible with binary sortable format used
> in Phoenix.
> It turns out the major difference is in all numeric data types accordingly
> officially provided documentation -
> https://phoenix.apache.org/language/datatypes.html.
> That's how I'm using it in my code:
> {code}
> private static String buildWithSerDeProperties(TableDescriptor
> tableDescriptor) {
> Map serdePropertiesMap = new HashMap<>();
> serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_NAME,
> tableDescriptor.getTableName());
> serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_DEFAULT_STORAGE_TYPE,
> BINARY_STORAGE_TYPE);
> serdePropertiesMap.put(HBaseSerDe.HBASE_COLUMNS_MAPPING,
> buildHBaseColumnsDefinition(tableDescriptor));
> serdePropertiesMap.put(HBaseSerDe.HBASE_VALUE_FACTORY_CLASS,
> PhoenixValueFactory.class.getName());
> /* Use different key factory for simple and composite primary key */
> if (tableDescriptor.getPkDescriptors().size() == 1) {
> serdePropertiesMap.put(HBaseSerDe.HBASE_KEY_FACTORY_CLASS,
> PhoenixKeyFactory.class.getName());
> } else {
> serdePropertiesMap.put(HBaseSerDe.HBASE_COMPOSITE_KEY_FACTORY,
> PhoenixCompositeKeyFactory.class.getName());
> }
> String serDeProperties = serdePropertiesMap.entrySet().stream()
> .map(e -> quoteInSingleQuotes(e.getKey()) + " = " +
> quoteInSingleQuotes(e.getValue()))
> .collect(Collectors.joining(COLUMNS_SEPARATOR));
> logger.debug("SERDEPROPERTIES are [{}]", serDeProperties);
> return serDeProperties;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13817) Allow DNS CNAME ALIAS Resolution from apache hive beeline JDBC URL to allow for failover
[
https://issues.apache.org/jira/browse/HIVE-13817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295473#comment-15295473
]
Vijay Singh edited comment on HIVE-13817 at 5/22/16 8:35 AM:
-
The last combination, where we use Canonicalname and canonicalname resolution
is turned on in connect string has been pasted below for reference as well.
{code}
[root@vjd-1 ~]# beeline -u
"jdbc:hive2://vjtest-local.gce.cloudera.com:1/default;principal=hive/_h...@gce.cloudera.com;useCanonicalName=true"
-e "show tables;"
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated
and will likely be removed in a future release
16/05/22 01:33:24 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree
module jar containing PrefixTreeCodec is not present. Continuing without it.
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
scan complete in 2ms
Connecting to
jdbc:hive2://vjtest-local.gce.cloudera.com:1/default;principal=hive/_h...@gce.cloudera.com;useCanonicalName=true
Connected to: Apache Hive (version 1.1.0-cdh5.7.0)
Driver: Hive JDBC (version 1.1.0-cdh5.7.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
INFO : Compiling
command(queryId=hive_2016052201_cd09f83a-5ce1-4cf9-8a3b-9175b00ed4bb): show
tables
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name,
type:string, comment:from deserializer)], properties:null)
INFO : Completed compiling
command(queryId=hive_2016052201_cd09f83a-5ce1-4cf9-8a3b-9175b00ed4bb); Time
taken: 0.07 seconds
INFO : Executing
command(queryId=hive_2016052201_cd09f83a-5ce1-4cf9-8a3b-9175b00ed4bb): show
tables
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing
command(queryId=hive_2016052201_cd09f83a-5ce1-4cf9-8a3b-9175b00ed4bb); Time
taken: 0.135 seconds
INFO : OK
++--+
| tab_name |
++--+
| customers |
| sample_07 |
| sample_08 |
| web_logs |
++--+
4 rows selected (0.357 seconds)
Beeline version 1.1.0-cdh5.7.0 by Apache Hive
Closing: 0:
jdbc:hive2://vjtest-local.gce.cloudera.com:1/default;principal=hive/_h...@gce.cloudera.com;useCanonicalName=true
[root@vjd-1 ~]#
{code}
was (Author: singhvjd):
The last combination, where we use Canonicalname and canonicalname resolution
is turned on in connect string has been pasted below for reference as well.
{code}
Closing: 0:
jdbc:hive2://vjtest-local.gce.cloudera.com:1/default;principal=hive/_h...@gce.cloudera.com
[root@vjd-1 ~]# beeline -u
"jdbc:hive2://vjtest-local.gce.cloudera.com:1/default;principal=hive/_h...@gce.cloudera.com;useCanonicalName=true"
-e "show tables;"
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated
and will likely be removed in a future release
16/05/22 01:33:24 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree
module jar containing PrefixTreeCodec is not present. Continuing without it.
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
scan complete in 2ms
Connecting to
jdbc:hive2://vjtest-local.gce.cloudera.com:1/default;principal=hive/_h...@gce.cloudera.com;useCanonicalName=true
Connected to: Apache Hive (version 1.1.0-cdh5.7.0)
Driver: Hive JDBC (version 1.1.0-cdh5.7.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
INFO : Compiling
command(queryId=hive_2016052201_cd09f83a-5ce1-4cf9-8a3b-9175b00ed4bb): show
tables
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name,
type:string, comment:from deserializer)], properties:null)
INFO : Completed compiling
command(queryId=hive_2016052201_cd09f83a-5ce1-4cf9-8a3b-9175b00ed4bb); Time
taken: 0.07 seconds
INFO : Executing
command(queryId=hive_2016052201_cd09f83a-5ce1-4cf9-8a3b-9175b00ed4bb): show
tables
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing
command(queryId=hive_2016052201_cd09f83a-5ce1-4cf9-8a3b-9175b00ed4bb); Time
taken: 0.135 seconds
INFO : OK
++--+
| tab_name |
++--+
| customers |
| sample_07 |
| sample_08 |
| web_logs |
++--+
4 rows selected (0.357 seconds)
Beeline version 1.1.0-cdh5.7.0 by Apache Hive
Closing: 0:
jdbc:hive2://vjtest-local.gce.cloudera.com:1/default;principal=hive/_h...@gce.cloudera.com;useCanonicalName=true
[root@vjd-1 ~]#
{code}
> Allow DNS CNAME ALIAS Resolution from apache hive beeline JDBC URL to allow
> for failover
>
>
>
[jira] [Updated] (HIVE-11233) Include Apache Phoenix support in HBaseStorageHandler
[
https://issues.apache.org/jira/browse/HIVE-11233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Svetozar Ivanov updated HIVE-11233:
---
Attachment: HIVE-11233.3.patch
> Include Apache Phoenix support in HBaseStorageHandler
> -
>
> Key: HIVE-11233
> URL: https://issues.apache.org/jira/browse/HIVE-11233
> Project: Hive
> Issue Type: New Feature
> Components: HBase Handler
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Svetozar Ivanov
>Assignee: Svetozar Ivanov
> Labels: Binary, Hbase, Numeric, Phoenix, Sortable
> Attachments: HIVE-11233-branch-1.2.patch,
> HIVE-11233-branch-2.0.patch, HIVE-11233.1.patch, HIVE-11233.2.patch,
> HIVE-11233.3.patch, HIVE-11233.patch
>
>
> Currently HBaseStorageHandler doesn't provide mechanism for storage of binary
> sortable key and values. It is necessary when given HBase table is used for
> persistence by Apache Hive and Apache Phoenix. In that way all byte arrays
> read or written by Hive will be compatible with binary sortable format used
> in Phoenix.
> It turns out the major difference is in all numeric data types accordingly
> officially provided documentation -
> https://phoenix.apache.org/language/datatypes.html.
> That's how I'm using it in my code:
> {code}
> private static String buildWithSerDeProperties(TableDescriptor
> tableDescriptor) {
> Map serdePropertiesMap = new HashMap<>();
> serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_NAME,
> tableDescriptor.getTableName());
> serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_DEFAULT_STORAGE_TYPE,
> BINARY_STORAGE_TYPE);
> serdePropertiesMap.put(HBaseSerDe.HBASE_COLUMNS_MAPPING,
> buildHBaseColumnsDefinition(tableDescriptor));
> serdePropertiesMap.put(HBaseSerDe.HBASE_VALUE_FACTORY_CLASS,
> PhoenixValueFactory.class.getName());
> /* Use different key factory for simple and composite primary key */
> if (tableDescriptor.getPkDescriptors().size() == 1) {
> serdePropertiesMap.put(HBaseSerDe.HBASE_KEY_FACTORY_CLASS,
> PhoenixKeyFactory.class.getName());
> } else {
> serdePropertiesMap.put(HBaseSerDe.HBASE_COMPOSITE_KEY_FACTORY,
> PhoenixCompositeKeyFactory.class.getName());
> }
> String serDeProperties = serdePropertiesMap.entrySet().stream()
> .map(e -> quoteInSingleQuotes(e.getKey()) + " = " +
> quoteInSingleQuotes(e.getValue()))
> .collect(Collectors.joining(COLUMNS_SEPARATOR));
> logger.debug("SERDEPROPERTIES are [{}]", serDeProperties);
> return serDeProperties;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11233) Include Apache Phoenix support in HBaseStorageHandler
[
https://issues.apache.org/jira/browse/HIVE-11233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Svetozar Ivanov updated HIVE-11233:
---
Status: Open (was: Patch Available)
> Include Apache Phoenix support in HBaseStorageHandler
> -
>
> Key: HIVE-11233
> URL: https://issues.apache.org/jira/browse/HIVE-11233
> Project: Hive
> Issue Type: New Feature
> Components: HBase Handler
>Affects Versions: 2.0.0, 1.2.1
>Reporter: Svetozar Ivanov
>Assignee: Svetozar Ivanov
> Labels: Binary, Hbase, Numeric, Phoenix, Sortable
> Attachments: HIVE-11233-branch-1.2.patch,
> HIVE-11233-branch-2.0.patch, HIVE-11233.1.patch, HIVE-11233.2.patch,
> HIVE-11233.3.patch, HIVE-11233.patch
>
>
> Currently HBaseStorageHandler doesn't provide mechanism for storage of binary
> sortable key and values. It is necessary when given HBase table is used for
> persistence by Apache Hive and Apache Phoenix. In that way all byte arrays
> read or written by Hive will be compatible with binary sortable format used
> in Phoenix.
> It turns out the major difference is in all numeric data types accordingly
> officially provided documentation -
> https://phoenix.apache.org/language/datatypes.html.
> That's how I'm using it in my code:
> {code}
> private static String buildWithSerDeProperties(TableDescriptor
> tableDescriptor) {
> Map serdePropertiesMap = new HashMap<>();
> serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_NAME,
> tableDescriptor.getTableName());
> serdePropertiesMap.put(HBaseSerDe.HBASE_TABLE_DEFAULT_STORAGE_TYPE,
> BINARY_STORAGE_TYPE);
> serdePropertiesMap.put(HBaseSerDe.HBASE_COLUMNS_MAPPING,
> buildHBaseColumnsDefinition(tableDescriptor));
> serdePropertiesMap.put(HBaseSerDe.HBASE_VALUE_FACTORY_CLASS,
> PhoenixValueFactory.class.getName());
> /* Use different key factory for simple and composite primary key */
> if (tableDescriptor.getPkDescriptors().size() == 1) {
> serdePropertiesMap.put(HBaseSerDe.HBASE_KEY_FACTORY_CLASS,
> PhoenixKeyFactory.class.getName());
> } else {
> serdePropertiesMap.put(HBaseSerDe.HBASE_COMPOSITE_KEY_FACTORY,
> PhoenixCompositeKeyFactory.class.getName());
> }
> String serDeProperties = serdePropertiesMap.entrySet().stream()
> .map(e -> quoteInSingleQuotes(e.getKey()) + " = " +
> quoteInSingleQuotes(e.getValue()))
> .collect(Collectors.joining(COLUMNS_SEPARATOR));
> logger.debug("SERDEPROPERTIES are [{}]", serDeProperties);
> return serDeProperties;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13817) Allow DNS CNAME ALIAS Resolution from apache hive beeline JDBC URL to allow for failover
[
https://issues.apache.org/jira/browse/HIVE-13817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295473#comment-15295473
]
Vijay Singh commented on HIVE-13817:
The last combination, where we use Canonicalname and canonicalname resolution
is turned on in connect string has been pasted below for reference as well.
{code}
Closing: 0:
jdbc:hive2://vjtest-local.gce.cloudera.com:1/default;principal=hive/_h...@gce.cloudera.com
[root@vjd-1 ~]# beeline -u
"jdbc:hive2://vjtest-local.gce.cloudera.com:1/default;principal=hive/_h...@gce.cloudera.com;useCanonicalName=true"
-e "show tables;"
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated
and will likely be removed in a future release
16/05/22 01:33:24 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree
module jar containing PrefixTreeCodec is not present. Continuing without it.
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
scan complete in 2ms
Connecting to
jdbc:hive2://vjtest-local.gce.cloudera.com:1/default;principal=hive/_h...@gce.cloudera.com;useCanonicalName=true
Connected to: Apache Hive (version 1.1.0-cdh5.7.0)
Driver: Hive JDBC (version 1.1.0-cdh5.7.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
INFO : Compiling
command(queryId=hive_2016052201_cd09f83a-5ce1-4cf9-8a3b-9175b00ed4bb): show
tables
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name,
type:string, comment:from deserializer)], properties:null)
INFO : Completed compiling
command(queryId=hive_2016052201_cd09f83a-5ce1-4cf9-8a3b-9175b00ed4bb); Time
taken: 0.07 seconds
INFO : Executing
command(queryId=hive_2016052201_cd09f83a-5ce1-4cf9-8a3b-9175b00ed4bb): show
tables
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing
command(queryId=hive_2016052201_cd09f83a-5ce1-4cf9-8a3b-9175b00ed4bb); Time
taken: 0.135 seconds
INFO : OK
++--+
| tab_name |
++--+
| customers |
| sample_07 |
| sample_08 |
| web_logs |
++--+
4 rows selected (0.357 seconds)
Beeline version 1.1.0-cdh5.7.0 by Apache Hive
Closing: 0:
jdbc:hive2://vjtest-local.gce.cloudera.com:1/default;principal=hive/_h...@gce.cloudera.com;useCanonicalName=true
[root@vjd-1 ~]#
{code}
> Allow DNS CNAME ALIAS Resolution from apache hive beeline JDBC URL to allow
> for failover
>
>
> Key: HIVE-13817
> URL: https://issues.apache.org/jira/browse/HIVE-13817
> Project: Hive
> Issue Type: New Feature
> Components: Beeline
>Affects Versions: 1.2.1
>Reporter: Vijay Singh
> Attachments: HIVE-13817.1.patch, HIVE-13817.2.patch,
> HIVE-13817.3.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Currently, in case of BDR clusters, DNS CNAME alias based connections fail.
> As _HOST resolves to exact endpoint specified in connection string and that
> may not be intended SPN for kerberos based on reverse DNS lookup.
> Consequently this JIRA proposes that client specific setting be used to
> resolv _HOST from CNAME DNS alias to A record entry on the fly in beeline.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13817) Allow DNS CNAME ALIAS Resolution from apache hive beeline JDBC URL to allow for failover
[
https://issues.apache.org/jira/browse/HIVE-13817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295470#comment-15295470
]
Vijay Singh commented on HIVE-13817:
Please see the following SPN values available to HiveServer2.
{code}
[root@vjd-1 ~]# klist -kt
/var/run/cloudera-scm-agent/process/2121-hive-HIVESERVER2/hive.keytab
Keytab name:
FILE:/var/run/cloudera-scm-agent/process/2121-hive-HIVESERVER2/hive.keytab
KVNO Timestamp Principal
--- --
1 05/22/2016 01:07:19 hive/vjtest-local.gce.cloudera@gce.cloudera.com
1 05/22/2016 01:07:19 hive/vjtest-local.gce.cloudera@gce.cloudera.com
1 05/22/2016 01:07:19 hive/vjtest-local.gce.cloudera@gce.cloudera.com
1 05/22/2016 01:07:19 hive/vjd-1.gce.cloudera@gce.cloudera.com
1 05/22/2016 01:07:19 hive/vjd-1.gce.cloudera@gce.cloudera.com
1 05/22/2016 01:07:19 hive/vjd-1.gce.cloudera@gce.cloudera.com
1 05/22/2016 01:07:19 HTTP/vjd-1.gce.cloudera@gce.cloudera.com
1 05/22/2016 01:07:19 HTTP/vjd-1.gce.cloudera@gce.cloudera.com
1 05/22/2016 01:07:19 HTTP/vjd-1.gce.cloudera@gce.cloudera.com
{code}
Please see the following program that determines canonical name.
{code}
[root@vjd-1 ~]# java NSLookupFwd vjlb
Host: vjlb
Canonical HostName: vjtest-local.gce.cloudera.com
{code}
Please find the log excerpt from executing beeline with new driver with cname
alias functionality turned on in connection string.
{code}
[root@vjd-1 ~]# hadoop version
Hadoop 2.6.0-cdh5.7.0
Subversion http://github.com/cloudera/hadoop -r
c00978c67b0d3fe9f3b896b5030741bd40bf541a
Compiled by jenkins on 2016-03-23T18:36Z
Compiled with protoc 2.5.0
>From source with checksum b2eabfa328e763c88cb14168f9b372
This command was run using
/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/jars/hadoop-common-2.6.0-cdh5.7.0.jar
[root@vjd-1 ~]# beeline -u
"jdbc:hive2://vjlb:1/default;principal=hive/_h...@gce.cloudera.com;useCanonicalName=true"
-e "show tables;"
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated
and will likely be removed in a future release
16/05/22 01:14:45 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree
module jar containing PrefixTreeCodec is not present. Continuing without it.
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
scan complete in 2ms
Connecting to
jdbc:hive2://vjlb:1/default;principal=hive/_h...@gce.cloudera.com;useCanonicalName=true
Connected to: Apache Hive (version 1.1.0-cdh5.7.0)
Driver: Hive JDBC (version 1.1.0-cdh5.7.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
INFO : Compiling
command(queryId=hive_20160522011414_690deeda-df58-469d-b18f-33890162cf6f): show
tables
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name,
type:string, comment:from deserializer)], properties:null)
INFO : Completed compiling
command(queryId=hive_20160522011414_690deeda-df58-469d-b18f-33890162cf6f); Time
taken: 0.069 seconds
INFO : Executing
command(queryId=hive_20160522011414_690deeda-df58-469d-b18f-33890162cf6f): show
tables
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing
command(queryId=hive_20160522011414_690deeda-df58-469d-b18f-33890162cf6f); Time
taken: 0.163 seconds
INFO : OK
++--+
| tab_name |
++--+
| customers |
| sample_07 |
| sample_08 |
| web_logs |
++--+
4 rows selected (0.376 seconds)
Beeline version 1.1.0-cdh5.7.0 by Apache Hive
Closing: 0:
jdbc:hive2://vjlb:1/default;principal=hive/_h...@gce.cloudera.com;useCanonicalName=true
[root@vjd-1 ~]#
{code}
Please find the log excerpt for connection without canonical name functionality
turned on in connection string.
{code}
[root@vjd-1 ~]# beeline -u
"jdbc:hive2://vjlb:1/default;principal=hive/_h...@gce.cloudera.com" -e
"show tables;"
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated
and will likely be removed in a future release
16/05/22 01:20:41 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree
module jar containing PrefixTreeCodec is not present. Continuing without it.
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
support was removed in 8.0
scan complete in 3ms
Connecting to
jdbc:hive2://vjlb:1/default;principal=hive/_h...@gce.cloudera.com
16/05/22 01:20:43 [main]: ERROR transport.TSaslTransport: SASL negotiation
failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Server not found in Kerberos
[jira] [Commented] (HIVE-9660) store end offset of compressed data for RG in RowIndex in ORC
[
https://issues.apache.org/jira/browse/HIVE-9660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295469#comment-15295469
]
Hive QA commented on HIVE-9660:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12805265/HIVE-9660.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 119 failed/errored test(s), 9855 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-constprog_dpp.q-dynamic_partition_pruning.q-vectorization_10.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-cte_4.q-vector_non_string_partition.q-delete_where_non_partitioned.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-dynpart_sort_optimization2.q-tez_dynpart_hashjoin_3.q-orc_vectorization_ppd.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-explainuser_4.q-update_after_multiple_inserts.q-mapreduce2.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-mapjoin_mapjoin.q-insert_into1.q-vector_decimal_2.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-order_null.q-vector_acid3.q-orc_merge10.q-and-12-more -
did not produce a TEST-*.xml file
TestMiniTezCliDriver-smb_cache.q-transform_ppr2.q-vector_outer_join0.q-and-5-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_coalesce.q-cbo_windowing.q-tez_join.q-and-12-more -
did not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorization_16.q-vector_decimal_round.q-orc_merge6.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-auto_join30.q-join2.q-input17.q-and-12-more - did not
produce a TEST-*.xml file
TestSparkCliDriver-avro_joins.q-join36.q-join1.q-and-12-more - did not produce
a TEST-*.xml file
TestSparkCliDriver-groupby2.q-custom_input_output_format.q-join41.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-groupby3_map.q-skewjoinopt8.q-union_remove_1.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-load_dyn_part5.q-load_dyn_part2.q-skewjoinopt16.q-and-12-more
- did not produce a TEST-*.xml file
TestSparkCliDriver-order.q-auto_join18_multi_distinct.q-union2.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-skewjoin_noskew.q-sample2.q-skewjoinopt10.q-and-12-more -
did not produce a TEST-*.xml file
TestSparkCliDriver-vector_distinct_2.q-join15.q-load_dyn_part3.q-and-12-more -
did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_orc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_stats_orc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnStatsUpdateForStatsOptimizer_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_deleteAnalyze
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_full
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_partial
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_partial_ndv
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_orig_table_use_metadata
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_uncompressed
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_analyze
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_file_dump
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_merge12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_predicate_pushdown
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_fetchwork_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_smb_mapjoin_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_fast_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_ptf
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
[jira] [Updated] (HIVE-13817) Allow DNS CNAME ALIAS Resolution from apache hive beeline JDBC URL to allow for failover
[ https://issues.apache.org/jira/browse/HIVE-13817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay Singh updated HIVE-13817: --- Attachment: HIVE-13817.3.patch Modified for NullPointerException for 3rd patch since SessionMap is required for Determining Canonical PAram. > Allow DNS CNAME ALIAS Resolution from apache hive beeline JDBC URL to allow > for failover > > > Key: HIVE-13817 > URL: https://issues.apache.org/jira/browse/HIVE-13817 > Project: Hive > Issue Type: New Feature > Components: Beeline >Affects Versions: 1.2.1 >Reporter: Vijay Singh > Attachments: HIVE-13817.1.patch, HIVE-13817.2.patch, > HIVE-13817.3.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Currently, in case of BDR clusters, DNS CNAME alias based connections fail. > As _HOST resolves to exact endpoint specified in connection string and that > may not be intended SPN for kerberos based on reverse DNS lookup. > Consequently this JIRA proposes that client specific setting be used to > resolv _HOST from CNAME DNS alias to A record entry on the fly in beeline. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13817) Allow DNS CNAME ALIAS Resolution from apache hive beeline JDBC URL to allow for failover
[ https://issues.apache.org/jira/browse/HIVE-13817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay Singh updated HIVE-13817: --- Attachment: HIVE-13817.2.patch Made changes to incorrect if else logic for CanonicalName property. > Allow DNS CNAME ALIAS Resolution from apache hive beeline JDBC URL to allow > for failover > > > Key: HIVE-13817 > URL: https://issues.apache.org/jira/browse/HIVE-13817 > Project: Hive > Issue Type: New Feature > Components: Beeline >Affects Versions: 1.2.1 >Reporter: Vijay Singh > Attachments: HIVE-13817.1.patch, HIVE-13817.2.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Currently, in case of BDR clusters, DNS CNAME alias based connections fail. > As _HOST resolves to exact endpoint specified in connection string and that > may not be intended SPN for kerberos based on reverse DNS lookup. > Consequently this JIRA proposes that client specific setting be used to > resolv _HOST from CNAME DNS alias to A record entry on the fly in beeline. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13817) Allow DNS CNAME ALIAS Resolution from apache hive beeline JDBC URL to allow for failover
[
https://issues.apache.org/jira/browse/HIVE-13817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295442#comment-15295442
]
Vijay Singh commented on HIVE-13817:
{panel:title=Test Setup for CNAME alias resolution to A record based on reverse
dns lookup
|borderStyle=dashed|borderColor=#ccd|titleBGColor=#F0D6C1|bgColor=#BBAFCE}
[ec2-user@ip-10-0-0-18 ~]$ nslookup testcnamerecord.gce.cloudera.com
Server: 10.0.0.11
Address:10.0.0.11#53
testcnamerecord.gce.cloudera.comcanonical name =
testarecord.gce.cloudera.com.
Name: testarecord.gce.cloudera.com
Address: 10.0.0.19
[ec2-user@ip-10-0-0-18 ~]$ java NSLookupFwd testcnamerecord.gce.cloudera.com
Host: testcnamerecord.gce.cloudera.com
Canonical HostName: testarecord.gce.cloudera.com
{panel}
Please find the code snippet for above app that successfully performs CNAME
alias to A record resolution based on reverse dns lookup.
{code:title= NSLookupFwd.java|borderStyle=solid}
[ec2-user@ip-10-0-0-18 ~]$ cat NSLookupFwd.java
import java.net.InetAddress;
import java.net.UnknownHostException;
import java.util.logging.Level;
import java.util.logging.Logger;
public class NSLookupFwd
{
public String canonicalize(String host) {
String cname=null;
try {
cname = InetAddress.getByName(host).getCanonicalHostName();
System.out.println("Host: " +
host);
System.out.println("Canonical HostName: " +
cname);
} catch (UnknownHostException ex) {
Logger.getLogger(NSLookupFwd.class.getName()).log(Level.SEVERE, null, ex);
}
return cname;
}
public static void main(String[] args)
{
new NSLookupFwd().canonicalize(args[0]);
}
}
{code}
> Allow DNS CNAME ALIAS Resolution from apache hive beeline JDBC URL to allow
> for failover
>
>
> Key: HIVE-13817
> URL: https://issues.apache.org/jira/browse/HIVE-13817
> Project: Hive
> Issue Type: New Feature
> Components: Beeline
>Affects Versions: 1.2.1
>Reporter: Vijay Singh
> Attachments: HIVE-13817.1.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Currently, in case of BDR clusters, DNS CNAME alias based connections fail.
> As _HOST resolves to exact endpoint specified in connection string and that
> may not be intended SPN for kerberos based on reverse DNS lookup.
> Consequently this JIRA proposes that client specific setting be used to
> resolv _HOST from CNAME DNS alias to A record entry on the fly in beeline.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13029) NVDIMM support for LLAP Cache
[ https://issues.apache.org/jira/browse/HIVE-13029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13029: --- Status: Patch Available (was: Open) > NVDIMM support for LLAP Cache > - > > Key: HIVE-13029 > URL: https://issues.apache.org/jira/browse/HIVE-13029 > Project: Hive > Issue Type: New Feature > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Critical > Attachments: HIVE-13029.1.patch, HIVE-13029.2.patch, > HIVE-13029.3.patch, HIVE-13029.4.patch > > > LLAP cache has been designed so that the cache can be offloaded easily to a > pmem API without restart coherence. > The tricky part about NVDIMMs are restart coherence, while most of the cache > gains can be obtained without keeping state across refreshes, since LLAP is > not the system of record, HDFS is. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13029) NVDIMM support for LLAP Cache
[ https://issues.apache.org/jira/browse/HIVE-13029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13029: --- Attachment: HIVE-13029.4.patch Add a parameterized test-case in TestBuddyAllocator > NVDIMM support for LLAP Cache > - > > Key: HIVE-13029 > URL: https://issues.apache.org/jira/browse/HIVE-13029 > Project: Hive > Issue Type: New Feature > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Critical > Attachments: HIVE-13029.1.patch, HIVE-13029.2.patch, > HIVE-13029.3.patch, HIVE-13029.4.patch > > > LLAP cache has been designed so that the cache can be offloaded easily to a > pmem API without restart coherence. > The tricky part about NVDIMMs are restart coherence, while most of the cache > gains can be obtained without keeping state across refreshes, since LLAP is > not the system of record, HDFS is. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13029) NVDIMM support for LLAP Cache
[ https://issues.apache.org/jira/browse/HIVE-13029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-13029: --- Status: Open (was: Patch Available) > NVDIMM support for LLAP Cache > - > > Key: HIVE-13029 > URL: https://issues.apache.org/jira/browse/HIVE-13029 > Project: Hive > Issue Type: New Feature > Components: llap >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Gopal V >Priority: Critical > Attachments: HIVE-13029.1.patch, HIVE-13029.2.patch, > HIVE-13029.3.patch, HIVE-13029.4.patch > > > LLAP cache has been designed so that the cache can be offloaded easily to a > pmem API without restart coherence. > The tricky part about NVDIMMs are restart coherence, while most of the cache > gains can be obtained without keeping state across refreshes, since LLAP is > not the system of record, HDFS is. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13029) NVDIMM support for LLAP Cache
[
https://issues.apache.org/jira/browse/HIVE-13029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15295431#comment-15295431
]
Gopal V commented on HIVE-13029:
Failed tests have
{code}
Exception: MetaException(message:Got exception:
org.apache.hadoop.hive.metastore.api.MetaException Unable to get databases,
Failed after attempts=36, exceptions:
Fri May 20 07:26:23 PDT 2016, null, java.net.SocketTimeoutException:
callTimeout=6, callDuration=68259: row 'HBMS_DBS,,99' on table
'hbase:meta' at region=hbase:meta,,1.1588230740,
hostname=ip-10-233-133-28.us-west-1.compute.internal,59291,1463749169723,
seqNum=0
)
org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got
exception: org.apache.hadoop.hive.metastore.api.MetaException Unable to get
databases, Failed after attempts=36, exceptions:
Fri May 20 07:26:23 PDT 2016, null, java.net.SocketTimeoutException:
callTimeout=6, callDuration=68259: row 'HBMS_DBS,,99' on table
'hbase:meta' at region=hbase:meta,,1.1588230740,
hostname=ip-10-233-133-28.us-west-1.compute.internal,59291,1463749169723,
seqNum=0
)
at
org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1373)
{code}
which looks like a metastore startup issue.
> NVDIMM support for LLAP Cache
> -
>
> Key: HIVE-13029
> URL: https://issues.apache.org/jira/browse/HIVE-13029
> Project: Hive
> Issue Type: New Feature
> Components: llap
>Affects Versions: 2.1.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Critical
> Attachments: HIVE-13029.1.patch, HIVE-13029.2.patch,
> HIVE-13029.3.patch
>
>
> LLAP cache has been designed so that the cache can be offloaded easily to a
> pmem API without restart coherence.
> The tricky part about NVDIMMs are restart coherence, while most of the cache
> gains can be obtained without keeping state across refreshes, since LLAP is
> not the system of record, HDFS is.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
