[jira] [Commented] (HIVE-20807) Refactor LlapStatusServiceDriver
[
https://issues.apache.org/jira/browse/HIVE-20807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671138#comment-16671138
]
Hive QA commented on HIVE-20807:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} llap-server in master has 83 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} llap-server: The patch generated 1 new + 4 unchanged -
98 fixed = 5 total (was 102) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} llap-server generated 0 new + 81 unchanged - 2 fixed
= 81 total (was 83) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} service in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14693/dev-support/hive-personality.sh
|
| git revision | master / 08349b1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14693/yetus/diff-checkstyle-llap-server.txt
|
| modules | C: . llap-server service U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14693/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Refactor LlapStatusServiceDriver
>
>
> Key: HIVE-20807
> URL: https://issues.apache.org/jira/browse/HIVE-20807
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20807.01.patch, HIVE-20807.02.patch,
> HIVE-20807.03.patch
>
>
> LlapStatusServiceDriver is the class used to determine if LLAP has started.
> The following problems should be solved by refactoring:
> 1. The main class is more than 800 lines long,should be cut into multiple
> smaller classes.
> 2. The current design makes it extremely hard to write unit tests.
> 3. There are some overcomplicated,
[jira] [Commented] (HIVE-20823) Make Compactor run in a transaction
[
https://issues.apache.org/jira/browse/HIVE-20823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671063#comment-16671063
]
Hive QA commented on HIVE-20823:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12946143/HIVE-20823.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 654 failed/errored test(s), 15490 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=180)
[tez_union_group_by.q,vector_like_2.q,llap_acid.q,sqlmerge.q,vector_udf_character_length.q,tez_dynpart_hashjoin_1.q,runtime_stats_merge.q,vector_groupby_grouping_sets1.q,vector_binary_join_groupby.q,runtime_stats_hs2.q,lateral_view.q,optimize_nullscan.q,nonmr_fetch_threshold.q,vectorization_decimal_date.q,schema_evol_orc_nonvec_table_llap_io.q,schema_evol_text_vecrow_part_all_complex.q,tez_fsstat.q,vector_fullouter_mapjoin_1_optimized_passthru.q,stats11.q,tez_join_tests.q,join_acid_non_acid.q,empty_join.q,vector_groupby_grouping_window.q,auto_join21.q,schema_evol_text_vecrow_part.q,vector_groupby_sort_11.q,results_cache_2.q,schema_evol_orc_nonvec_part_all_complex_llap_io.q,orc_ppd_timestamp.q,vector_decimal_1.q]
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_buckets]
(batchId=275)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_format_nonpart]
(batchId=275)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_if_with_path_filter]
(batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[char_serde] (batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[convert_decimal64_to_decimal]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[date_serde] (batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_join2]
(batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynamic_partition_insert]
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[implicit_cast_during_insert]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_authorization_sqlstd]
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_avro_partition_uniontype]
(batchId=50)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_createas1]
(batchId=94)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_dictionary_threshold]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_diff_part_cols2]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_diff_part_cols]
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_empty_strings]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_ends_with_nulls]
(batchId=82)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_file_dump]
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_int_type_promotion]
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge10] (batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge11] (batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge12] (batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge13] (batchId=92)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge1] (batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge3] (batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge4] (batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge5] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge6] (batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge_diff_fs]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge_incompat1]
(batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge_incompat2]
(batchId=90)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_ppd_boolean]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_ppd_char]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_ppd_is_null]
(batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_ppd_same_table_multiple_aliases]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_ppd_schema_evol_1a]
(batchId=45)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_ppd_schema_evol_1b]
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_ppd_schema_evol_2a]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_ppd_schema_evol_2b]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_remove_cols]
(batchId=7)
[jira] [Commented] (HIVE-20823) Make Compactor run in a transaction
[
https://issues.apache.org/jira/browse/HIVE-20823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671042#comment-16671042
]
Hive QA commented on HIVE-20823:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
54s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 2 new + 633 unchanged - 0
fixed = 635 total (was 633) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14692/dev-support/hive-personality.sh
|
| git revision | master / 08349b1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14692/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14692/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Make Compactor run in a transaction
> ---
>
> Key: HIVE-20823
> URL: https://issues.apache.org/jira/browse/HIVE-20823
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-20823.01.patch
>
>
> Have compactor open a transaction and run the job in that transaction.
> # make compactor produced base/delta include this txn id in the folder name,
> e.g. base_7_c17 where 17 is the txnid.
> # add {{CQ_TXN_ID bigint}} to COMPACTION_QUEUE and COMPLETED_COMPACTIONS to
> record this txn id
> # make sure {{AcidUtils.getAcidState()}} pays attention to this transaction
> on read and ignores this dir if this txn id is not committed in the current
> snapshot
> ## this means not only validWriteIdList but ValidTxnIdList should be passed
> along in config (if it isn't yet)
> # once this is done, {{CompactorMR.createCompactorMarker()}} can be
> eliminated and {{AcidUtils.isValidBase}} modified accordingly
> # modify Cleaner so that it doesn't clean old files until new file is visible
> to all readers
> #
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20833) package.jdo needs to be updated to conform with HIVE-20221 changes
[
https://issues.apache.org/jira/browse/HIVE-20833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20833:
---
Attachment: HIVE-20833.3.patch
> package.jdo needs to be updated to conform with HIVE-20221 changes
> --
>
> Key: HIVE-20833
> URL: https://issues.apache.org/jira/browse/HIVE-20833
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20833.1.patch, HIVE-20833.2.patch,
> HIVE-20833.3.patch
>
>
> Following test if run with TestMiniLlapLocalCliDriver will fail:
> {code:sql}
> CREATE TABLE `alterPartTbl`(
>`po_header_id` bigint,
>`vendor_num` string,
>`requester_name` string,
>`approver_name` string,
>`buyer_name` string,
>`preparer_name` string,
>`po_requisition_number` string,
>`po_requisition_id` bigint,
>`po_requisition_desc` string,
>`rate_type` string,
>`rate_date` date,
>`rate` double,
>`blanket_total_amount` double,
>`authorization_status` string,
>`revision_num` bigint,
>`revised_date` date,
>`approved_flag` string,
>`approved_date` timestamp,
>`amount_limit` double,
>`note_to_authorizer` string,
>`note_to_vendor` string,
>`note_to_receiver` string,
>`vendor_order_num` string,
>`comments` string,
>`acceptance_required_flag` string,
>`acceptance_due_date` date,
>`closed_date` timestamp,
>`user_hold_flag` string,
>`approval_required_flag` string,
>`cancel_flag` string,
>`firm_status_lookup_code` string,
>`firm_date` date,
>`frozen_flag` string,
>`closed_code` string,
>`org_id` bigint,
>`reference_num` string,
>`wf_item_type` string,
>`wf_item_key` string,
>`submit_date` date,
>`sap_company_code` string,
>`sap_fiscal_year` bigint,
>`po_number` string,
>`sap_line_item` bigint,
>`closed_status_flag` string,
>`balancing_segment` string,
>`cost_center_segment` string,
>`base_amount_limit` double,
>`base_blanket_total_amount` double,
>`base_open_amount` double,
>`base_ordered_amount` double,
>`cancel_date` timestamp,
>`cbc_accounting_date` date,
>`change_requested_by` string,
>`change_summary` string,
>`confirming_order_flag` string,
>`document_creation_method` string,
>`edi_processed_flag` string,
>`edi_processed_status` string,
>`enabled_flag` string,
>`encumbrance_required_flag` string,
>`end_date` date,
>`end_date_active` date,
>`from_header_id` bigint,
>`from_type_lookup_code` string,
>`global_agreement_flag` string,
>`government_context` string,
>`interface_source_code` string,
>`ledger_currency_code` string,
>`open_amount` double,
>`ordered_amount` double,
>`pay_on_code` string,
>`payment_term_name` string,
>`pending_signature_flag` string,
>`po_revision_num` double,
>`preparer_id` bigint,
>`price_update_tolerance` double,
>`print_count` double,
>`printed_date` date,
>`reply_date` date,
>`reply_method_lookup_code` string,
>`rfq_close_date` date,
>`segment2` string,
>`segment3` string,
>`segment4` string,
>`segment5` string,
>`shipping_control` string,
>`start_date` date,
>`start_date_active` date,
>`summary_flag` string,
>`supply_agreement_flag` string,
>`usd_amount_limit` double,
>`usd_blanket_total_amount` double,
>`usd_exchange_rate` double,
>`usd_open_amount` double,
>`usd_order_amount` double,
>`ussgl_transaction_code` string,
>`xml_flag` string,
>`purchasing_organization_id` bigint,
>`purchasing_group_code` string,
>`last_updated_by_name` string,
>`created_by_name` string,
>`incoterms_1` string,
>`incoterms_2` string,
>`ame_approval_id` double,
>`ame_transaction_type` string,
>`auto_sourcing_flag` string,
>`cat_admin_auth_enabled_flag` string,
>`clm_document_number` string,
>`comm_rev_num` double,
>`consigned_consumption_flag` string,
>`consume_req_demand_flag` string,
>`conterms_articles_upd_date` timestamp,
>`conterms_deliv_upd_date` timestamp,
>`conterms_exist_flag` string,
>`cpa_reference` double,
>`created_language` string,
>`email_address` string,
>`enable_all_sites` string,
>`fax` string,
>`lock_owner_role` string,
>`lock_owner_user_id` double,
>`min_release_amount` double,
>`mrc_rate` string,
>`mrc_rate_date` string,
>`mrc_rate_type` string,
>`otm_recovery_flag` string,
>`otm_status_code` string,
>`pay_when_paid` string,
>`pcard_id` bigint,
>`program_update_date` timestamp,
>`quotation_class_code` string,
>
[jira] [Updated] (HIVE-20833) package.jdo needs to be updated to conform with HIVE-20221 changes
[
https://issues.apache.org/jira/browse/HIVE-20833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20833:
---
Status: Open (was: Patch Available)
> package.jdo needs to be updated to conform with HIVE-20221 changes
> --
>
> Key: HIVE-20833
> URL: https://issues.apache.org/jira/browse/HIVE-20833
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20833.1.patch, HIVE-20833.2.patch,
> HIVE-20833.3.patch
>
>
> Following test if run with TestMiniLlapLocalCliDriver will fail:
> {code:sql}
> CREATE TABLE `alterPartTbl`(
>`po_header_id` bigint,
>`vendor_num` string,
>`requester_name` string,
>`approver_name` string,
>`buyer_name` string,
>`preparer_name` string,
>`po_requisition_number` string,
>`po_requisition_id` bigint,
>`po_requisition_desc` string,
>`rate_type` string,
>`rate_date` date,
>`rate` double,
>`blanket_total_amount` double,
>`authorization_status` string,
>`revision_num` bigint,
>`revised_date` date,
>`approved_flag` string,
>`approved_date` timestamp,
>`amount_limit` double,
>`note_to_authorizer` string,
>`note_to_vendor` string,
>`note_to_receiver` string,
>`vendor_order_num` string,
>`comments` string,
>`acceptance_required_flag` string,
>`acceptance_due_date` date,
>`closed_date` timestamp,
>`user_hold_flag` string,
>`approval_required_flag` string,
>`cancel_flag` string,
>`firm_status_lookup_code` string,
>`firm_date` date,
>`frozen_flag` string,
>`closed_code` string,
>`org_id` bigint,
>`reference_num` string,
>`wf_item_type` string,
>`wf_item_key` string,
>`submit_date` date,
>`sap_company_code` string,
>`sap_fiscal_year` bigint,
>`po_number` string,
>`sap_line_item` bigint,
>`closed_status_flag` string,
>`balancing_segment` string,
>`cost_center_segment` string,
>`base_amount_limit` double,
>`base_blanket_total_amount` double,
>`base_open_amount` double,
>`base_ordered_amount` double,
>`cancel_date` timestamp,
>`cbc_accounting_date` date,
>`change_requested_by` string,
>`change_summary` string,
>`confirming_order_flag` string,
>`document_creation_method` string,
>`edi_processed_flag` string,
>`edi_processed_status` string,
>`enabled_flag` string,
>`encumbrance_required_flag` string,
>`end_date` date,
>`end_date_active` date,
>`from_header_id` bigint,
>`from_type_lookup_code` string,
>`global_agreement_flag` string,
>`government_context` string,
>`interface_source_code` string,
>`ledger_currency_code` string,
>`open_amount` double,
>`ordered_amount` double,
>`pay_on_code` string,
>`payment_term_name` string,
>`pending_signature_flag` string,
>`po_revision_num` double,
>`preparer_id` bigint,
>`price_update_tolerance` double,
>`print_count` double,
>`printed_date` date,
>`reply_date` date,
>`reply_method_lookup_code` string,
>`rfq_close_date` date,
>`segment2` string,
>`segment3` string,
>`segment4` string,
>`segment5` string,
>`shipping_control` string,
>`start_date` date,
>`start_date_active` date,
>`summary_flag` string,
>`supply_agreement_flag` string,
>`usd_amount_limit` double,
>`usd_blanket_total_amount` double,
>`usd_exchange_rate` double,
>`usd_open_amount` double,
>`usd_order_amount` double,
>`ussgl_transaction_code` string,
>`xml_flag` string,
>`purchasing_organization_id` bigint,
>`purchasing_group_code` string,
>`last_updated_by_name` string,
>`created_by_name` string,
>`incoterms_1` string,
>`incoterms_2` string,
>`ame_approval_id` double,
>`ame_transaction_type` string,
>`auto_sourcing_flag` string,
>`cat_admin_auth_enabled_flag` string,
>`clm_document_number` string,
>`comm_rev_num` double,
>`consigned_consumption_flag` string,
>`consume_req_demand_flag` string,
>`conterms_articles_upd_date` timestamp,
>`conterms_deliv_upd_date` timestamp,
>`conterms_exist_flag` string,
>`cpa_reference` double,
>`created_language` string,
>`email_address` string,
>`enable_all_sites` string,
>`fax` string,
>`lock_owner_role` string,
>`lock_owner_user_id` double,
>`min_release_amount` double,
>`mrc_rate` string,
>`mrc_rate_date` string,
>`mrc_rate_type` string,
>`otm_recovery_flag` string,
>`otm_status_code` string,
>`pay_when_paid` string,
>`pcard_id` bigint,
>`program_update_date` timestamp,
>`quotation_class_code` string,
>
[jira] [Updated] (HIVE-20833) package.jdo needs to be updated to conform with HIVE-20221 changes
[
https://issues.apache.org/jira/browse/HIVE-20833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20833:
---
Status: Patch Available (was: Open)
> package.jdo needs to be updated to conform with HIVE-20221 changes
> --
>
> Key: HIVE-20833
> URL: https://issues.apache.org/jira/browse/HIVE-20833
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20833.1.patch, HIVE-20833.2.patch,
> HIVE-20833.3.patch
>
>
> Following test if run with TestMiniLlapLocalCliDriver will fail:
> {code:sql}
> CREATE TABLE `alterPartTbl`(
>`po_header_id` bigint,
>`vendor_num` string,
>`requester_name` string,
>`approver_name` string,
>`buyer_name` string,
>`preparer_name` string,
>`po_requisition_number` string,
>`po_requisition_id` bigint,
>`po_requisition_desc` string,
>`rate_type` string,
>`rate_date` date,
>`rate` double,
>`blanket_total_amount` double,
>`authorization_status` string,
>`revision_num` bigint,
>`revised_date` date,
>`approved_flag` string,
>`approved_date` timestamp,
>`amount_limit` double,
>`note_to_authorizer` string,
>`note_to_vendor` string,
>`note_to_receiver` string,
>`vendor_order_num` string,
>`comments` string,
>`acceptance_required_flag` string,
>`acceptance_due_date` date,
>`closed_date` timestamp,
>`user_hold_flag` string,
>`approval_required_flag` string,
>`cancel_flag` string,
>`firm_status_lookup_code` string,
>`firm_date` date,
>`frozen_flag` string,
>`closed_code` string,
>`org_id` bigint,
>`reference_num` string,
>`wf_item_type` string,
>`wf_item_key` string,
>`submit_date` date,
>`sap_company_code` string,
>`sap_fiscal_year` bigint,
>`po_number` string,
>`sap_line_item` bigint,
>`closed_status_flag` string,
>`balancing_segment` string,
>`cost_center_segment` string,
>`base_amount_limit` double,
>`base_blanket_total_amount` double,
>`base_open_amount` double,
>`base_ordered_amount` double,
>`cancel_date` timestamp,
>`cbc_accounting_date` date,
>`change_requested_by` string,
>`change_summary` string,
>`confirming_order_flag` string,
>`document_creation_method` string,
>`edi_processed_flag` string,
>`edi_processed_status` string,
>`enabled_flag` string,
>`encumbrance_required_flag` string,
>`end_date` date,
>`end_date_active` date,
>`from_header_id` bigint,
>`from_type_lookup_code` string,
>`global_agreement_flag` string,
>`government_context` string,
>`interface_source_code` string,
>`ledger_currency_code` string,
>`open_amount` double,
>`ordered_amount` double,
>`pay_on_code` string,
>`payment_term_name` string,
>`pending_signature_flag` string,
>`po_revision_num` double,
>`preparer_id` bigint,
>`price_update_tolerance` double,
>`print_count` double,
>`printed_date` date,
>`reply_date` date,
>`reply_method_lookup_code` string,
>`rfq_close_date` date,
>`segment2` string,
>`segment3` string,
>`segment4` string,
>`segment5` string,
>`shipping_control` string,
>`start_date` date,
>`start_date_active` date,
>`summary_flag` string,
>`supply_agreement_flag` string,
>`usd_amount_limit` double,
>`usd_blanket_total_amount` double,
>`usd_exchange_rate` double,
>`usd_open_amount` double,
>`usd_order_amount` double,
>`ussgl_transaction_code` string,
>`xml_flag` string,
>`purchasing_organization_id` bigint,
>`purchasing_group_code` string,
>`last_updated_by_name` string,
>`created_by_name` string,
>`incoterms_1` string,
>`incoterms_2` string,
>`ame_approval_id` double,
>`ame_transaction_type` string,
>`auto_sourcing_flag` string,
>`cat_admin_auth_enabled_flag` string,
>`clm_document_number` string,
>`comm_rev_num` double,
>`consigned_consumption_flag` string,
>`consume_req_demand_flag` string,
>`conterms_articles_upd_date` timestamp,
>`conterms_deliv_upd_date` timestamp,
>`conterms_exist_flag` string,
>`cpa_reference` double,
>`created_language` string,
>`email_address` string,
>`enable_all_sites` string,
>`fax` string,
>`lock_owner_role` string,
>`lock_owner_user_id` double,
>`min_release_amount` double,
>`mrc_rate` string,
>`mrc_rate_date` string,
>`mrc_rate_type` string,
>`otm_recovery_flag` string,
>`otm_status_code` string,
>`pay_when_paid` string,
>`pcard_id` bigint,
>`program_update_date` timestamp,
>`quotation_class_code` string,
>
[jira] [Updated] (HIVE-20826) Enhance HiveSemiJoin rule to convert join + group by on left side to Left Semi Join
[
https://issues.apache.org/jira/browse/HIVE-20826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20826:
---
Status: Patch Available (was: Open)
> Enhance HiveSemiJoin rule to convert join + group by on left side to Left
> Semi Join
> ---
>
> Key: HIVE-20826
> URL: https://issues.apache.org/jira/browse/HIVE-20826
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20826.1.patch
>
>
> Currently HiveSemiJoin rule looks for pattern where group by is on right side.
> We can convert joins which have group by on left side (assuming group by keys
> are same as join keys and none of the columns are being projected from left
> side) to LEFT SEMI JOIN by swapping the inputs. e.g. queries such as:
> {code:sql}
> explain select pp.p_partkey from (select distinct p_name from part) p join
> part pp on pp.p_name = p.p_name;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20826) Enhance HiveSemiJoin rule to convert join + group by on left side to Left Semi Join
[
https://issues.apache.org/jira/browse/HIVE-20826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20826:
---
Attachment: HIVE-20826.1.patch
> Enhance HiveSemiJoin rule to convert join + group by on left side to Left
> Semi Join
> ---
>
> Key: HIVE-20826
> URL: https://issues.apache.org/jira/browse/HIVE-20826
> Project: Hive
> Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20826.1.patch
>
>
> Currently HiveSemiJoin rule looks for pattern where group by is on right side.
> We can convert joins which have group by on left side (assuming group by keys
> are same as join keys and none of the columns are being projected from left
> side) to LEFT SEMI JOIN by swapping the inputs. e.g. queries such as:
> {code:sql}
> explain select pp.p_partkey from (select distinct p_name from part) p join
> part pp on pp.p_name = p.p_name;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20786) Maven Build Failed with group id is too big
[
https://issues.apache.org/jira/browse/HIVE-20786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670957#comment-16670957
]
Vihang Karajgaonkar commented on HIVE-20786:
Thanks for the fix. LGTM +1
> Maven Build Failed with group id is too big
>
>
> Key: HIVE-20786
> URL: https://issues.apache.org/jira/browse/HIVE-20786
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
> Environment:
> OS: MacOS 10.13.6
> Java:
> {code}
> java version "1.8.0_192"
> Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
> {code}
> Maven:
> {code}
> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
> 2018-06-18T02:33:14+08:00)
> Maven home: /usr/local/Cellar/maven/3.5.4/libexec
> Java version: 1.8.0_192, vendor: Oracle Corporation, runtime:
> /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre
> Default locale: en_CN, platform encoding: UTF-8
> OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac"
> {code}
>
>
>Reporter: PENG Zhengshuai
>Assignee: Szehon Ho
>Priority: Major
> Labels: maven
> Attachments: HIVE-20786.2.patch, HIVE-20786.patch,
> hive_build_error.log
>
>
> When executing
> {code}
> mvn clean install -DskipTests
> {code}
> Build Failed:
> {code}
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Hive Storage API 2.7.0-SNAPSHOT SUCCESS [ 5.299
> s]
> [INFO] Hive 4.0.0-SNAPSHOT SUCCESS [ 0.750
> s]
> [INFO] Hive Classifications ... SUCCESS [ 1.057
> s]
> [INFO] Hive Shims Common .. SUCCESS [ 3.882
> s]
> [INFO] Hive Shims 0.23 SUCCESS [ 5.020
> s]
> [INFO] Hive Shims Scheduler ... SUCCESS [ 2.587
> s]
> [INFO] Hive Shims . SUCCESS [ 2.038
> s]
> [INFO] Hive Common SUCCESS [ 6.921
> s]
> [INFO] Hive Service RPC ... SUCCESS [ 3.503
> s]
> [INFO] Hive Serde . SUCCESS [ 6.322
> s]
> [INFO] Hive Standalone Metastore .. FAILURE [ 0.557
> s]
> [INFO] Hive Standalone Metastore Common Code .. SKIPPED
> [INFO] Hive Metastore . SKIPPED
> [INFO] Hive Vector-Code-Gen Utilities . SKIPPED
> [INFO] Hive Llap Common ... SKIPPED
> [INFO] Hive Llap Client ... SKIPPED
> [INFO] Hive Llap Tez .. SKIPPED
> [INFO] Hive Spark Remote Client ... SKIPPED
> [INFO] Hive Metastore Server .. SKIPPED
> [INFO] Hive Query Language SKIPPED
> [INFO] Hive Llap Server ... SKIPPED
> [INFO] Hive Service ... SKIPPED
> [INFO] Hive Accumulo Handler .. SKIPPED
> [INFO] Hive JDBC .. SKIPPED
> [INFO] Hive Beeline ... SKIPPED
> [INFO] Hive CLI ... SKIPPED
> [INFO] Hive Contrib ... SKIPPED
> [INFO] Hive Druid Handler . SKIPPED
> [INFO] Hive HBase Handler . SKIPPED
> [INFO] Hive JDBC Handler .. SKIPPED
> [INFO] Hive HCatalog .. SKIPPED
> [INFO] Hive HCatalog Core . SKIPPED
> [INFO] Hive HCatalog Pig Adapter .. SKIPPED
> [INFO] Hive HCatalog Server Extensions SKIPPED
> [INFO] Hive HCatalog Webhcat Java Client .. SKIPPED
> [INFO] Hive HCatalog Webhcat .. SKIPPED
> [INFO] Hive HCatalog Streaming SKIPPED
> [INFO] Hive HPL/SQL ... SKIPPED
> [INFO] Hive Streaming . SKIPPED
> [INFO] Hive Llap External Client .. SKIPPED
> [INFO] Hive Shims Aggregator .. SKIPPED
> [INFO] Hive Kryo Registrator .. SKIPPED
> [INFO] Hive TestUtils . SKIPPED
> [INFO] Hive Kafka Storage Handler . SKIPPED
> [INFO] Hive Packaging . SKIPPED
> [INFO]
[jira] [Commented] (HIVE-20786) Maven Build Failed with group id is too big
[
https://issues.apache.org/jira/browse/HIVE-20786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670912#comment-16670912
]
Szehon Ho commented on HIVE-20786:
--
So the problem was the top-level packaging was using an old version of maven
assembly before posix was even supported. This latest patch builds but also
upgrade the maven.assembly.plugin to the same one used in standalone-metastore.
> Maven Build Failed with group id is too big
>
>
> Key: HIVE-20786
> URL: https://issues.apache.org/jira/browse/HIVE-20786
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
> Environment:
> OS: MacOS 10.13.6
> Java:
> {code}
> java version "1.8.0_192"
> Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
> {code}
> Maven:
> {code}
> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
> 2018-06-18T02:33:14+08:00)
> Maven home: /usr/local/Cellar/maven/3.5.4/libexec
> Java version: 1.8.0_192, vendor: Oracle Corporation, runtime:
> /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre
> Default locale: en_CN, platform encoding: UTF-8
> OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac"
> {code}
>
>
>Reporter: PENG Zhengshuai
>Assignee: Szehon Ho
>Priority: Major
> Labels: maven
> Attachments: HIVE-20786.patch, hive_build_error.log
>
>
> When executing
> {code}
> mvn clean install -DskipTests
> {code}
> Build Failed:
> {code}
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Hive Storage API 2.7.0-SNAPSHOT SUCCESS [ 5.299
> s]
> [INFO] Hive 4.0.0-SNAPSHOT SUCCESS [ 0.750
> s]
> [INFO] Hive Classifications ... SUCCESS [ 1.057
> s]
> [INFO] Hive Shims Common .. SUCCESS [ 3.882
> s]
> [INFO] Hive Shims 0.23 SUCCESS [ 5.020
> s]
> [INFO] Hive Shims Scheduler ... SUCCESS [ 2.587
> s]
> [INFO] Hive Shims . SUCCESS [ 2.038
> s]
> [INFO] Hive Common SUCCESS [ 6.921
> s]
> [INFO] Hive Service RPC ... SUCCESS [ 3.503
> s]
> [INFO] Hive Serde . SUCCESS [ 6.322
> s]
> [INFO] Hive Standalone Metastore .. FAILURE [ 0.557
> s]
> [INFO] Hive Standalone Metastore Common Code .. SKIPPED
> [INFO] Hive Metastore . SKIPPED
> [INFO] Hive Vector-Code-Gen Utilities . SKIPPED
> [INFO] Hive Llap Common ... SKIPPED
> [INFO] Hive Llap Client ... SKIPPED
> [INFO] Hive Llap Tez .. SKIPPED
> [INFO] Hive Spark Remote Client ... SKIPPED
> [INFO] Hive Metastore Server .. SKIPPED
> [INFO] Hive Query Language SKIPPED
> [INFO] Hive Llap Server ... SKIPPED
> [INFO] Hive Service ... SKIPPED
> [INFO] Hive Accumulo Handler .. SKIPPED
> [INFO] Hive JDBC .. SKIPPED
> [INFO] Hive Beeline ... SKIPPED
> [INFO] Hive CLI ... SKIPPED
> [INFO] Hive Contrib ... SKIPPED
> [INFO] Hive Druid Handler . SKIPPED
> [INFO] Hive HBase Handler . SKIPPED
> [INFO] Hive JDBC Handler .. SKIPPED
> [INFO] Hive HCatalog .. SKIPPED
> [INFO] Hive HCatalog Core . SKIPPED
> [INFO] Hive HCatalog Pig Adapter .. SKIPPED
> [INFO] Hive HCatalog Server Extensions SKIPPED
> [INFO] Hive HCatalog Webhcat Java Client .. SKIPPED
> [INFO] Hive HCatalog Webhcat .. SKIPPED
> [INFO] Hive HCatalog Streaming SKIPPED
> [INFO] Hive HPL/SQL ... SKIPPED
> [INFO] Hive Streaming . SKIPPED
> [INFO] Hive Llap External Client .. SKIPPED
> [INFO] Hive Shims Aggregator .. SKIPPED
> [INFO] Hive Kryo Registrator .. SKIPPED
> [INFO] Hive TestUtils
[jira] [Updated] (HIVE-20786) Maven Build Failed with group id is too big
[
https://issues.apache.org/jira/browse/HIVE-20786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-20786:
-
Attachment: (was: HIVE-20789.2.patch)
> Maven Build Failed with group id is too big
>
>
> Key: HIVE-20786
> URL: https://issues.apache.org/jira/browse/HIVE-20786
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
> Environment:
> OS: MacOS 10.13.6
> Java:
> {code}
> java version "1.8.0_192"
> Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
> {code}
> Maven:
> {code}
> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
> 2018-06-18T02:33:14+08:00)
> Maven home: /usr/local/Cellar/maven/3.5.4/libexec
> Java version: 1.8.0_192, vendor: Oracle Corporation, runtime:
> /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre
> Default locale: en_CN, platform encoding: UTF-8
> OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac"
> {code}
>
>
>Reporter: PENG Zhengshuai
>Assignee: Szehon Ho
>Priority: Major
> Labels: maven
> Attachments: HIVE-20786.patch, hive_build_error.log
>
>
> When executing
> {code}
> mvn clean install -DskipTests
> {code}
> Build Failed:
> {code}
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Hive Storage API 2.7.0-SNAPSHOT SUCCESS [ 5.299
> s]
> [INFO] Hive 4.0.0-SNAPSHOT SUCCESS [ 0.750
> s]
> [INFO] Hive Classifications ... SUCCESS [ 1.057
> s]
> [INFO] Hive Shims Common .. SUCCESS [ 3.882
> s]
> [INFO] Hive Shims 0.23 SUCCESS [ 5.020
> s]
> [INFO] Hive Shims Scheduler ... SUCCESS [ 2.587
> s]
> [INFO] Hive Shims . SUCCESS [ 2.038
> s]
> [INFO] Hive Common SUCCESS [ 6.921
> s]
> [INFO] Hive Service RPC ... SUCCESS [ 3.503
> s]
> [INFO] Hive Serde . SUCCESS [ 6.322
> s]
> [INFO] Hive Standalone Metastore .. FAILURE [ 0.557
> s]
> [INFO] Hive Standalone Metastore Common Code .. SKIPPED
> [INFO] Hive Metastore . SKIPPED
> [INFO] Hive Vector-Code-Gen Utilities . SKIPPED
> [INFO] Hive Llap Common ... SKIPPED
> [INFO] Hive Llap Client ... SKIPPED
> [INFO] Hive Llap Tez .. SKIPPED
> [INFO] Hive Spark Remote Client ... SKIPPED
> [INFO] Hive Metastore Server .. SKIPPED
> [INFO] Hive Query Language SKIPPED
> [INFO] Hive Llap Server ... SKIPPED
> [INFO] Hive Service ... SKIPPED
> [INFO] Hive Accumulo Handler .. SKIPPED
> [INFO] Hive JDBC .. SKIPPED
> [INFO] Hive Beeline ... SKIPPED
> [INFO] Hive CLI ... SKIPPED
> [INFO] Hive Contrib ... SKIPPED
> [INFO] Hive Druid Handler . SKIPPED
> [INFO] Hive HBase Handler . SKIPPED
> [INFO] Hive JDBC Handler .. SKIPPED
> [INFO] Hive HCatalog .. SKIPPED
> [INFO] Hive HCatalog Core . SKIPPED
> [INFO] Hive HCatalog Pig Adapter .. SKIPPED
> [INFO] Hive HCatalog Server Extensions SKIPPED
> [INFO] Hive HCatalog Webhcat Java Client .. SKIPPED
> [INFO] Hive HCatalog Webhcat .. SKIPPED
> [INFO] Hive HCatalog Streaming SKIPPED
> [INFO] Hive HPL/SQL ... SKIPPED
> [INFO] Hive Streaming . SKIPPED
> [INFO] Hive Llap External Client .. SKIPPED
> [INFO] Hive Shims Aggregator .. SKIPPED
> [INFO] Hive Kryo Registrator .. SKIPPED
> [INFO] Hive TestUtils . SKIPPED
> [INFO] Hive Kafka Storage Handler . SKIPPED
> [INFO] Hive Packaging . SKIPPED
> [INFO] Hive Metastore Tools ... SKIPPED
>
[jira] [Updated] (HIVE-20786) Maven Build Failed with group id is too big
[
https://issues.apache.org/jira/browse/HIVE-20786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-20786:
-
Attachment: HIVE-20786.2.patch
> Maven Build Failed with group id is too big
>
>
> Key: HIVE-20786
> URL: https://issues.apache.org/jira/browse/HIVE-20786
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
> Environment:
> OS: MacOS 10.13.6
> Java:
> {code}
> java version "1.8.0_192"
> Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
> {code}
> Maven:
> {code}
> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
> 2018-06-18T02:33:14+08:00)
> Maven home: /usr/local/Cellar/maven/3.5.4/libexec
> Java version: 1.8.0_192, vendor: Oracle Corporation, runtime:
> /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre
> Default locale: en_CN, platform encoding: UTF-8
> OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac"
> {code}
>
>
>Reporter: PENG Zhengshuai
>Assignee: Szehon Ho
>Priority: Major
> Labels: maven
> Attachments: HIVE-20786.2.patch, HIVE-20786.patch,
> hive_build_error.log
>
>
> When executing
> {code}
> mvn clean install -DskipTests
> {code}
> Build Failed:
> {code}
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Hive Storage API 2.7.0-SNAPSHOT SUCCESS [ 5.299
> s]
> [INFO] Hive 4.0.0-SNAPSHOT SUCCESS [ 0.750
> s]
> [INFO] Hive Classifications ... SUCCESS [ 1.057
> s]
> [INFO] Hive Shims Common .. SUCCESS [ 3.882
> s]
> [INFO] Hive Shims 0.23 SUCCESS [ 5.020
> s]
> [INFO] Hive Shims Scheduler ... SUCCESS [ 2.587
> s]
> [INFO] Hive Shims . SUCCESS [ 2.038
> s]
> [INFO] Hive Common SUCCESS [ 6.921
> s]
> [INFO] Hive Service RPC ... SUCCESS [ 3.503
> s]
> [INFO] Hive Serde . SUCCESS [ 6.322
> s]
> [INFO] Hive Standalone Metastore .. FAILURE [ 0.557
> s]
> [INFO] Hive Standalone Metastore Common Code .. SKIPPED
> [INFO] Hive Metastore . SKIPPED
> [INFO] Hive Vector-Code-Gen Utilities . SKIPPED
> [INFO] Hive Llap Common ... SKIPPED
> [INFO] Hive Llap Client ... SKIPPED
> [INFO] Hive Llap Tez .. SKIPPED
> [INFO] Hive Spark Remote Client ... SKIPPED
> [INFO] Hive Metastore Server .. SKIPPED
> [INFO] Hive Query Language SKIPPED
> [INFO] Hive Llap Server ... SKIPPED
> [INFO] Hive Service ... SKIPPED
> [INFO] Hive Accumulo Handler .. SKIPPED
> [INFO] Hive JDBC .. SKIPPED
> [INFO] Hive Beeline ... SKIPPED
> [INFO] Hive CLI ... SKIPPED
> [INFO] Hive Contrib ... SKIPPED
> [INFO] Hive Druid Handler . SKIPPED
> [INFO] Hive HBase Handler . SKIPPED
> [INFO] Hive JDBC Handler .. SKIPPED
> [INFO] Hive HCatalog .. SKIPPED
> [INFO] Hive HCatalog Core . SKIPPED
> [INFO] Hive HCatalog Pig Adapter .. SKIPPED
> [INFO] Hive HCatalog Server Extensions SKIPPED
> [INFO] Hive HCatalog Webhcat Java Client .. SKIPPED
> [INFO] Hive HCatalog Webhcat .. SKIPPED
> [INFO] Hive HCatalog Streaming SKIPPED
> [INFO] Hive HPL/SQL ... SKIPPED
> [INFO] Hive Streaming . SKIPPED
> [INFO] Hive Llap External Client .. SKIPPED
> [INFO] Hive Shims Aggregator .. SKIPPED
> [INFO] Hive Kryo Registrator .. SKIPPED
> [INFO] Hive TestUtils . SKIPPED
> [INFO] Hive Kafka Storage Handler . SKIPPED
> [INFO] Hive Packaging . SKIPPED
> [INFO] Hive Metastore Tools ...
[jira] [Updated] (HIVE-20786) Maven Build Failed with group id is too big
[
https://issues.apache.org/jira/browse/HIVE-20786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szehon Ho updated HIVE-20786:
-
Attachment: HIVE-20789.2.patch
> Maven Build Failed with group id is too big
>
>
> Key: HIVE-20786
> URL: https://issues.apache.org/jira/browse/HIVE-20786
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
> Environment:
> OS: MacOS 10.13.6
> Java:
> {code}
> java version "1.8.0_192"
> Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
> {code}
> Maven:
> {code}
> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
> 2018-06-18T02:33:14+08:00)
> Maven home: /usr/local/Cellar/maven/3.5.4/libexec
> Java version: 1.8.0_192, vendor: Oracle Corporation, runtime:
> /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre
> Default locale: en_CN, platform encoding: UTF-8
> OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac"
> {code}
>
>
>Reporter: PENG Zhengshuai
>Assignee: Szehon Ho
>Priority: Major
> Labels: maven
> Attachments: HIVE-20786.patch, HIVE-20789.2.patch,
> hive_build_error.log
>
>
> When executing
> {code}
> mvn clean install -DskipTests
> {code}
> Build Failed:
> {code}
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Hive Storage API 2.7.0-SNAPSHOT SUCCESS [ 5.299
> s]
> [INFO] Hive 4.0.0-SNAPSHOT SUCCESS [ 0.750
> s]
> [INFO] Hive Classifications ... SUCCESS [ 1.057
> s]
> [INFO] Hive Shims Common .. SUCCESS [ 3.882
> s]
> [INFO] Hive Shims 0.23 SUCCESS [ 5.020
> s]
> [INFO] Hive Shims Scheduler ... SUCCESS [ 2.587
> s]
> [INFO] Hive Shims . SUCCESS [ 2.038
> s]
> [INFO] Hive Common SUCCESS [ 6.921
> s]
> [INFO] Hive Service RPC ... SUCCESS [ 3.503
> s]
> [INFO] Hive Serde . SUCCESS [ 6.322
> s]
> [INFO] Hive Standalone Metastore .. FAILURE [ 0.557
> s]
> [INFO] Hive Standalone Metastore Common Code .. SKIPPED
> [INFO] Hive Metastore . SKIPPED
> [INFO] Hive Vector-Code-Gen Utilities . SKIPPED
> [INFO] Hive Llap Common ... SKIPPED
> [INFO] Hive Llap Client ... SKIPPED
> [INFO] Hive Llap Tez .. SKIPPED
> [INFO] Hive Spark Remote Client ... SKIPPED
> [INFO] Hive Metastore Server .. SKIPPED
> [INFO] Hive Query Language SKIPPED
> [INFO] Hive Llap Server ... SKIPPED
> [INFO] Hive Service ... SKIPPED
> [INFO] Hive Accumulo Handler .. SKIPPED
> [INFO] Hive JDBC .. SKIPPED
> [INFO] Hive Beeline ... SKIPPED
> [INFO] Hive CLI ... SKIPPED
> [INFO] Hive Contrib ... SKIPPED
> [INFO] Hive Druid Handler . SKIPPED
> [INFO] Hive HBase Handler . SKIPPED
> [INFO] Hive JDBC Handler .. SKIPPED
> [INFO] Hive HCatalog .. SKIPPED
> [INFO] Hive HCatalog Core . SKIPPED
> [INFO] Hive HCatalog Pig Adapter .. SKIPPED
> [INFO] Hive HCatalog Server Extensions SKIPPED
> [INFO] Hive HCatalog Webhcat Java Client .. SKIPPED
> [INFO] Hive HCatalog Webhcat .. SKIPPED
> [INFO] Hive HCatalog Streaming SKIPPED
> [INFO] Hive HPL/SQL ... SKIPPED
> [INFO] Hive Streaming . SKIPPED
> [INFO] Hive Llap External Client .. SKIPPED
> [INFO] Hive Shims Aggregator .. SKIPPED
> [INFO] Hive Kryo Registrator .. SKIPPED
> [INFO] Hive TestUtils . SKIPPED
> [INFO] Hive Kafka Storage Handler . SKIPPED
> [INFO] Hive Packaging . SKIPPED
> [INFO] Hive Metastore Tools ...
[jira] [Commented] (HIVE-20653) Schema change in HIVE-19166 should also go to hive-schema-4.0.0.hive.sql
[ https://issues.apache.org/jira/browse/HIVE-20653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670909#comment-16670909 ] Daniel Dai commented on HIVE-20653: --- Should be fixed by now. Thanks [~vgarg] bringing it out. > Schema change in HIVE-19166 should also go to hive-schema-4.0.0.hive.sql > > > Key: HIVE-20653 > URL: https://issues.apache.org/jira/browse/HIVE-20653 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20653.1.patch, HIVE-20653.2.patch, > HIVE-20653.3.patch, HIVE-20653.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[
https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670892#comment-16670892
]
Antal Sinkovits commented on HIVE-20512:
LGTM
> Improve record and memory usage logging in SparkRecordHandler
> -
>
> Key: HIVE-20512
> URL: https://issues.apache.org/jira/browse/HIVE-20512
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20512.1.patch, HIVE-20512.2.patch,
> HIVE-20512.3.patch, HIVE-20512.4.patch, HIVE-20512.5.patch, HIVE-20512.6.patch
>
>
> We currently log memory usage and # of records processed in Spark tasks, but
> we should improve the methodology for how frequently we log this info.
> Currently we use the following code:
> {code:java}
> private long getNextLogThreshold(long currentThreshold) {
> // A very simple counter to keep track of number of rows processed by the
> // reducer. It dumps
> // every 1 million times, and quickly before that
> if (currentThreshold >= 100) {
> return currentThreshold + 100;
> }
> return 10 * currentThreshold;
> }
> {code}
> The issue is that after a while, the increase by 10x factor means that you
> have to process a huge # of records before this gets triggered.
> A better approach would be to log this info at a given interval. This would
> help in debugging tasks that are seemingly hung.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20850) Push case conditional from projections to dimension tables if possible
[
https://issues.apache.org/jira/browse/HIVE-20850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-20850:
Attachment: HIVE-20850.01.patch
> Push case conditional from projections to dimension tables if possible
> --
>
> Key: HIVE-20850
> URL: https://issues.apache.org/jira/browse/HIVE-20850
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20850.01.patch, HIVE-20850.01.patch
>
>
> noticed by [~gopalv]: If there is a project which could be only evaluated
> after the join; but the condition references only a single column from a
> small dimension table; hive will end up evaluating the same thing over and
> over again...
> {code}
> explain
> select s_store_name, s_store_id,
> sum(case when (d_day_name='Sunday') then ss_sales_price else null
> end) sun_sales,
> sum(case when (d_day_name='Monday') then ss_sales_price else null
> end) mon_sales,
> sum(case when (d_day_name='Tuesday') then ss_sales_price else null
> end) tue_sales,
> sum(case when (d_day_name='Wednesday') then ss_sales_price else null
> end) wed_sales,
> sum(case when (d_day_name='Thursday') then ss_sales_price else null
> end) thu_sales,
> sum(case when (d_day_name='Friday') then ss_sales_price else null
> end) fri_sales,
> sum(case when (d_day_name='Saturday') then ss_sales_price else null
> end) sat_sales
> from date_dim, store_sales, store
> where d_date_sk = ss_sold_date_sk and
>s_store_sk = ss_store_sk and
>s_gmt_offset = -6 and
>d_year = 1998
> group by s_store_name, s_store_id
> order by s_store_name,
> s_store_id,sun_sales,mon_sales,tue_sales,wed_sales,thu_sales,fri_sales,sat_sales
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[
https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-20512:
Attachment: HIVE-20512.6.patch
> Improve record and memory usage logging in SparkRecordHandler
> -
>
> Key: HIVE-20512
> URL: https://issues.apache.org/jira/browse/HIVE-20512
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-20512.1.patch, HIVE-20512.2.patch,
> HIVE-20512.3.patch, HIVE-20512.4.patch, HIVE-20512.5.patch, HIVE-20512.6.patch
>
>
> We currently log memory usage and # of records processed in Spark tasks, but
> we should improve the methodology for how frequently we log this info.
> Currently we use the following code:
> {code:java}
> private long getNextLogThreshold(long currentThreshold) {
> // A very simple counter to keep track of number of rows processed by the
> // reducer. It dumps
> // every 1 million times, and quickly before that
> if (currentThreshold >= 100) {
> return currentThreshold + 100;
> }
> return 10 * currentThreshold;
> }
> {code}
> The issue is that after a while, the increase by 10x factor means that you
> have to process a huge # of records before this gets triggered.
> A better approach would be to log this info at a given interval. This would
> help in debugging tasks that are seemingly hung.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20653) Schema change in HIVE-19166 should also go to hive-schema-4.0.0.hive.sql
[ https://issues.apache.org/jira/browse/HIVE-20653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670865#comment-16670865 ] Vineet Garg commented on HIVE-20653: [~daijy] Looks like this commit is failing sysdb, resourceplan and strict_managed_tables_sysdb tests - https://builds.apache.org/job/PreCommit-HIVE-Build/14691/testReport/ Can you please take a look? > Schema change in HIVE-19166 should also go to hive-schema-4.0.0.hive.sql > > > Key: HIVE-20653 > URL: https://issues.apache.org/jira/browse/HIVE-20653 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20653.1.patch, HIVE-20653.2.patch, > HIVE-20653.3.patch, HIVE-20653.4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20850) Push case conditional from projections to dimension tables if possible
[
https://issues.apache.org/jira/browse/HIVE-20850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-20850:
---
Summary: Push case conditional from projections to dimension tables if
possible (was: Puse case conditional from projections to dimension tables if
possible)
> Push case conditional from projections to dimension tables if possible
> --
>
> Key: HIVE-20850
> URL: https://issues.apache.org/jira/browse/HIVE-20850
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20850.01.patch
>
>
> noticed by [~gopalv]: If there is a project which could be only evaluated
> after the join; but the condition references only a single column from a
> small dimension table; hive will end up evaluating the same thing over and
> over again...
> {code}
> explain
> select s_store_name, s_store_id,
> sum(case when (d_day_name='Sunday') then ss_sales_price else null
> end) sun_sales,
> sum(case when (d_day_name='Monday') then ss_sales_price else null
> end) mon_sales,
> sum(case when (d_day_name='Tuesday') then ss_sales_price else null
> end) tue_sales,
> sum(case when (d_day_name='Wednesday') then ss_sales_price else null
> end) wed_sales,
> sum(case when (d_day_name='Thursday') then ss_sales_price else null
> end) thu_sales,
> sum(case when (d_day_name='Friday') then ss_sales_price else null
> end) fri_sales,
> sum(case when (d_day_name='Saturday') then ss_sales_price else null
> end) sat_sales
> from date_dim, store_sales, store
> where d_date_sk = ss_sold_date_sk and
>s_store_sk = ss_store_sk and
>s_gmt_offset = -6 and
>d_year = 1998
> group by s_store_name, s_store_id
> order by s_store_name,
> s_store_id,sun_sales,mon_sales,tue_sales,wed_sales,thu_sales,fri_sales,sat_sales
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20850) Puse case conditional from projections to dimension tables if possible
[
https://issues.apache.org/jira/browse/HIVE-20850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-20850:
Status: Patch Available (was: Open)
> Puse case conditional from projections to dimension tables if possible
> --
>
> Key: HIVE-20850
> URL: https://issues.apache.org/jira/browse/HIVE-20850
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20850.01.patch
>
>
> noticed by [~gopalv]: If there is a project which could be only evaluated
> after the join; but the condition references only a single column from a
> small dimension table; hive will end up evaluating the same thing over and
> over again...
> {code}
> explain
> select s_store_name, s_store_id,
> sum(case when (d_day_name='Sunday') then ss_sales_price else null
> end) sun_sales,
> sum(case when (d_day_name='Monday') then ss_sales_price else null
> end) mon_sales,
> sum(case when (d_day_name='Tuesday') then ss_sales_price else null
> end) tue_sales,
> sum(case when (d_day_name='Wednesday') then ss_sales_price else null
> end) wed_sales,
> sum(case when (d_day_name='Thursday') then ss_sales_price else null
> end) thu_sales,
> sum(case when (d_day_name='Friday') then ss_sales_price else null
> end) fri_sales,
> sum(case when (d_day_name='Saturday') then ss_sales_price else null
> end) sat_sales
> from date_dim, store_sales, store
> where d_date_sk = ss_sold_date_sk and
>s_store_sk = ss_store_sk and
>s_gmt_offset = -6 and
>d_year = 1998
> group by s_store_name, s_store_id
> order by s_store_name,
> s_store_id,sun_sales,mon_sales,tue_sales,wed_sales,thu_sales,fri_sales,sat_sales
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20850) Puse case conditional from projections to dimension tables if possible
[
https://issues.apache.org/jira/browse/HIVE-20850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670864#comment-16670864
]
Zoltan Haindrich commented on HIVE-20850:
-
patch#1 also contains HIVE-20835
> Puse case conditional from projections to dimension tables if possible
> --
>
> Key: HIVE-20850
> URL: https://issues.apache.org/jira/browse/HIVE-20850
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20850.01.patch
>
>
> noticed by [~gopalv]: If there is a project which could be only evaluated
> after the join; but the condition references only a single column from a
> small dimension table; hive will end up evaluating the same thing over and
> over again...
> {code}
> explain
> select s_store_name, s_store_id,
> sum(case when (d_day_name='Sunday') then ss_sales_price else null
> end) sun_sales,
> sum(case when (d_day_name='Monday') then ss_sales_price else null
> end) mon_sales,
> sum(case when (d_day_name='Tuesday') then ss_sales_price else null
> end) tue_sales,
> sum(case when (d_day_name='Wednesday') then ss_sales_price else null
> end) wed_sales,
> sum(case when (d_day_name='Thursday') then ss_sales_price else null
> end) thu_sales,
> sum(case when (d_day_name='Friday') then ss_sales_price else null
> end) fri_sales,
> sum(case when (d_day_name='Saturday') then ss_sales_price else null
> end) sat_sales
> from date_dim, store_sales, store
> where d_date_sk = ss_sold_date_sk and
>s_store_sk = ss_store_sk and
>s_gmt_offset = -6 and
>d_year = 1998
> group by s_store_name, s_store_id
> order by s_store_name,
> s_store_id,sun_sales,mon_sales,tue_sales,wed_sales,thu_sales,fri_sales,sat_sales
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20850) Puse case conditional from projections to dimension tables if possible
[
https://issues.apache.org/jira/browse/HIVE-20850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-20850:
Attachment: HIVE-20850.01.patch
> Puse case conditional from projections to dimension tables if possible
> --
>
> Key: HIVE-20850
> URL: https://issues.apache.org/jira/browse/HIVE-20850
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20850.01.patch
>
>
> noticed by [~gopalv]: If there is a project which could be only evaluated
> after the join; but the condition references only a single column from a
> small dimension table; hive will end up evaluating the same thing over and
> over again...
> {code}
> explain
> select s_store_name, s_store_id,
> sum(case when (d_day_name='Sunday') then ss_sales_price else null
> end) sun_sales,
> sum(case when (d_day_name='Monday') then ss_sales_price else null
> end) mon_sales,
> sum(case when (d_day_name='Tuesday') then ss_sales_price else null
> end) tue_sales,
> sum(case when (d_day_name='Wednesday') then ss_sales_price else null
> end) wed_sales,
> sum(case when (d_day_name='Thursday') then ss_sales_price else null
> end) thu_sales,
> sum(case when (d_day_name='Friday') then ss_sales_price else null
> end) fri_sales,
> sum(case when (d_day_name='Saturday') then ss_sales_price else null
> end) sat_sales
> from date_dim, store_sales, store
> where d_date_sk = ss_sold_date_sk and
>s_store_sk = ss_store_sk and
>s_gmt_offset = -6 and
>d_year = 1998
> group by s_store_name, s_store_id
> order by s_store_name,
> s_store_id,sun_sales,mon_sales,tue_sales,wed_sales,thu_sales,fri_sales,sat_sales
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20850) Puse case conditional from projections to dimension tables if possible
[
https://issues.apache.org/jira/browse/HIVE-20850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670863#comment-16670863
]
Zoltan Haindrich commented on HIVE-20850:
-
Apparently {{HiveProjectJoinTransposeRule}} is already able to extract the
expression from the condition - and the cbo plans which are [using
contraints|https://github.com/apache/hive/blob/4b40f7d2b35df26af773ec138fdf521231792cdf/ql/src/test/results/clientpositive/perf/tez/constraints/cbo_query43.q.out#L55]
already push these beyond the join - but [without
constraints|https://github.com/apache/hive/blob/4b40f7d2b35df26af773ec138fdf521231792cdf/ql/src/test/results/clientpositive/perf/tez/cbo_query43.q.out#L49]
the rule is not in action.
I'll try to enable enable it globally.
> Puse case conditional from projections to dimension tables if possible
> --
>
> Key: HIVE-20850
> URL: https://issues.apache.org/jira/browse/HIVE-20850
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20850.01.patch
>
>
> noticed by [~gopalv]: If there is a project which could be only evaluated
> after the join; but the condition references only a single column from a
> small dimension table; hive will end up evaluating the same thing over and
> over again...
> {code}
> explain
> select s_store_name, s_store_id,
> sum(case when (d_day_name='Sunday') then ss_sales_price else null
> end) sun_sales,
> sum(case when (d_day_name='Monday') then ss_sales_price else null
> end) mon_sales,
> sum(case when (d_day_name='Tuesday') then ss_sales_price else null
> end) tue_sales,
> sum(case when (d_day_name='Wednesday') then ss_sales_price else null
> end) wed_sales,
> sum(case when (d_day_name='Thursday') then ss_sales_price else null
> end) thu_sales,
> sum(case when (d_day_name='Friday') then ss_sales_price else null
> end) fri_sales,
> sum(case when (d_day_name='Saturday') then ss_sales_price else null
> end) sat_sales
> from date_dim, store_sales, store
> where d_date_sk = ss_sold_date_sk and
>s_store_sk = ss_store_sk and
>s_gmt_offset = -6 and
>d_year = 1998
> group by s_store_name, s_store_id
> order by s_store_name,
> s_store_id,sun_sales,mon_sales,tue_sales,wed_sales,thu_sales,fri_sales,sat_sales
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20850) Puse case conditional from projections to dimension tables if possible
[
https://issues.apache.org/jira/browse/HIVE-20850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-20850:
Summary: Puse case conditional from projections to dimension tables if
possible (was: Add rule to extract case conditional from projections)
> Puse case conditional from projections to dimension tables if possible
> --
>
> Key: HIVE-20850
> URL: https://issues.apache.org/jira/browse/HIVE-20850
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>
> noticed by [~gopalv]: If there is a project which could be only evaluated
> after the join; but the condition references only a single column from a
> small dimension table; hive will end up evaluating the same thing over and
> over again...
> {code}
> explain
> select s_store_name, s_store_id,
> sum(case when (d_day_name='Sunday') then ss_sales_price else null
> end) sun_sales,
> sum(case when (d_day_name='Monday') then ss_sales_price else null
> end) mon_sales,
> sum(case when (d_day_name='Tuesday') then ss_sales_price else null
> end) tue_sales,
> sum(case when (d_day_name='Wednesday') then ss_sales_price else null
> end) wed_sales,
> sum(case when (d_day_name='Thursday') then ss_sales_price else null
> end) thu_sales,
> sum(case when (d_day_name='Friday') then ss_sales_price else null
> end) fri_sales,
> sum(case when (d_day_name='Saturday') then ss_sales_price else null
> end) sat_sales
> from date_dim, store_sales, store
> where d_date_sk = ss_sold_date_sk and
>s_store_sk = ss_store_sk and
>s_gmt_offset = -6 and
>d_year = 1998
> group by s_store_name, s_store_id
> order by s_store_name,
> s_store_id,sun_sales,mon_sales,tue_sales,wed_sales,thu_sales,fri_sales,sat_sales
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20833) package.jdo needs to be updated to conform with HIVE-20221 changes
[
https://issues.apache.org/jira/browse/HIVE-20833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670857#comment-16670857
]
Hive QA commented on HIVE-20833:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12946451/HIVE-20833.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 15520 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_change_col]
(batchId=27)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[strict_managed_tables_sysdb]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=168)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/14691/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14691/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14691/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12946451 - PreCommit-HIVE-Build
> package.jdo needs to be updated to conform with HIVE-20221 changes
> --
>
> Key: HIVE-20833
> URL: https://issues.apache.org/jira/browse/HIVE-20833
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20833.1.patch, HIVE-20833.2.patch
>
>
> Following test if run with TestMiniLlapLocalCliDriver will fail:
> {code:sql}
> CREATE TABLE `alterPartTbl`(
>`po_header_id` bigint,
>`vendor_num` string,
>`requester_name` string,
>`approver_name` string,
>`buyer_name` string,
>`preparer_name` string,
>`po_requisition_number` string,
>`po_requisition_id` bigint,
>`po_requisition_desc` string,
>`rate_type` string,
>`rate_date` date,
>`rate` double,
>`blanket_total_amount` double,
>`authorization_status` string,
>`revision_num` bigint,
>`revised_date` date,
>`approved_flag` string,
>`approved_date` timestamp,
>`amount_limit` double,
>`note_to_authorizer` string,
>`note_to_vendor` string,
>`note_to_receiver` string,
>`vendor_order_num` string,
>`comments` string,
>`acceptance_required_flag` string,
>`acceptance_due_date` date,
>`closed_date` timestamp,
>`user_hold_flag` string,
>`approval_required_flag` string,
>`cancel_flag` string,
>`firm_status_lookup_code` string,
>`firm_date` date,
>`frozen_flag` string,
>`closed_code` string,
>`org_id` bigint,
>`reference_num` string,
>`wf_item_type` string,
>`wf_item_key` string,
>`submit_date` date,
>`sap_company_code` string,
>`sap_fiscal_year` bigint,
>`po_number` string,
>`sap_line_item` bigint,
>`closed_status_flag` string,
>`balancing_segment` string,
>`cost_center_segment` string,
>`base_amount_limit` double,
>`base_blanket_total_amount` double,
>`base_open_amount` double,
>`base_ordered_amount` double,
>`cancel_date` timestamp,
>`cbc_accounting_date` date,
>`change_requested_by` string,
>`change_summary` string,
>`confirming_order_flag` string,
>`document_creation_method` string,
>`edi_processed_flag` string,
>`edi_processed_status` string,
>`enabled_flag` string,
>`encumbrance_required_flag` string,
>`end_date` date,
>`end_date_active` date,
>`from_header_id` bigint,
>`from_type_lookup_code` string,
>`global_agreement_flag` string,
>`government_context` string,
>`interface_source_code` string,
>`ledger_currency_code` string,
>`open_amount` double,
>`ordered_amount` double,
>`pay_on_code` string,
>`payment_term_name` string,
>`pending_signature_flag` string,
>`po_revision_num` double,
>`preparer_id` bigint,
>`price_update_tolerance` double,
>`print_count` double,
>`printed_date` date,
>`reply_date` date,
>`reply_method_lookup_code` string,
>`rfq_close_date` date,
>`segment2` string,
>`segment3` string,
>`segment4` string,
>`segment5` string,
>`shipping_control` string,
>`start_date` date,
>`start_date_active` date,
>`summary_flag` string,
>
[jira] [Commented] (HIVE-20833) package.jdo needs to be updated to conform with HIVE-20221 changes
[
https://issues.apache.org/jira/browse/HIVE-20833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670816#comment-16670816
]
Hive QA commented on HIVE-20833:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
44s{color} | {color:blue} ql in master has 2315 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
6s{color} | {color:blue} standalone-metastore/metastore-server in master has
185 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14691/dev-support/hive-personality.sh
|
| git revision | master / 4b40f7d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: itests ql standalone-metastore/metastore-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-14691/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> package.jdo needs to be updated to conform with HIVE-20221 changes
> --
>
> Key: HIVE-20833
> URL: https://issues.apache.org/jira/browse/HIVE-20833
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20833.1.patch, HIVE-20833.2.patch
>
>
> Following test if run with TestMiniLlapLocalCliDriver will fail:
> {code:sql}
> CREATE TABLE `alterPartTbl`(
>`po_header_id` bigint,
>`vendor_num` string,
>`requester_name` string,
>`approver_name` string,
>`buyer_name` string,
>`preparer_name` string,
>`po_requisition_number` string,
>`po_requisition_id` bigint,
>`po_requisition_desc` string,
>`rate_type` string,
>`rate_date` date,
>`rate` double,
>`blanket_total_amount` double,
>`authorization_status` string,
>`revision_num` bigint,
>`revised_date` date,
>`approved_flag` string,
>`approved_date` timestamp,
>`amount_limit` double,
>`note_to_authorizer` string,
[jira] [Commented] (HIVE-20843) RELY constraints on primary keys and foreign keys are not recognized
[ https://issues.apache.org/jira/browse/HIVE-20843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670795#comment-16670795 ] Vihang Karajgaonkar commented on HIVE-20843: Lets wait for tests to come back (although given the backlog it may take a while) > RELY constraints on primary keys and foreign keys are not recognized > > > Key: HIVE-20843 > URL: https://issues.apache.org/jira/browse/HIVE-20843 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1, 3.0.0 >Reporter: Anurag Mantripragada >Assignee: Anurag Mantripragada >Priority: Major > Attachments: HIVE-20843.1-branch-2.patch, HIVE-20843.1.patch > > > Hive doesn't recognize RELY constraints after > https://issues.apache.org/jira/browse/HIVE-13076. The issue is in > BaseSemanticAnalyzer.java where we assign RELY. > An unrelated patch fixed this issue in later versions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20843) RELY constraints on primary keys and foreign keys are not recognized
[ https://issues.apache.org/jira/browse/HIVE-20843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670793#comment-16670793 ] Vihang Karajgaonkar commented on HIVE-20843: LGTM +1 > RELY constraints on primary keys and foreign keys are not recognized > > > Key: HIVE-20843 > URL: https://issues.apache.org/jira/browse/HIVE-20843 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1, 3.0.0 >Reporter: Anurag Mantripragada >Assignee: Anurag Mantripragada >Priority: Major > Attachments: HIVE-20843.1-branch-2.patch, HIVE-20843.1.patch > > > Hive doesn't recognize RELY constraints after > https://issues.apache.org/jira/browse/HIVE-13076. The issue is in > BaseSemanticAnalyzer.java where we assign RELY. > An unrelated patch fixed this issue in later versions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20460) AcidUtils.Directory.getAbortedDirectories() may be missed for full CRUD tables
[
https://issues.apache.org/jira/browse/HIVE-20460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670775#comment-16670775
]
Eugene Koifman commented on HIVE-20460:
---
the system isn't quite ready for this. full CRUD still relies on rename().
> AcidUtils.Directory.getAbortedDirectories() may be missed for full CRUD tables
> --
>
> Key: HIVE-20460
> URL: https://issues.apache.org/jira/browse/HIVE-20460
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> {{Directory.getAbortedDirectories()}} lists deltas where all txns in the
> range are aborted.
> These are then purged by {{Worker}} (\{{CompactorMR}} but only for
> insert-only tables.
> Full CRUD tables currently rely on {{FileSystem.rename()}} in {{MoveTask}}
> and so no reader (or {{Cleaner}} should every see a delta where all data is
> aborted.
>
> Once rename() is eliminated for full CRUD (just like insert-only)
> transactional tables, Cleaner (or Worker) should take care of these.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20850) Add rule to extract case conditional from projections
[
https://issues.apache.org/jira/browse/HIVE-20850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich reassigned HIVE-20850:
---
> Add rule to extract case conditional from projections
> -
>
> Key: HIVE-20850
> URL: https://issues.apache.org/jira/browse/HIVE-20850
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>
> noticed by [~gopalv]: If there is a project which could be only evaluated
> after the join; but the condition references only a single column from a
> small dimension table; hive will end up evaluating the same thing over and
> over again...
> {code}
> explain
> select s_store_name, s_store_id,
> sum(case when (d_day_name='Sunday') then ss_sales_price else null
> end) sun_sales,
> sum(case when (d_day_name='Monday') then ss_sales_price else null
> end) mon_sales,
> sum(case when (d_day_name='Tuesday') then ss_sales_price else null
> end) tue_sales,
> sum(case when (d_day_name='Wednesday') then ss_sales_price else null
> end) wed_sales,
> sum(case when (d_day_name='Thursday') then ss_sales_price else null
> end) thu_sales,
> sum(case when (d_day_name='Friday') then ss_sales_price else null
> end) fri_sales,
> sum(case when (d_day_name='Saturday') then ss_sales_price else null
> end) sat_sales
> from date_dim, store_sales, store
> where d_date_sk = ss_sold_date_sk and
>s_store_sk = ss_store_sk and
>s_gmt_offset = -6 and
>d_year = 1998
> group by s_store_name, s_store_id
> order by s_store_name,
> s_store_id,sun_sales,mon_sales,tue_sales,wed_sales,thu_sales,fri_sales,sat_sales
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20617) Fix type of constants in IN expressions to have correct type
[
https://issues.apache.org/jira/browse/HIVE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-20617:
Attachment: HIVE-20617.16.patch
> Fix type of constants in IN expressions to have correct type
>
>
> Key: HIVE-20617
> URL: https://issues.apache.org/jira/browse/HIVE-20617
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20617.01.patch, HIVE-20617.02.patch,
> HIVE-20617.03.patch, HIVE-20617.05.patch, HIVE-20617.06.patch,
> HIVE-20617.07.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.09.patch, HIVE-20617.10.patch, HIVE-20617.10.patch,
> HIVE-20617.11.patch, HIVE-20617.11.patch, HIVE-20617.12.patch,
> HIVE-20617.13.patch, HIVE-20617.14.patch, HIVE-20617.15.patch,
> HIVE-20617.16.patch
>
>
> In statements like {{struct(a,b) IN (const struct('x','y'), ... )}} the
> comparision in UDFIn may fail because if a or b is of char/varchar type the
> constants will retain string type - especially after PointlookupOptimizer
> compaction.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20833) package.jdo needs to be updated to conform with HIVE-20221 changes
[
https://issues.apache.org/jira/browse/HIVE-20833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20833:
---
Attachment: HIVE-20833.2.patch
> package.jdo needs to be updated to conform with HIVE-20221 changes
> --
>
> Key: HIVE-20833
> URL: https://issues.apache.org/jira/browse/HIVE-20833
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20833.1.patch, HIVE-20833.2.patch
>
>
> Following test if run with TestMiniLlapLocalCliDriver will fail:
> {code:sql}
> CREATE TABLE `alterPartTbl`(
>`po_header_id` bigint,
>`vendor_num` string,
>`requester_name` string,
>`approver_name` string,
>`buyer_name` string,
>`preparer_name` string,
>`po_requisition_number` string,
>`po_requisition_id` bigint,
>`po_requisition_desc` string,
>`rate_type` string,
>`rate_date` date,
>`rate` double,
>`blanket_total_amount` double,
>`authorization_status` string,
>`revision_num` bigint,
>`revised_date` date,
>`approved_flag` string,
>`approved_date` timestamp,
>`amount_limit` double,
>`note_to_authorizer` string,
>`note_to_vendor` string,
>`note_to_receiver` string,
>`vendor_order_num` string,
>`comments` string,
>`acceptance_required_flag` string,
>`acceptance_due_date` date,
>`closed_date` timestamp,
>`user_hold_flag` string,
>`approval_required_flag` string,
>`cancel_flag` string,
>`firm_status_lookup_code` string,
>`firm_date` date,
>`frozen_flag` string,
>`closed_code` string,
>`org_id` bigint,
>`reference_num` string,
>`wf_item_type` string,
>`wf_item_key` string,
>`submit_date` date,
>`sap_company_code` string,
>`sap_fiscal_year` bigint,
>`po_number` string,
>`sap_line_item` bigint,
>`closed_status_flag` string,
>`balancing_segment` string,
>`cost_center_segment` string,
>`base_amount_limit` double,
>`base_blanket_total_amount` double,
>`base_open_amount` double,
>`base_ordered_amount` double,
>`cancel_date` timestamp,
>`cbc_accounting_date` date,
>`change_requested_by` string,
>`change_summary` string,
>`confirming_order_flag` string,
>`document_creation_method` string,
>`edi_processed_flag` string,
>`edi_processed_status` string,
>`enabled_flag` string,
>`encumbrance_required_flag` string,
>`end_date` date,
>`end_date_active` date,
>`from_header_id` bigint,
>`from_type_lookup_code` string,
>`global_agreement_flag` string,
>`government_context` string,
>`interface_source_code` string,
>`ledger_currency_code` string,
>`open_amount` double,
>`ordered_amount` double,
>`pay_on_code` string,
>`payment_term_name` string,
>`pending_signature_flag` string,
>`po_revision_num` double,
>`preparer_id` bigint,
>`price_update_tolerance` double,
>`print_count` double,
>`printed_date` date,
>`reply_date` date,
>`reply_method_lookup_code` string,
>`rfq_close_date` date,
>`segment2` string,
>`segment3` string,
>`segment4` string,
>`segment5` string,
>`shipping_control` string,
>`start_date` date,
>`start_date_active` date,
>`summary_flag` string,
>`supply_agreement_flag` string,
>`usd_amount_limit` double,
>`usd_blanket_total_amount` double,
>`usd_exchange_rate` double,
>`usd_open_amount` double,
>`usd_order_amount` double,
>`ussgl_transaction_code` string,
>`xml_flag` string,
>`purchasing_organization_id` bigint,
>`purchasing_group_code` string,
>`last_updated_by_name` string,
>`created_by_name` string,
>`incoterms_1` string,
>`incoterms_2` string,
>`ame_approval_id` double,
>`ame_transaction_type` string,
>`auto_sourcing_flag` string,
>`cat_admin_auth_enabled_flag` string,
>`clm_document_number` string,
>`comm_rev_num` double,
>`consigned_consumption_flag` string,
>`consume_req_demand_flag` string,
>`conterms_articles_upd_date` timestamp,
>`conterms_deliv_upd_date` timestamp,
>`conterms_exist_flag` string,
>`cpa_reference` double,
>`created_language` string,
>`email_address` string,
>`enable_all_sites` string,
>`fax` string,
>`lock_owner_role` string,
>`lock_owner_user_id` double,
>`min_release_amount` double,
>`mrc_rate` string,
>`mrc_rate_date` string,
>`mrc_rate_type` string,
>`otm_recovery_flag` string,
>`otm_status_code` string,
>`pay_when_paid` string,
>`pcard_id` bigint,
>`program_update_date` timestamp,
>`quotation_class_code` string,
>`quote_type_lookup_code` string,
>
[jira] [Updated] (HIVE-20833) package.jdo needs to be updated to conform with HIVE-20221 changes
[
https://issues.apache.org/jira/browse/HIVE-20833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20833:
---
Attachment: (was: HIVE-20833.2.patch)
> package.jdo needs to be updated to conform with HIVE-20221 changes
> --
>
> Key: HIVE-20833
> URL: https://issues.apache.org/jira/browse/HIVE-20833
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20833.1.patch, HIVE-20833.2.patch
>
>
> Following test if run with TestMiniLlapLocalCliDriver will fail:
> {code:sql}
> CREATE TABLE `alterPartTbl`(
>`po_header_id` bigint,
>`vendor_num` string,
>`requester_name` string,
>`approver_name` string,
>`buyer_name` string,
>`preparer_name` string,
>`po_requisition_number` string,
>`po_requisition_id` bigint,
>`po_requisition_desc` string,
>`rate_type` string,
>`rate_date` date,
>`rate` double,
>`blanket_total_amount` double,
>`authorization_status` string,
>`revision_num` bigint,
>`revised_date` date,
>`approved_flag` string,
>`approved_date` timestamp,
>`amount_limit` double,
>`note_to_authorizer` string,
>`note_to_vendor` string,
>`note_to_receiver` string,
>`vendor_order_num` string,
>`comments` string,
>`acceptance_required_flag` string,
>`acceptance_due_date` date,
>`closed_date` timestamp,
>`user_hold_flag` string,
>`approval_required_flag` string,
>`cancel_flag` string,
>`firm_status_lookup_code` string,
>`firm_date` date,
>`frozen_flag` string,
>`closed_code` string,
>`org_id` bigint,
>`reference_num` string,
>`wf_item_type` string,
>`wf_item_key` string,
>`submit_date` date,
>`sap_company_code` string,
>`sap_fiscal_year` bigint,
>`po_number` string,
>`sap_line_item` bigint,
>`closed_status_flag` string,
>`balancing_segment` string,
>`cost_center_segment` string,
>`base_amount_limit` double,
>`base_blanket_total_amount` double,
>`base_open_amount` double,
>`base_ordered_amount` double,
>`cancel_date` timestamp,
>`cbc_accounting_date` date,
>`change_requested_by` string,
>`change_summary` string,
>`confirming_order_flag` string,
>`document_creation_method` string,
>`edi_processed_flag` string,
>`edi_processed_status` string,
>`enabled_flag` string,
>`encumbrance_required_flag` string,
>`end_date` date,
>`end_date_active` date,
>`from_header_id` bigint,
>`from_type_lookup_code` string,
>`global_agreement_flag` string,
>`government_context` string,
>`interface_source_code` string,
>`ledger_currency_code` string,
>`open_amount` double,
>`ordered_amount` double,
>`pay_on_code` string,
>`payment_term_name` string,
>`pending_signature_flag` string,
>`po_revision_num` double,
>`preparer_id` bigint,
>`price_update_tolerance` double,
>`print_count` double,
>`printed_date` date,
>`reply_date` date,
>`reply_method_lookup_code` string,
>`rfq_close_date` date,
>`segment2` string,
>`segment3` string,
>`segment4` string,
>`segment5` string,
>`shipping_control` string,
>`start_date` date,
>`start_date_active` date,
>`summary_flag` string,
>`supply_agreement_flag` string,
>`usd_amount_limit` double,
>`usd_blanket_total_amount` double,
>`usd_exchange_rate` double,
>`usd_open_amount` double,
>`usd_order_amount` double,
>`ussgl_transaction_code` string,
>`xml_flag` string,
>`purchasing_organization_id` bigint,
>`purchasing_group_code` string,
>`last_updated_by_name` string,
>`created_by_name` string,
>`incoterms_1` string,
>`incoterms_2` string,
>`ame_approval_id` double,
>`ame_transaction_type` string,
>`auto_sourcing_flag` string,
>`cat_admin_auth_enabled_flag` string,
>`clm_document_number` string,
>`comm_rev_num` double,
>`consigned_consumption_flag` string,
>`consume_req_demand_flag` string,
>`conterms_articles_upd_date` timestamp,
>`conterms_deliv_upd_date` timestamp,
>`conterms_exist_flag` string,
>`cpa_reference` double,
>`created_language` string,
>`email_address` string,
>`enable_all_sites` string,
>`fax` string,
>`lock_owner_role` string,
>`lock_owner_user_id` double,
>`min_release_amount` double,
>`mrc_rate` string,
>`mrc_rate_date` string,
>`mrc_rate_type` string,
>`otm_recovery_flag` string,
>`otm_status_code` string,
>`pay_when_paid` string,
>`pcard_id` bigint,
>`program_update_date` timestamp,
>`quotation_class_code` string,
>`quote_type_lookup_code`
[jira] [Updated] (HIVE-20848) After setting UpdateInputAccessTimeHook query fail with Table Not Found.
[
https://issues.apache.org/jira/browse/HIVE-20848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajkumar Singh updated HIVE-20848:
--
Attachment: HIVE-20848.patch
Status: Patch Available (was: In Progress)
> After setting UpdateInputAccessTimeHook query fail with Table Not Found.
>
>
> Key: HIVE-20848
> URL: https://issues.apache.org/jira/browse/HIVE-20848
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-20848.patch
>
>
> {code}
> select from_unixtime(1540495168);
> set
> hive.exec.pre.hooks=org.apache.hadoop.hive.ql.hooks.ATSHook,org.apache.hadoop.hive.ql.hooks.UpdateInputAccessTimeHook$PreExec;
> select from_unixtime(1540495168);
> {code}
> the second select fail with following exception
> {code}
> ERROR ql.Driver: FAILED: Hive Internal Error:
> org.apache.hadoop.hive.ql.metadata.InvalidTableException(Table not found
> _dummy_table)
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found
> _dummy_table
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1217)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1168)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1155)
> at
> org.apache.hadoop.hive.ql.hooks.UpdateInputAccessTimeHook$PreExec.run(UpdateInputAccessTimeHook.java:67)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1444)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1294)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20849) Review of ConstantPropagateProcFactory
[ https://issues.apache.org/jira/browse/HIVE-20849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-20849: --- Status: Patch Available (was: Open) > Review of ConstantPropagateProcFactory > -- > > Key: HIVE-20849 > URL: https://issues.apache.org/jira/browse/HIVE-20849 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 3.1.0, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-20849.1.patch > > > I was looking at this class because it blasts a lot of useless (to an admin) > information to the logs. Especially if the table has a lot of columns, I see > big blocks of logging that are meaningless to me. I request that the logging > is toned down to debug, and some other improvements to the code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20849) Review of ConstantPropagateProcFactory
[ https://issues.apache.org/jira/browse/HIVE-20849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-20849: --- Attachment: HIVE-20849.1.patch > Review of ConstantPropagateProcFactory > -- > > Key: HIVE-20849 > URL: https://issues.apache.org/jira/browse/HIVE-20849 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 3.1.0, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-20849.1.patch > > > I was looking at this class because it blasts a lot of useless (to an admin) > information to the logs. Especially if the table has a lot of columns, I see > big blocks of logging that are meaningless to me. I request that the logging > is toned down to debug, and some other improvements to the code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20849) Review of ConstantPropagateProcFactory
[ https://issues.apache.org/jira/browse/HIVE-20849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR reassigned HIVE-20849: -- Assignee: BELUGA BEHR > Review of ConstantPropagateProcFactory > -- > > Key: HIVE-20849 > URL: https://issues.apache.org/jira/browse/HIVE-20849 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 3.1.0, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-20849.1.patch > > > I was looking at this class because it blasts a lot of useless (to an admin) > information to the logs. Especially if the table has a lot of columns, I see > big blocks of logging that are meaningless to me. I request that the logging > is toned down to debug, and some other improvements to the code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20848) After setting UpdateInputAccessTimeHook query fail with Table Not Found.
[
https://issues.apache.org/jira/browse/HIVE-20848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajkumar Singh reassigned HIVE-20848:
-
> After setting UpdateInputAccessTimeHook query fail with Table Not Found.
>
>
> Key: HIVE-20848
> URL: https://issues.apache.org/jira/browse/HIVE-20848
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
>
> {code}
> select from_unixtime(1540495168);
> set
> hive.exec.pre.hooks=org.apache.hadoop.hive.ql.hooks.ATSHook,org.apache.hadoop.hive.ql.hooks.UpdateInputAccessTimeHook$PreExec;
> select from_unixtime(1540495168);
> {code}
> the second select fail with following exception
> {code}
> ERROR ql.Driver: FAILED: Hive Internal Error:
> org.apache.hadoop.hive.ql.metadata.InvalidTableException(Table not found
> _dummy_table)
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found
> _dummy_table
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1217)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1168)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1155)
> at
> org.apache.hadoop.hive.ql.hooks.UpdateInputAccessTimeHook$PreExec.run(UpdateInputAccessTimeHook.java:67)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1444)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1294)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-20848) After setting UpdateInputAccessTimeHook query fail with Table Not Found.
[
https://issues.apache.org/jira/browse/HIVE-20848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-20848 started by Rajkumar Singh.
-
> After setting UpdateInputAccessTimeHook query fail with Table Not Found.
>
>
> Key: HIVE-20848
> URL: https://issues.apache.org/jira/browse/HIVE-20848
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
>
> {code}
> select from_unixtime(1540495168);
> set
> hive.exec.pre.hooks=org.apache.hadoop.hive.ql.hooks.ATSHook,org.apache.hadoop.hive.ql.hooks.UpdateInputAccessTimeHook$PreExec;
> select from_unixtime(1540495168);
> {code}
> the second select fail with following exception
> {code}
> ERROR ql.Driver: FAILED: Hive Internal Error:
> org.apache.hadoop.hive.ql.metadata.InvalidTableException(Table not found
> _dummy_table)
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found
> _dummy_table
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1217)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1168)
> at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:1155)
> at
> org.apache.hadoop.hive.ql.hooks.UpdateInputAccessTimeHook$PreExec.run(UpdateInputAccessTimeHook.java:67)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1444)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1294)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1156)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
> at
> org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20843) RELY constraints on primary keys and foreign keys are not recognized
[ https://issues.apache.org/jira/browse/HIVE-20843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670668#comment-16670668 ] Anurag Mantripragada edited comment on HIVE-20843 at 10/31/18 8:22 PM: --- Thanks [~vihangk1] . Added patch for branch-2. The latest version 3 release doesn't have the bug. was (Author: anuragmantri): Thanks [~vihangk1] . Added patch for branch-2. The latest 3 release doesn't have the bug. > RELY constraints on primary keys and foreign keys are not recognized > > > Key: HIVE-20843 > URL: https://issues.apache.org/jira/browse/HIVE-20843 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1, 3.0.0 >Reporter: Anurag Mantripragada >Assignee: Anurag Mantripragada >Priority: Major > Attachments: HIVE-20843.1-branch-2.patch, HIVE-20843.1.patch > > > Hive doesn't recognize RELY constraints after > https://issues.apache.org/jira/browse/HIVE-13076. The issue is in > BaseSemanticAnalyzer.java where we assign RELY. > An unrelated patch fixed this issue in later versions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20843) RELY constraints on primary keys and foreign keys are not recognized
[ https://issues.apache.org/jira/browse/HIVE-20843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada updated HIVE-20843: Attachment: HIVE-20843.1-branch-2.patch Status: Patch Available (was: Open) Thanks [~vihangk1] . Added patch for branch-2. The latest 3 release doesn't have the bug. > RELY constraints on primary keys and foreign keys are not recognized > > > Key: HIVE-20843 > URL: https://issues.apache.org/jira/browse/HIVE-20843 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.1 >Reporter: Anurag Mantripragada >Assignee: Anurag Mantripragada >Priority: Major > Attachments: HIVE-20843.1-branch-2.patch, HIVE-20843.1.patch > > > Hive doesn't recognize RELY constraints after > https://issues.apache.org/jira/browse/HIVE-13076. The issue is in > BaseSemanticAnalyzer.java where we assign RELY. > An unrelated patch fixed this issue in later versions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20843) RELY constraints on primary keys and foreign keys are not recognized
[ https://issues.apache.org/jira/browse/HIVE-20843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anurag Mantripragada updated HIVE-20843: Status: Open (was: Patch Available) > RELY constraints on primary keys and foreign keys are not recognized > > > Key: HIVE-20843 > URL: https://issues.apache.org/jira/browse/HIVE-20843 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.1 >Reporter: Anurag Mantripragada >Assignee: Anurag Mantripragada >Priority: Major > Attachments: HIVE-20843.1.patch > > > Hive doesn't recognize RELY constraints after > https://issues.apache.org/jira/browse/HIVE-13076. The issue is in > BaseSemanticAnalyzer.java where we assign RELY. > An unrelated patch fixed this issue in later versions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20460) AcidUtils.Directory.getAbortedDirectories() may be missed for full CRUD tables
[
https://issues.apache.org/jira/browse/HIVE-20460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670648#comment-16670648
]
Igor Kryvenko commented on HIVE-20460:
--
Hi [~ekoifman] . Can I take this JIRA?
> AcidUtils.Directory.getAbortedDirectories() may be missed for full CRUD tables
> --
>
> Key: HIVE-20460
> URL: https://issues.apache.org/jira/browse/HIVE-20460
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> {{Directory.getAbortedDirectories()}} lists deltas where all txns in the
> range are aborted.
> These are then purged by {{Worker}} (\{{CompactorMR}} but only for
> insert-only tables.
> Full CRUD tables currently rely on {{FileSystem.rename()}} in {{MoveTask}}
> and so no reader (or {{Cleaner}} should every see a delta where all data is
> aborted.
>
> Once rename() is eliminated for full CRUD (just like insert-only)
> transactional tables, Cleaner (or Worker) should take care of these.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20847) Review of NullScan Code
[
https://issues.apache.org/jira/browse/HIVE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20847:
---
Summary: Review of NullScan Code (was: Review of NullScanCode)
> Review of NullScan Code
> ---
>
> Key: HIVE-20847
> URL: https://issues.apache.org/jira/browse/HIVE-20847
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20847.1.patch
>
>
> What got me looking at this class was the verboseness of some of the logging.
> I would like to request that we DEBUG the logging since this level of detail
> means nothing to a cluster admin.
> Also... this {{contains}} call would be better applied onto a {{HashSet}}
> instead of an {{ArrayList}}.
> {code:java|title=NullScanTaskDispatcher.java}
> private void processAlias(MapWork work, Path path, ArrayList
> aliasesAffected, ArrayList aliases) {
> // the aliases that are allowed to map to a null scan.
> ArrayList allowed = new ArrayList();
> for (String alias : aliasesAffected) {
> if (aliases.contains(alias)) {
> allowed.add(alias);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20833) package.jdo needs to be updated to conform with HIVE-20221 changes
[
https://issues.apache.org/jira/browse/HIVE-20833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20833:
---
Status: Open (was: Patch Available)
> package.jdo needs to be updated to conform with HIVE-20221 changes
> --
>
> Key: HIVE-20833
> URL: https://issues.apache.org/jira/browse/HIVE-20833
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20833.1.patch, HIVE-20833.2.patch
>
>
> Following test if run with TestMiniLlapLocalCliDriver will fail:
> {code:sql}
> CREATE TABLE `alterPartTbl`(
>`po_header_id` bigint,
>`vendor_num` string,
>`requester_name` string,
>`approver_name` string,
>`buyer_name` string,
>`preparer_name` string,
>`po_requisition_number` string,
>`po_requisition_id` bigint,
>`po_requisition_desc` string,
>`rate_type` string,
>`rate_date` date,
>`rate` double,
>`blanket_total_amount` double,
>`authorization_status` string,
>`revision_num` bigint,
>`revised_date` date,
>`approved_flag` string,
>`approved_date` timestamp,
>`amount_limit` double,
>`note_to_authorizer` string,
>`note_to_vendor` string,
>`note_to_receiver` string,
>`vendor_order_num` string,
>`comments` string,
>`acceptance_required_flag` string,
>`acceptance_due_date` date,
>`closed_date` timestamp,
>`user_hold_flag` string,
>`approval_required_flag` string,
>`cancel_flag` string,
>`firm_status_lookup_code` string,
>`firm_date` date,
>`frozen_flag` string,
>`closed_code` string,
>`org_id` bigint,
>`reference_num` string,
>`wf_item_type` string,
>`wf_item_key` string,
>`submit_date` date,
>`sap_company_code` string,
>`sap_fiscal_year` bigint,
>`po_number` string,
>`sap_line_item` bigint,
>`closed_status_flag` string,
>`balancing_segment` string,
>`cost_center_segment` string,
>`base_amount_limit` double,
>`base_blanket_total_amount` double,
>`base_open_amount` double,
>`base_ordered_amount` double,
>`cancel_date` timestamp,
>`cbc_accounting_date` date,
>`change_requested_by` string,
>`change_summary` string,
>`confirming_order_flag` string,
>`document_creation_method` string,
>`edi_processed_flag` string,
>`edi_processed_status` string,
>`enabled_flag` string,
>`encumbrance_required_flag` string,
>`end_date` date,
>`end_date_active` date,
>`from_header_id` bigint,
>`from_type_lookup_code` string,
>`global_agreement_flag` string,
>`government_context` string,
>`interface_source_code` string,
>`ledger_currency_code` string,
>`open_amount` double,
>`ordered_amount` double,
>`pay_on_code` string,
>`payment_term_name` string,
>`pending_signature_flag` string,
>`po_revision_num` double,
>`preparer_id` bigint,
>`price_update_tolerance` double,
>`print_count` double,
>`printed_date` date,
>`reply_date` date,
>`reply_method_lookup_code` string,
>`rfq_close_date` date,
>`segment2` string,
>`segment3` string,
>`segment4` string,
>`segment5` string,
>`shipping_control` string,
>`start_date` date,
>`start_date_active` date,
>`summary_flag` string,
>`supply_agreement_flag` string,
>`usd_amount_limit` double,
>`usd_blanket_total_amount` double,
>`usd_exchange_rate` double,
>`usd_open_amount` double,
>`usd_order_amount` double,
>`ussgl_transaction_code` string,
>`xml_flag` string,
>`purchasing_organization_id` bigint,
>`purchasing_group_code` string,
>`last_updated_by_name` string,
>`created_by_name` string,
>`incoterms_1` string,
>`incoterms_2` string,
>`ame_approval_id` double,
>`ame_transaction_type` string,
>`auto_sourcing_flag` string,
>`cat_admin_auth_enabled_flag` string,
>`clm_document_number` string,
>`comm_rev_num` double,
>`consigned_consumption_flag` string,
>`consume_req_demand_flag` string,
>`conterms_articles_upd_date` timestamp,
>`conterms_deliv_upd_date` timestamp,
>`conterms_exist_flag` string,
>`cpa_reference` double,
>`created_language` string,
>`email_address` string,
>`enable_all_sites` string,
>`fax` string,
>`lock_owner_role` string,
>`lock_owner_user_id` double,
>`min_release_amount` double,
>`mrc_rate` string,
>`mrc_rate_date` string,
>`mrc_rate_type` string,
>`otm_recovery_flag` string,
>`otm_status_code` string,
>`pay_when_paid` string,
>`pcard_id` bigint,
>`program_update_date` timestamp,
>`quotation_class_code` string,
>`quote_type_lookup_code` string,
>
[jira] [Updated] (HIVE-20833) package.jdo needs to be updated to conform with HIVE-20221 changes
[
https://issues.apache.org/jira/browse/HIVE-20833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20833:
---
Status: Patch Available (was: Open)
> package.jdo needs to be updated to conform with HIVE-20221 changes
> --
>
> Key: HIVE-20833
> URL: https://issues.apache.org/jira/browse/HIVE-20833
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20833.1.patch, HIVE-20833.2.patch
>
>
> Following test if run with TestMiniLlapLocalCliDriver will fail:
> {code:sql}
> CREATE TABLE `alterPartTbl`(
>`po_header_id` bigint,
>`vendor_num` string,
>`requester_name` string,
>`approver_name` string,
>`buyer_name` string,
>`preparer_name` string,
>`po_requisition_number` string,
>`po_requisition_id` bigint,
>`po_requisition_desc` string,
>`rate_type` string,
>`rate_date` date,
>`rate` double,
>`blanket_total_amount` double,
>`authorization_status` string,
>`revision_num` bigint,
>`revised_date` date,
>`approved_flag` string,
>`approved_date` timestamp,
>`amount_limit` double,
>`note_to_authorizer` string,
>`note_to_vendor` string,
>`note_to_receiver` string,
>`vendor_order_num` string,
>`comments` string,
>`acceptance_required_flag` string,
>`acceptance_due_date` date,
>`closed_date` timestamp,
>`user_hold_flag` string,
>`approval_required_flag` string,
>`cancel_flag` string,
>`firm_status_lookup_code` string,
>`firm_date` date,
>`frozen_flag` string,
>`closed_code` string,
>`org_id` bigint,
>`reference_num` string,
>`wf_item_type` string,
>`wf_item_key` string,
>`submit_date` date,
>`sap_company_code` string,
>`sap_fiscal_year` bigint,
>`po_number` string,
>`sap_line_item` bigint,
>`closed_status_flag` string,
>`balancing_segment` string,
>`cost_center_segment` string,
>`base_amount_limit` double,
>`base_blanket_total_amount` double,
>`base_open_amount` double,
>`base_ordered_amount` double,
>`cancel_date` timestamp,
>`cbc_accounting_date` date,
>`change_requested_by` string,
>`change_summary` string,
>`confirming_order_flag` string,
>`document_creation_method` string,
>`edi_processed_flag` string,
>`edi_processed_status` string,
>`enabled_flag` string,
>`encumbrance_required_flag` string,
>`end_date` date,
>`end_date_active` date,
>`from_header_id` bigint,
>`from_type_lookup_code` string,
>`global_agreement_flag` string,
>`government_context` string,
>`interface_source_code` string,
>`ledger_currency_code` string,
>`open_amount` double,
>`ordered_amount` double,
>`pay_on_code` string,
>`payment_term_name` string,
>`pending_signature_flag` string,
>`po_revision_num` double,
>`preparer_id` bigint,
>`price_update_tolerance` double,
>`print_count` double,
>`printed_date` date,
>`reply_date` date,
>`reply_method_lookup_code` string,
>`rfq_close_date` date,
>`segment2` string,
>`segment3` string,
>`segment4` string,
>`segment5` string,
>`shipping_control` string,
>`start_date` date,
>`start_date_active` date,
>`summary_flag` string,
>`supply_agreement_flag` string,
>`usd_amount_limit` double,
>`usd_blanket_total_amount` double,
>`usd_exchange_rate` double,
>`usd_open_amount` double,
>`usd_order_amount` double,
>`ussgl_transaction_code` string,
>`xml_flag` string,
>`purchasing_organization_id` bigint,
>`purchasing_group_code` string,
>`last_updated_by_name` string,
>`created_by_name` string,
>`incoterms_1` string,
>`incoterms_2` string,
>`ame_approval_id` double,
>`ame_transaction_type` string,
>`auto_sourcing_flag` string,
>`cat_admin_auth_enabled_flag` string,
>`clm_document_number` string,
>`comm_rev_num` double,
>`consigned_consumption_flag` string,
>`consume_req_demand_flag` string,
>`conterms_articles_upd_date` timestamp,
>`conterms_deliv_upd_date` timestamp,
>`conterms_exist_flag` string,
>`cpa_reference` double,
>`created_language` string,
>`email_address` string,
>`enable_all_sites` string,
>`fax` string,
>`lock_owner_role` string,
>`lock_owner_user_id` double,
>`min_release_amount` double,
>`mrc_rate` string,
>`mrc_rate_date` string,
>`mrc_rate_type` string,
>`otm_recovery_flag` string,
>`otm_status_code` string,
>`pay_when_paid` string,
>`pcard_id` bigint,
>`program_update_date` timestamp,
>`quotation_class_code` string,
>`quote_type_lookup_code` string,
>
[jira] [Assigned] (HIVE-20847) Review of NullScanCode
[
https://issues.apache.org/jira/browse/HIVE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR reassigned HIVE-20847:
--
Assignee: BELUGA BEHR
> Review of NullScanCode
> --
>
> Key: HIVE-20847
> URL: https://issues.apache.org/jira/browse/HIVE-20847
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20847.1.patch
>
>
> What got me looking at this class was the verboseness of some of the logging.
> I would like to request that we DEBUG the logging since this level of detail
> means nothing to a cluster admin.
> Also... this {{contains}} call would be better applied onto a {{HashSet}}
> instead of an {{ArrayList}}.
> {code:java|title=NullScanTaskDispatcher.java}
> private void processAlias(MapWork work, Path path, ArrayList
> aliasesAffected, ArrayList aliases) {
> // the aliases that are allowed to map to a null scan.
> ArrayList allowed = new ArrayList();
> for (String alias : aliasesAffected) {
> if (aliases.contains(alias)) {
> allowed.add(alias);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20847) Review of NullScanCode
[
https://issues.apache.org/jira/browse/HIVE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20847:
---
Flags: Patch
> Review of NullScanCode
> --
>
> Key: HIVE-20847
> URL: https://issues.apache.org/jira/browse/HIVE-20847
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20847.1.patch
>
>
> What got me looking at this class was the verboseness of some of the logging.
> I would like to request that we DEBUG the logging since this level of detail
> means nothing to a cluster admin.
> Also... this {{contains}} call would be better applied onto a {{HashSet}}
> instead of an {{ArrayList}}.
> {code:java|title=NullScanTaskDispatcher.java}
> private void processAlias(MapWork work, Path path, ArrayList
> aliasesAffected, ArrayList aliases) {
> // the aliases that are allowed to map to a null scan.
> ArrayList allowed = new ArrayList();
> for (String alias : aliasesAffected) {
> if (aliases.contains(alias)) {
> allowed.add(alias);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20847) Review of NullScanCode
[
https://issues.apache.org/jira/browse/HIVE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20847:
---
Status: Patch Available (was: Open)
> Review of NullScanCode
> --
>
> Key: HIVE-20847
> URL: https://issues.apache.org/jira/browse/HIVE-20847
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20847.1.patch
>
>
> What got me looking at this class was the verboseness of some of the logging.
> I would like to request that we DEBUG the logging since this level of detail
> means nothing to a cluster admin.
> Also... this {{contains}} call would be better applied onto a {{HashSet}}
> instead of an {{ArrayList}}.
> {code:java|title=NullScanTaskDispatcher.java}
> private void processAlias(MapWork work, Path path, ArrayList
> aliasesAffected, ArrayList aliases) {
> // the aliases that are allowed to map to a null scan.
> ArrayList allowed = new ArrayList();
> for (String alias : aliasesAffected) {
> if (aliases.contains(alias)) {
> allowed.add(alias);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20847) Review of NullScanCode
[
https://issues.apache.org/jira/browse/HIVE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20847:
---
Attachment: HIVE-20847.1.patch
> Review of NullScanCode
> --
>
> Key: HIVE-20847
> URL: https://issues.apache.org/jira/browse/HIVE-20847
> Project: Hive
> Issue Type: Improvement
> Components: Physical Optimizer
>Affects Versions: 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20847.1.patch
>
>
> What got me looking at this class was the verboseness of some of the logging.
> I would like to request that we DEBUG the logging since this level of detail
> means nothing to a cluster admin.
> Also... this {{contains}} call would be better applied onto a {{HashSet}}
> instead of an {{ArrayList}}.
> {code:java|title=NullScanTaskDispatcher.java}
> private void processAlias(MapWork work, Path path, ArrayList
> aliasesAffected, ArrayList aliases) {
> // the aliases that are allowed to map to a null scan.
> ArrayList allowed = new ArrayList();
> for (String alias : aliasesAffected) {
> if (aliases.contains(alias)) {
> allowed.add(alias);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20833) package.jdo needs to be updated to conform with HIVE-20221 changes
[
https://issues.apache.org/jira/browse/HIVE-20833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-20833:
---
Attachment: HIVE-20833.2.patch
> package.jdo needs to be updated to conform with HIVE-20221 changes
> --
>
> Key: HIVE-20833
> URL: https://issues.apache.org/jira/browse/HIVE-20833
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20833.1.patch, HIVE-20833.2.patch
>
>
> Following test if run with TestMiniLlapLocalCliDriver will fail:
> {code:sql}
> CREATE TABLE `alterPartTbl`(
>`po_header_id` bigint,
>`vendor_num` string,
>`requester_name` string,
>`approver_name` string,
>`buyer_name` string,
>`preparer_name` string,
>`po_requisition_number` string,
>`po_requisition_id` bigint,
>`po_requisition_desc` string,
>`rate_type` string,
>`rate_date` date,
>`rate` double,
>`blanket_total_amount` double,
>`authorization_status` string,
>`revision_num` bigint,
>`revised_date` date,
>`approved_flag` string,
>`approved_date` timestamp,
>`amount_limit` double,
>`note_to_authorizer` string,
>`note_to_vendor` string,
>`note_to_receiver` string,
>`vendor_order_num` string,
>`comments` string,
>`acceptance_required_flag` string,
>`acceptance_due_date` date,
>`closed_date` timestamp,
>`user_hold_flag` string,
>`approval_required_flag` string,
>`cancel_flag` string,
>`firm_status_lookup_code` string,
>`firm_date` date,
>`frozen_flag` string,
>`closed_code` string,
>`org_id` bigint,
>`reference_num` string,
>`wf_item_type` string,
>`wf_item_key` string,
>`submit_date` date,
>`sap_company_code` string,
>`sap_fiscal_year` bigint,
>`po_number` string,
>`sap_line_item` bigint,
>`closed_status_flag` string,
>`balancing_segment` string,
>`cost_center_segment` string,
>`base_amount_limit` double,
>`base_blanket_total_amount` double,
>`base_open_amount` double,
>`base_ordered_amount` double,
>`cancel_date` timestamp,
>`cbc_accounting_date` date,
>`change_requested_by` string,
>`change_summary` string,
>`confirming_order_flag` string,
>`document_creation_method` string,
>`edi_processed_flag` string,
>`edi_processed_status` string,
>`enabled_flag` string,
>`encumbrance_required_flag` string,
>`end_date` date,
>`end_date_active` date,
>`from_header_id` bigint,
>`from_type_lookup_code` string,
>`global_agreement_flag` string,
>`government_context` string,
>`interface_source_code` string,
>`ledger_currency_code` string,
>`open_amount` double,
>`ordered_amount` double,
>`pay_on_code` string,
>`payment_term_name` string,
>`pending_signature_flag` string,
>`po_revision_num` double,
>`preparer_id` bigint,
>`price_update_tolerance` double,
>`print_count` double,
>`printed_date` date,
>`reply_date` date,
>`reply_method_lookup_code` string,
>`rfq_close_date` date,
>`segment2` string,
>`segment3` string,
>`segment4` string,
>`segment5` string,
>`shipping_control` string,
>`start_date` date,
>`start_date_active` date,
>`summary_flag` string,
>`supply_agreement_flag` string,
>`usd_amount_limit` double,
>`usd_blanket_total_amount` double,
>`usd_exchange_rate` double,
>`usd_open_amount` double,
>`usd_order_amount` double,
>`ussgl_transaction_code` string,
>`xml_flag` string,
>`purchasing_organization_id` bigint,
>`purchasing_group_code` string,
>`last_updated_by_name` string,
>`created_by_name` string,
>`incoterms_1` string,
>`incoterms_2` string,
>`ame_approval_id` double,
>`ame_transaction_type` string,
>`auto_sourcing_flag` string,
>`cat_admin_auth_enabled_flag` string,
>`clm_document_number` string,
>`comm_rev_num` double,
>`consigned_consumption_flag` string,
>`consume_req_demand_flag` string,
>`conterms_articles_upd_date` timestamp,
>`conterms_deliv_upd_date` timestamp,
>`conterms_exist_flag` string,
>`cpa_reference` double,
>`created_language` string,
>`email_address` string,
>`enable_all_sites` string,
>`fax` string,
>`lock_owner_role` string,
>`lock_owner_user_id` double,
>`min_release_amount` double,
>`mrc_rate` string,
>`mrc_rate_date` string,
>`mrc_rate_type` string,
>`otm_recovery_flag` string,
>`otm_status_code` string,
>`pay_when_paid` string,
>`pcard_id` bigint,
>`program_update_date` timestamp,
>`quotation_class_code` string,
>`quote_type_lookup_code` string,
>
[jira] [Commented] (HIVE-20786) Maven Build Failed with group id is too big
[
https://issues.apache.org/jira/browse/HIVE-20786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670502#comment-16670502
]
Szehon Ho commented on HIVE-20786:
--
Hey Vihang for some reason when I just change it on packaging/pom.xml like that
it gives an error
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-assembly-plugin:2.3:single (assemble) on project
hive-packaging: Failed to create assembly: Error creating assembly archive bin:
posix is not a legal value for this attribute -> [Help 1]
I need to debug it further.
> Maven Build Failed with group id is too big
>
>
> Key: HIVE-20786
> URL: https://issues.apache.org/jira/browse/HIVE-20786
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
> Environment:
> OS: MacOS 10.13.6
> Java:
> {code}
> java version "1.8.0_192"
> Java(TM) SE Runtime Environment (build 1.8.0_192-b12)
> Java HotSpot(TM) 64-Bit Server VM (build 25.192-b12, mixed mode)
> {code}
> Maven:
> {code}
> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
> 2018-06-18T02:33:14+08:00)
> Maven home: /usr/local/Cellar/maven/3.5.4/libexec
> Java version: 1.8.0_192, vendor: Oracle Corporation, runtime:
> /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre
> Default locale: en_CN, platform encoding: UTF-8
> OS name: "mac os x", version: "10.13.6", arch: "x86_64", family: "mac"
> {code}
>
>
>Reporter: PENG Zhengshuai
>Assignee: Szehon Ho
>Priority: Major
> Labels: maven
> Attachments: HIVE-20786.patch, hive_build_error.log
>
>
> When executing
> {code}
> mvn clean install -DskipTests
> {code}
> Build Failed:
> {code}
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Hive Storage API 2.7.0-SNAPSHOT SUCCESS [ 5.299
> s]
> [INFO] Hive 4.0.0-SNAPSHOT SUCCESS [ 0.750
> s]
> [INFO] Hive Classifications ... SUCCESS [ 1.057
> s]
> [INFO] Hive Shims Common .. SUCCESS [ 3.882
> s]
> [INFO] Hive Shims 0.23 SUCCESS [ 5.020
> s]
> [INFO] Hive Shims Scheduler ... SUCCESS [ 2.587
> s]
> [INFO] Hive Shims . SUCCESS [ 2.038
> s]
> [INFO] Hive Common SUCCESS [ 6.921
> s]
> [INFO] Hive Service RPC ... SUCCESS [ 3.503
> s]
> [INFO] Hive Serde . SUCCESS [ 6.322
> s]
> [INFO] Hive Standalone Metastore .. FAILURE [ 0.557
> s]
> [INFO] Hive Standalone Metastore Common Code .. SKIPPED
> [INFO] Hive Metastore . SKIPPED
> [INFO] Hive Vector-Code-Gen Utilities . SKIPPED
> [INFO] Hive Llap Common ... SKIPPED
> [INFO] Hive Llap Client ... SKIPPED
> [INFO] Hive Llap Tez .. SKIPPED
> [INFO] Hive Spark Remote Client ... SKIPPED
> [INFO] Hive Metastore Server .. SKIPPED
> [INFO] Hive Query Language SKIPPED
> [INFO] Hive Llap Server ... SKIPPED
> [INFO] Hive Service ... SKIPPED
> [INFO] Hive Accumulo Handler .. SKIPPED
> [INFO] Hive JDBC .. SKIPPED
> [INFO] Hive Beeline ... SKIPPED
> [INFO] Hive CLI ... SKIPPED
> [INFO] Hive Contrib ... SKIPPED
> [INFO] Hive Druid Handler . SKIPPED
> [INFO] Hive HBase Handler . SKIPPED
> [INFO] Hive JDBC Handler .. SKIPPED
> [INFO] Hive HCatalog .. SKIPPED
> [INFO] Hive HCatalog Core . SKIPPED
> [INFO] Hive HCatalog Pig Adapter .. SKIPPED
> [INFO] Hive HCatalog Server Extensions SKIPPED
> [INFO] Hive HCatalog Webhcat Java Client .. SKIPPED
> [INFO] Hive HCatalog Webhcat .. SKIPPED
> [INFO] Hive HCatalog Streaming SKIPPED
> [INFO] Hive HPL/SQL ... SKIPPED
> [INFO] Hive Streaming . SKIPPED
> [INFO] Hive Llap External Client .. SKIPPED
> [INFO] Hive Shims Aggregator
[jira] [Commented] (HIVE-20843) RELY constraints on primary keys and foreign keys are not recognized
[ https://issues.apache.org/jira/browse/HIVE-20843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670498#comment-16670498 ] Vihang Karajgaonkar commented on HIVE-20843: The patch looks good to me, although I have limited understanding of this part of the code. Perhaps [~vgarg] or [~ashutoshc] could take a look too? This issue doesn't affect so the patch needs to be committed only in branch-2 and branch-3. Can you please provide a patch for those branches? The patch naming convention should be HIVE-20843.1-branch-3.patch > RELY constraints on primary keys and foreign keys are not recognized > > > Key: HIVE-20843 > URL: https://issues.apache.org/jira/browse/HIVE-20843 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1, 3.0.0 >Reporter: Anurag Mantripragada >Assignee: Anurag Mantripragada >Priority: Major > Attachments: HIVE-20843.1.patch > > > Hive doesn't recognize RELY constraints after > https://issues.apache.org/jira/browse/HIVE-13076. The issue is in > BaseSemanticAnalyzer.java where we assign RELY. > An unrelated patch fixed this issue in later versions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20617) Fix type of constants in IN expressions to have correct type
[
https://issues.apache.org/jira/browse/HIVE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-20617:
Attachment: HIVE-20617.15.patch
> Fix type of constants in IN expressions to have correct type
>
>
> Key: HIVE-20617
> URL: https://issues.apache.org/jira/browse/HIVE-20617
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20617.01.patch, HIVE-20617.02.patch,
> HIVE-20617.03.patch, HIVE-20617.05.patch, HIVE-20617.06.patch,
> HIVE-20617.07.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch,
> HIVE-20617.09.patch, HIVE-20617.10.patch, HIVE-20617.10.patch,
> HIVE-20617.11.patch, HIVE-20617.11.patch, HIVE-20617.12.patch,
> HIVE-20617.13.patch, HIVE-20617.14.patch, HIVE-20617.15.patch
>
>
> In statements like {{struct(a,b) IN (const struct('x','y'), ... )}} the
> comparision in UDFIn may fail because if a or b is of char/varchar type the
> constants will retain string type - especially after PointlookupOptimizer
> compaction.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20839) "Cannot find field" error during dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-20839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20839:
--
Status: Patch Available (was: In Progress)
> "Cannot find field" error during dynamically partitioned hash join
> --
>
> Key: HIVE-20839
> URL: https://issues.apache.org/jira/browse/HIVE-20839
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20839.1.patch, HIVE-20839.2.patch
>
>
> Occurs in some cases in the non-CBO optimized queries, either if CBO is
> disabled or has failed due to error.
> {noformat}
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.ReduceRecordProcessor: Hit error
> while closing operators - failing tree
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.TezProcessor:
> java.lang.RuntimeException: cannot find field _col304 from [0:_col0, 1:_col1,
> 2:_col2, 3:_col3, 4:_col4, 5:_col5, 6:_col6, 7:_col7, 8:_col8, 9:_col9,
> 10:_col10, 11:_col11, 12:_col12, 13:_col13, 14:_col15, 15:_col16, 16:_col17,
> 17:_col18, 18:_col19, 19:_col20, 20:_col21, 21:_col22, 22:_col23, 23:_col24,
> 24:_col25, 25:_col26, 26:_col27, 27:_col28, 28:_col29, 29:_col30, 30:_col31,
> 31:_col32, 32:_col33, 33:_col34, 34:_col35, 35:_col36, 36:_col37, 37:_col38,
> 38:_col39, 39:_col40, 40:_col41, 41:_col42, 42:_col43, 43:_col44, 44:_col45,
> 45:_col46, 46:_col47, 47:_col48, 48:_col49, 49:_col50, 50:_col51, 51:_col52,
> 52:_col53, 53:_col54, 54:_col55, 55:_col56, 56:_col57, 57:_col58, 58:_col59,
> 59:_col60, 60:_col61, 61:_col62, 62:_col63, 63:_col64, 64:_col65, 65:_col66,
> 66:_col67, 67:_col68, 68:_col70, 69:_col72, 70:_col73, 71:_col74, 72:_col75,
> 73:_col76, 74:_col77, 75:_col78, 76:_col79, 77:_col80, 78:_col81, 79:_col82,
> 80:_col83, 81:_col84, 82:_col85, 83:_col86, 84:_col87, 85:_col88, 86:_col89,
> 87:_col90, 88:_col91, 89:_col92, 90:_col93, 91:_col94, 92:_col95, 93:_col96,
> 94:_col97, 95:_col98, 96:_col99, 97:_col100, 98:_col101, 99:_col102,
> 100:_col103, 101:_col104, 102:_col105, 103:_col106, 104:_col107, 105:_col108,
> 106:_col109, 107:_col110, 108:_col111, 109:_col112, 110:_col113, 111:_col114,
> 112:_col115, 113:_col116, 114:_col117, 115:_col118, 116:_col119, 117:_col120,
> 118:_col121, 119:_col122, 120:_col123, 121:_col124, 122:_col125, 123:_col126,
> 124:_col127, 125:_col128, 126:_col129, 127:_col130, 128:_col131, 129:_col132,
> 130:_col133, 131:_col134, 132:_col135, 133:_col136, 134:_col137, 135:_col138,
> 136:_col139, 137:_col140, 138:_col141, 139:_col142, 140:_col143, 141:_col144,
> 142:_col145, 143:_col146, 144:_col147, 145:_col148, 146:_col149, 147:_col150,
> 148:_col151, 149:_col152, 150:_col153, 151:_col154, 152:_col155, 153:_col156,
> 154:_col157, 155:_col158, 156:_col159, 157:_col160, 158:_col161, 159:_col162,
> 160:_col163, 161:_col164, 162:_col165, 163:_col166, 164:_col167, 165:_col168,
> 166:_col169, 167:_col170, 168:_col171, 169:_col318]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:485)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:80)
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:74)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:144)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:374)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:188)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20839) "Cannot find field" error during dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-20839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670446#comment-16670446
]
Jason Dere commented on HIVE-20839:
---
Thanks [~vgarg]. Attaching new patch using .equals() rather than .isSame().
Also removed the test, will create a new Jira with the testcase once HIVE-20833
is done.
> "Cannot find field" error during dynamically partitioned hash join
> --
>
> Key: HIVE-20839
> URL: https://issues.apache.org/jira/browse/HIVE-20839
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20839.1.patch, HIVE-20839.2.patch
>
>
> Occurs in some cases in the non-CBO optimized queries, either if CBO is
> disabled or has failed due to error.
> {noformat}
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.ReduceRecordProcessor: Hit error
> while closing operators - failing tree
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.TezProcessor:
> java.lang.RuntimeException: cannot find field _col304 from [0:_col0, 1:_col1,
> 2:_col2, 3:_col3, 4:_col4, 5:_col5, 6:_col6, 7:_col7, 8:_col8, 9:_col9,
> 10:_col10, 11:_col11, 12:_col12, 13:_col13, 14:_col15, 15:_col16, 16:_col17,
> 17:_col18, 18:_col19, 19:_col20, 20:_col21, 21:_col22, 22:_col23, 23:_col24,
> 24:_col25, 25:_col26, 26:_col27, 27:_col28, 28:_col29, 29:_col30, 30:_col31,
> 31:_col32, 32:_col33, 33:_col34, 34:_col35, 35:_col36, 36:_col37, 37:_col38,
> 38:_col39, 39:_col40, 40:_col41, 41:_col42, 42:_col43, 43:_col44, 44:_col45,
> 45:_col46, 46:_col47, 47:_col48, 48:_col49, 49:_col50, 50:_col51, 51:_col52,
> 52:_col53, 53:_col54, 54:_col55, 55:_col56, 56:_col57, 57:_col58, 58:_col59,
> 59:_col60, 60:_col61, 61:_col62, 62:_col63, 63:_col64, 64:_col65, 65:_col66,
> 66:_col67, 67:_col68, 68:_col70, 69:_col72, 70:_col73, 71:_col74, 72:_col75,
> 73:_col76, 74:_col77, 75:_col78, 76:_col79, 77:_col80, 78:_col81, 79:_col82,
> 80:_col83, 81:_col84, 82:_col85, 83:_col86, 84:_col87, 85:_col88, 86:_col89,
> 87:_col90, 88:_col91, 89:_col92, 90:_col93, 91:_col94, 92:_col95, 93:_col96,
> 94:_col97, 95:_col98, 96:_col99, 97:_col100, 98:_col101, 99:_col102,
> 100:_col103, 101:_col104, 102:_col105, 103:_col106, 104:_col107, 105:_col108,
> 106:_col109, 107:_col110, 108:_col111, 109:_col112, 110:_col113, 111:_col114,
> 112:_col115, 113:_col116, 114:_col117, 115:_col118, 116:_col119, 117:_col120,
> 118:_col121, 119:_col122, 120:_col123, 121:_col124, 122:_col125, 123:_col126,
> 124:_col127, 125:_col128, 126:_col129, 127:_col130, 128:_col131, 129:_col132,
> 130:_col133, 131:_col134, 132:_col135, 133:_col136, 134:_col137, 135:_col138,
> 136:_col139, 137:_col140, 138:_col141, 139:_col142, 140:_col143, 141:_col144,
> 142:_col145, 143:_col146, 144:_col147, 145:_col148, 146:_col149, 147:_col150,
> 148:_col151, 149:_col152, 150:_col153, 151:_col154, 152:_col155, 153:_col156,
> 154:_col157, 155:_col158, 156:_col159, 157:_col160, 158:_col161, 159:_col162,
> 160:_col163, 161:_col164, 162:_col165, 163:_col166, 164:_col167, 165:_col168,
> 166:_col169, 167:_col170, 168:_col171, 169:_col318]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:485)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:80)
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:74)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:144)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:374)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:188)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
>
[jira] [Updated] (HIVE-20839) "Cannot find field" error during dynamically partitioned hash join
[
https://issues.apache.org/jira/browse/HIVE-20839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-20839:
--
Attachment: HIVE-20839.2.patch
> "Cannot find field" error during dynamically partitioned hash join
> --
>
> Key: HIVE-20839
> URL: https://issues.apache.org/jira/browse/HIVE-20839
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20839.1.patch, HIVE-20839.2.patch
>
>
> Occurs in some cases in the non-CBO optimized queries, either if CBO is
> disabled or has failed due to error.
> {noformat}
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.ReduceRecordProcessor: Hit error
> while closing operators - failing tree
> 2018-10-11T04:40:22,724 ERROR [TezTR-85144_8944_1085_28_996_2
> (1539092085144_8944_1085_28_000996_2)] tez.TezProcessor:
> java.lang.RuntimeException: cannot find field _col304 from [0:_col0, 1:_col1,
> 2:_col2, 3:_col3, 4:_col4, 5:_col5, 6:_col6, 7:_col7, 8:_col8, 9:_col9,
> 10:_col10, 11:_col11, 12:_col12, 13:_col13, 14:_col15, 15:_col16, 16:_col17,
> 17:_col18, 18:_col19, 19:_col20, 20:_col21, 21:_col22, 22:_col23, 23:_col24,
> 24:_col25, 25:_col26, 26:_col27, 27:_col28, 28:_col29, 29:_col30, 30:_col31,
> 31:_col32, 32:_col33, 33:_col34, 34:_col35, 35:_col36, 36:_col37, 37:_col38,
> 38:_col39, 39:_col40, 40:_col41, 41:_col42, 42:_col43, 43:_col44, 44:_col45,
> 45:_col46, 46:_col47, 47:_col48, 48:_col49, 49:_col50, 50:_col51, 51:_col52,
> 52:_col53, 53:_col54, 54:_col55, 55:_col56, 56:_col57, 57:_col58, 58:_col59,
> 59:_col60, 60:_col61, 61:_col62, 62:_col63, 63:_col64, 64:_col65, 65:_col66,
> 66:_col67, 67:_col68, 68:_col70, 69:_col72, 70:_col73, 71:_col74, 72:_col75,
> 73:_col76, 74:_col77, 75:_col78, 76:_col79, 77:_col80, 78:_col81, 79:_col82,
> 80:_col83, 81:_col84, 82:_col85, 83:_col86, 84:_col87, 85:_col88, 86:_col89,
> 87:_col90, 88:_col91, 89:_col92, 90:_col93, 91:_col94, 92:_col95, 93:_col96,
> 94:_col97, 95:_col98, 96:_col99, 97:_col100, 98:_col101, 99:_col102,
> 100:_col103, 101:_col104, 102:_col105, 103:_col106, 104:_col107, 105:_col108,
> 106:_col109, 107:_col110, 108:_col111, 109:_col112, 110:_col113, 111:_col114,
> 112:_col115, 113:_col116, 114:_col117, 115:_col118, 116:_col119, 117:_col120,
> 118:_col121, 119:_col122, 120:_col123, 121:_col124, 122:_col125, 123:_col126,
> 124:_col127, 125:_col128, 126:_col129, 127:_col130, 128:_col131, 129:_col132,
> 130:_col133, 131:_col134, 132:_col135, 133:_col136, 134:_col137, 135:_col138,
> 136:_col139, 137:_col140, 138:_col141, 139:_col142, 140:_col143, 141:_col144,
> 142:_col145, 143:_col146, 144:_col147, 145:_col148, 146:_col149, 147:_col150,
> 148:_col151, 149:_col152, 150:_col153, 151:_col154, 152:_col155, 153:_col156,
> 154:_col157, 155:_col158, 156:_col159, 157:_col160, 158:_col161, 159:_col162,
> 160:_col163, 161:_col164, 162:_col165, 163:_col166, 164:_col167, 165:_col168,
> 166:_col169, 167:_col170, 168:_col171, 169:_col318]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:485)
> at
> org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:153)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:80)
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapJoinOperator.initializeOp(AbstractMapJoinOperator.java:74)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:144)
> at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:374)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.init(ReduceRecordProcessor.java:195)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:188)
> at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:172)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19026) Configurable serde for druid kafka indexing
[ https://issues.apache.org/jira/browse/HIVE-19026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nishant Bangarwa updated HIVE-19026: Attachment: HIVE-19026.2.patch > Configurable serde for druid kafka indexing > > > Key: HIVE-19026 > URL: https://issues.apache.org/jira/browse/HIVE-19026 > Project: Hive > Issue Type: Task >Reporter: Nishant Bangarwa >Assignee: Nishant Bangarwa >Priority: Major > Attachments: HIVE-19026.1.patch, HIVE-19026.2.patch, HIVE-19026.patch > > > https://issues.apache.org/jira/browse/HIVE-18976 introduces support for > setting up druid kafka-indexing service. > Input serialization should be configurable. for now we can say we only > support json, but there should be a mechanism to support other formats. > Perhaps, we can make use of Hive's serde library like LazySimpleSerde etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20751) Upgrade arrow version to 0.10.0
[ https://issues.apache.org/jira/browse/HIVE-20751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-20751: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Upgrade arrow version to 0.10.0 > --- > > Key: HIVE-20751 > URL: https://issues.apache.org/jira/browse/HIVE-20751 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.0 >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20751.1.patch, HIVE-20751.2.patch > > > Need to upgrade arrow version as spark is moving to arrow version 0.10.0 in > it's upcoming release 2.4.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20845) Fix TestJdbcWithDBTokenStoreNoDoAs flakiness
[ https://issues.apache.org/jira/browse/HIVE-20845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary reassigned HIVE-20845: - > Fix TestJdbcWithDBTokenStoreNoDoAs flakiness > > > Key: HIVE-20845 > URL: https://issues.apache.org/jira/browse/HIVE-20845 > Project: Hive > Issue Type: Test >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > > Previously did a dirty fix for TestJdbcWithDBTokenStoreNoDoAs and > TestJdbcWithDBTokenStore > Found out the issue is that we do not wait enough for HS2 to come up. > Need to fix in MiniHS2.waitForStartup() -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20233) Review Operator.java
[
https://issues.apache.org/jira/browse/HIVE-20233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20233:
---
Status: Patch Available (was: Open)
> Review Operator.java
>
>
> Key: HIVE-20233
> URL: https://issues.apache.org/jira/browse/HIVE-20233
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20233.1.patch, HIVE-20233.2.patch,
> HIVE-20233.3.patch
>
>
> Various improvements to {{Operator.java}}
>
> * Code formatting
> * Remove superfluous logging guards
> * Add logging message parameters
> * Minor performance improvements
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20233) Review Operator.java
[
https://issues.apache.org/jira/browse/HIVE-20233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20233:
---
Attachment: HIVE-20233.3.patch
> Review Operator.java
>
>
> Key: HIVE-20233
> URL: https://issues.apache.org/jira/browse/HIVE-20233
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20233.1.patch, HIVE-20233.2.patch,
> HIVE-20233.3.patch
>
>
> Various improvements to {{Operator.java}}
>
> * Code formatting
> * Remove superfluous logging guards
> * Add logging message parameters
> * Minor performance improvements
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20233) Review Operator.java
[
https://issues.apache.org/jira/browse/HIVE-20233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20233:
---
Status: Open (was: Patch Available)
> Review Operator.java
>
>
> Key: HIVE-20233
> URL: https://issues.apache.org/jira/browse/HIVE-20233
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 3.0.0, 4.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-20233.1.patch, HIVE-20233.2.patch,
> HIVE-20233.3.patch
>
>
> Various improvements to {{Operator.java}}
>
> * Code formatting
> * Remove superfluous logging guards
> * Add logging message parameters
> * Minor performance improvements
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20751) Upgrade arrow version to 0.10.0
[ https://issues.apache.org/jira/browse/HIVE-20751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670227#comment-16670227 ] Shubham Chaurasia commented on HIVE-20751: -- Done. Thanks :) > Upgrade arrow version to 0.10.0 > --- > > Key: HIVE-20751 > URL: https://issues.apache.org/jira/browse/HIVE-20751 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.0 >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20751.1.patch, HIVE-20751.2.patch > > > Need to upgrade arrow version as spark is moving to arrow version 0.10.0 in > it's upcoming release 2.4.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20751) Upgrade arrow version to 0.10.0
[ https://issues.apache.org/jira/browse/HIVE-20751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shubham Chaurasia updated HIVE-20751: - Attachment: HIVE-20751.2.patch > Upgrade arrow version to 0.10.0 > --- > > Key: HIVE-20751 > URL: https://issues.apache.org/jira/browse/HIVE-20751 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.0 >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20751.1.patch, HIVE-20751.2.patch > > > Need to upgrade arrow version as spark is moving to arrow version 0.10.0 in > it's upcoming release 2.4.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20751) Upgrade arrow version to 0.10.0
[ https://issues.apache.org/jira/browse/HIVE-20751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670187#comment-16670187 ] Teddy Choi commented on HIVE-20751: --- Pushed to master. Thanks! Also please update the patch file as I sent. > Upgrade arrow version to 0.10.0 > --- > > Key: HIVE-20751 > URL: https://issues.apache.org/jira/browse/HIVE-20751 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.1.0 >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20751.1.patch > > > Need to upgrade arrow version as spark is moving to arrow version 0.10.0 in > it's upcoming release 2.4.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20844) Cache Instances of LockManager in DummyTxnManager
[
https://issues.apache.org/jira/browse/HIVE-20844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
BELUGA BEHR updated HIVE-20844:
---
Summary: Cache Instances of LockManager in DummyTxnManager (was: Cache
Instances of CacheManager in DummyTxnManager)
> Cache Instances of LockManager in DummyTxnManager
> -
>
> Key: HIVE-20844
> URL: https://issues.apache.org/jira/browse/HIVE-20844
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, Locking
>Affects Versions: 2.3.2, 3.1.0, 4.0.0
>Reporter: BELUGA BEHR
>Priority: Major
>
> I noticed that the {{DummyTxnManager}} class instantiates quite a few
> instances of {{ZooKeeperHiveLockManager}}. The ZooKeeper LM creates a
> connection to ZK for each instance created. It also does some initialization
> steps that are almost always just noise and pressure on ZooKeeper because it
> has already been initialized and the steps are therefore NOOPs.
> {{ZooKeeperHiveLockManager}} should be a singleton class with one long-lived
> connection to the ZooKeeper service. Perhaps the {{HiveLockManager}}
> interface could have a {{isSingleton()}} method which indicates that the LM
> should only be instantiated once and cached for subsequent sessions.
>
> {code:java}
> 2018-05-14 22:45:30,574 INFO
> org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager:
> [HiveServer2-Background-Pool: Thread-1252389]: Creating lock manager of type
> org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 22:51:27,865 INFO
> org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager:
> [HiveServer2-Background-Pool: Thread-1252671]: Creating lock manager of type
> org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 22:51:37,552 INFO
> org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager:
> [HiveServer2-Background-Pool: Thread-1252686]: Creating lock manager of type
> org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 22:51:49,046 INFO
> org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager:
> [HiveServer2-Background-Pool: Thread-1252736]: Creating lock manager of type
> org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 22:51:50,664 INFO
> org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager:
> [HiveServer2-Background-Pool: Thread-1252742]: Creating lock manager of type
> org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 23:00:54,314 INFO
> org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager:
> [HiveServer2-Background-Pool: Thread-1253479]: Creating lock manager of type
> org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 23:17:26,867 INFO
> org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager:
> [HiveServer2-Background-Pool: Thread-1254180]: Creating lock manager of type
> org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> 2018-05-14 23:24:25,426 INFO
> org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager:
> [HiveServer2-Background-Pool: Thread-1255493]: Creating lock manager of type
> org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager
> {code}
> {code:java|title=DummyTxnManager.java}
> @Override
> public HiveLockManager getLockManager() throws LockException {
> if (lockMgr == null) {
> boolean supportConcurrency =
> conf.getBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY);
> if (supportConcurrency) {
> String lockMgrName =
> conf.getVar(HiveConf.ConfVars.HIVE_LOCK_MANAGER);
> if ((lockMgrName == null) || (lockMgrName.isEmpty())) {
> throw new LockException(ErrorMsg.LOCKMGR_NOT_SPECIFIED.getMsg());
> }
> try {
> // CACHE LM HERE
> LOG.info("Creating lock manager of type " + lockMgrName);
> lockMgr = (HiveLockManager)ReflectionUtils.newInstance(
> conf.getClassByName(lockMgrName), conf);
> lockManagerCtx = new HiveLockManagerCtx(conf);
> lockMgr.setContext(lockManagerCtx);
> } catch (Exception e) {
> ...
> {code}
> [https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockManager.java]
> {code:java|title=ZooKeeperHiveLockManager Initialization}
> try {
> curatorFramework = CuratorFrameworkSingleton.getInstance(conf);
> parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE);
> try{
> curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/"
> + parent, new byte[0]);
> } catch (Exception e) {
> // ignore if the parent already exists
> if (!(e instanceof KeeperException) || ((KeeperException)e).code() !=
> KeeperException.Code.NODEEXISTS) {
>
[jira] [Commented] (HIVE-20304) When hive.optimize.skewjoin and hive.auto.convert.join are both set to true, and the execution engine is mr, same stage may launch twice due to the wrong generated plan
[
https://issues.apache.org/jira/browse/HIVE-20304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670153#comment-16670153
]
Yongzhi Chen commented on HIVE-20304:
-
[~BIGrey], could you attach a new patch to make the pre-commit build pass?
Thanks
> When hive.optimize.skewjoin and hive.auto.convert.join are both set to true,
> and the execution engine is mr, same stage may launch twice due to the wrong
> generated plan
>
>
> Key: HIVE-20304
> URL: https://issues.apache.org/jira/browse/HIVE-20304
> Project: Hive
> Issue Type: Bug
> Components: CLI
>Affects Versions: 1.2.1, 2.3.3
>Reporter: Hui Huang
>Assignee: Hui Huang
>Priority: Major
> Fix For: 1.2.1
>
> Attachments: HIVE-20304.1.patch, HIVE-20304.patch
>
>
> `When hive.optimize.skewjoin and hive.auto.convert.join are both set to true,
> and the execution engine is set to mr, same stage of a query may launch twice
> due to the wrong generated plan. If hive.exec.parallel is also true, the same
> stage will launch at the same time and the job will failed due to the first
> completed stage clear the map.xml/reduce.xml file stored in the hdfs.
> use following sql to reproduce the issue:
> {code:java}
> CREATE TABLE `tbl1`(
> `fence` string);
> CREATE TABLE `tbl2`(
> `order_id` string,
> `phone` string,
> `search_id` string
> )
> PARTITIONED BY (
> `dt` string);
> CREATE TABLE `tbl3`(
> `order_id` string,
> `platform` string)
> PARTITIONED BY (
> `dt` string);
> CREATE TABLE `tbl4`(
> `groupname` string,
> `phone` string)
> PARTITIONED BY (
> `dt` string);
> CREATE TABLE `tbl5`(
> `search_id` string,
> `fence` string)
> PARTITIONED BY (
> `dt` string);
> SET hive.exec.parallel = TRUE;
> SET hive.auto.convert.join = TRUE;
> SET hive.optimize.skewjoin = TRUE;
> SELECT dt,
>platform,
>groupname,
>count(1) as cnt
> FROM
> (SELECT dt,
> platform,
> groupname
> FROM
> (SELECT fence
> FROM tbl1)ta
>JOIN
>(SELECT a0.dt,
>a1.platform,
>a2.groupname,
>a3.fence
> FROM
> (SELECT dt,
> order_id,
> phone,
> search_id
> FROM tbl2
> WHERE dt =20180703 )a0
> JOIN
> (SELECT order_id,
> platform,
> dt
>FROM tbl3
>WHERE dt =20180703 )a1 ON a0.order_id = a1.order_id
> INNER JOIN
> (SELECT groupname,
> phone,
> dt
>FROM tbl4
>WHERE dt =20180703 )a2 ON a0.phone = a2.phone
> LEFT JOIN
> (SELECT search_id,
> fence,
> dt
>FROM tbl5
>WHERE dt =20180703)a3 ON a0.search_id = a3.search_id)t0 ON
> ta.fence = t0.fence)t11
> GROUP BY dt,
> platform,
> groupname;
> DROP TABLE tbl1;
> DROP TABLE tbl2;
> DROP TABLE tbl3;
> DROP TABLE tbl4;
> DROP TABLE tbl5;
> {code}
> We will get some error message like this:
> Examining task ID: task_1531284442065_3637_m_00 (and more) from job
> job_1531284442065_3637
> Task with the most failures(4):
>
> Task ID:
> task_1531284442065_3637_m_00
> URL:
>
> [http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1531284442065_3637=task_1531284442065_3637_m_00]
>
> Diagnostic Messages for this Task:
> File does not exist:
> hdfs://test/tmp/hive-hadoop/hadoop/fe5efa94-abb1-420f-b6ba-ec782e7b79ad/hive_2018-08-03_17-00-17_707_592882314975289971-5/-mr-10045/757eb1f7-7a37-4a7e-abc0-4a3b8b06510c/reduce.xml
> java.io.FileNotFoundException: File does not exist:
> hdfs://test/tmp/hive-hadoop/hadoop/fe5efa94-abb1-420f-b6ba-ec782e7b79ad/hive_2018-08-03_17-00-17_707_592882314975289971-5/-mr-10045/757eb1f7-7a37-4a7e-abc0-4a3b8b06510c/reduce.xml
> Looking into the plan by executing explain, I found that the Stage-4 and
> Stage-5 can reached from multi root tasks.
> {code:java}
> Explain
> STAGE DEPENDENCIES:
> Stage-21 is a root stage , consists of Stage-34, Stage-5
> Stage-34 has a backup stage: Stage-5
> Stage-20 depends on stages: Stage-34
> Stage-17 depends on stages: Stage-5, Stage-18, Stage-20 , consists of
> Stage-32, Stage-33, Stage-1
> Stage-32 has a backup stage: Stage-1
> Stage-15 depends on stages: Stage-32
> Stage-10 depends on stages: Stage-1, Stage-15, Stage-16 , consists of
[jira] [Comment Edited] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.
[
https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665492#comment-16665492
]
Sankar Hariappan edited comment on HIVE-20682 at 10/31/18 7:03 AM:
---
[~pvary], Thanks for your thoughts!
* H1 is sessionHive object and will be preserved in session object through out
the life time of session. HMS connection of sessionHive will be closed only
when we close the session.
* H2 is local to given thread and will be closed when we set sessionHive again
in next query. ThreadLocalHive.set() method closes the previous thread local
Hive object H2 before overwriting it.
* Also, if an async thread reallocates Hive object (let's say H3), then it
will be closed by Hive.closeCurrent call when the thread exits.
* So, I don't think there would be any HMS connection leak here.
* parentHive is always sessionHive object where allowClose flag is false. So,
the "assert (!parentHive.allowClose());" will never fail.
But I agree, there is a chance that each and every query re-creates Hive object
if the HMS relevant configs are changed in between queries. I missed this part
where sessionConf would be changed when user sets any Hive configurations via
cli.
So, I think, the earlier thread reference count solution would solve this
issue. [~maheshk114], please comment if you think otherwise.
cc [~daijy], [~thejas]
was (Author: sankarh):
[~pvary], Thanks for your thoughts!
* H1 is sessionHive object and will be preserved in session object through out
the life time of session. HMS connection of sessionHive will be closed only
when we close the session.
* H2 is local to given thread and will be closed when we set sessionHive again
in next query. ThreadLocalHive.set() method closes the previous thread local
Hive object H2 before overwriting it.
* Also, if an async thread reallocates Hive object (let's say H3), then it
will be closed by Hive.closeCurrent call when the thread exits.
* So, I don't think there would be any HMS connection leak here.
* parentHive is always sessionHive object where allowClose flag is false. So,
the "assert (!parentHive.allowClose());" will never fail.
But I agree, there is a chance that each and every query re-creates Hive object
if the HMS relevant configs are changed in between queries. I missed this part
where sessionConf would be changed when user sets any Hive configurations via
cli.
So, I think, the earlier thread reference count solution would solve this
issue. [~maheshk114], please comment if you think otherwise.
cc [~daijy], [~thejas], [~maheshk114]
> Async query execution can potentially fail if shared sessionHive is closed by
> master thread.
>
>
> Key: HIVE-20682
> URL: https://issues.apache.org/jira/browse/HIVE-20682
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20682.01.patch, HIVE-20682.02.patch,
> HIVE-20682.03.patch, HIVE-20682.04.patch
>
>
> *Problem description:*
> The master thread initializes the *sessionHive* object in *HiveSessionImpl*
> class when we open a new session for a client connection and by default all
> queries from this connection shares the same sessionHive object.
> If the master thread executes a *synchronous* query, it closes the
> sessionHive object (referred via thread local hiveDb) if
> {{Hive.isCompatible}} returns false and sets new Hive object in thread local
> HiveDb but doesn't change the sessionHive object in the session. Whereas,
> *asynchronous* query execution via async threads never closes the sessionHive
> object and it just creates a new one if needed and sets it as their thread
> local hiveDb.
> So, the problem can happen in the case where an *asynchronous* query is being
> executed by async threads refers to sessionHive object and the master thread
> receives a *synchronous* query that closes the same sessionHive object.
> Also, each query execution overwrites the thread local hiveDb object to
> sessionHive object which potentially leaks a metastore connection if the
> previous synchronous query execution re-created the Hive object.
> *Possible Fix:*
> The *sessionHive* object could be shared my multiple threads and so it
> shouldn't be allowed to be closed by any query execution threads when they
> re-create the Hive object due to changes in Hive configurations. But the Hive
> objects created by query execution threads should be closed when the thread
> exits.
> So, it is proposed to have an *isAllowClose* flag (default: *true*) in Hive
> object which should be set to
[jira] [Comment Edited] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.
[
https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665492#comment-16665492
]
Sankar Hariappan edited comment on HIVE-20682 at 10/31/18 7:02 AM:
---
[~pvary], Thanks for your thoughts!
* H1 is sessionHive object and will be preserved in session object through out
the life time of session. HMS connection of sessionHive will be closed only
when we close the session.
* H2 is local to given thread and will be closed when we set sessionHive again
in next query. ThreadLocalHive.set() method closes the previous thread local
Hive object H2 before overwriting it.
* Also, if an async thread reallocates Hive object (let's say H3), then it
will be closed by Hive.closeCurrent call when the thread exits.
* So, I don't think there would be any HMS connection leak here.
* parentHive is always sessionHive object where allowClose flag is false. So,
the "assert (!parentHive.allowClose());" will never fail.
But I agree, there is a chance that each and every query re-creates Hive object
if the HMS relevant configs are changed in between queries. I missed this part
where sessionConf would be changed when user sets any Hive configurations via
cli.
So, I think, the earlier thread reference count solution would solve this
issue. [~maheshk114], please comment if you think otherwise.
cc [~daijy], [~thejas], [~maheshk114]
was (Author: sankarh):
[~pvary], Thanks for your thoughts!
* H1 is sessionHive object and will be preserved in session object through out
the life time of session. HMS connection of sessionHive will be closed only
when we close the session.
* H2 is local to given thread and will be closed when we set sessionHive again
in next query. ThreadLocalHive.set() method closes the previous thread local
Hive object H2 before overwriting it.
* Also, if an async thread reallocates Hive object (let's say H3), then it
will be closed by Hive.closeCurrent call when the thread exits.
* So, I don't think there would be any HMS connection leak here.
* parentHive is always sessionHive object where allowClose flag is false. So,
the "assert (!parentHive.allowClose());" will never fail.
But I agree, there is a chance that each and every query re-creates Hive object
if the HMS relevant configs are changed in between queries. I missed this part
where sessionConf would be changed when user sets any Hive configurations via
cli.
So, I think, the earlier thread reference count solution would solve this
issue. [~maheshk114], please comment if you think otherwise.
Btw, there is one another issue where Hive.get(sessionConf) directly stores the
reference to sessionConf in Hive object which makes isCompatible() method to
return true always.
{code:java}
private static Hive getInternal(HiveConf c, boolean needsRefresh, boolean
isFastCheck,
boolean doRegisterAllFns) throws HiveException {
Hive db = hiveDB.get();
if (db == null || !db.isCurrentUserOwner() || needsRefresh
|| (c != null && !isCompatible(db, c, isFastCheck))) {
if (db != null) {
LOG.debug("Creating new db. db = " + db + ", needsRefresh = " +
needsRefresh +
", db.isCurrentUserOwner = " + db.isCurrentUserOwner());
closeCurrent();
}
db = create(c, doRegisterAllFns);
}
if (c != null) {
db.conf = c;
}
return db;
}{code}
So, as of today, Thread local Hive object never gets recreated even if we
change any HMS configs. Only REPL commands re-create Hive object as it uses
different HiveConf object not sessionConf to set HMS configs.
This issue can be resolved by always cloning HiveConf object instead of
storing the reference to input HiveConf.
But, it will be problem for Hive.getWithFastCheck(conf). I think, this method
is added as an optimisation to avoid checking individual HMS configs. It only
checks if the input conf object is same as the one stored in Hive object and if
not, then re-create Hive object.
So, this needs to be fixed carefully.
Please share your thoughts on this.
cc [~daijy], [~thejas], [~maheshk114]
> Async query execution can potentially fail if shared sessionHive is closed by
> master thread.
>
>
> Key: HIVE-20682
> URL: https://issues.apache.org/jira/browse/HIVE-20682
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20682.01.patch, HIVE-20682.02.patch,
> HIVE-20682.03.patch, HIVE-20682.04.patch
>
>
> *Problem description:*
> The master thread initializes the *sessionHive* object in *HiveSessionImpl*
> class when we open a
[jira] [Comment Edited] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.
[
https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669666#comment-16669666
]
Sankar Hariappan edited comment on HIVE-20682 at 10/31/18 6:59 AM:
---
[~maheshk114]
I think there is misunderstanding. There won't be any impact in current usage.
* Reference count of Hive object should be incremented only when some thread
sets it in Thread local. Not for every Hive.get() call.
* It will be decremented when the thread removes it from Thread local. If the
reference count is 0, then current thread will close the MS connection as well.
* If the Thread local is forcefully overwritten without removing the previous
Thread local properly, then the set() method ensures the previous one is
removed gracefully before overwriting with new one.
* So, just like current usage, it is expected to close the Hive object only
once within a thread after setting it in thread local.
Regarding, changes in sessionConf, we need to do the following.
* Reset sessionHive object using new sessionConf. But, the MS connection in
sessionHive object should be closed based on reference count as it might be
referred by some other async thread. All these methods are synchronised and so
no race conditions possible.
* If the previous query execution itself has recreated Thread local due to
config change, then sessionHive should be reset to previous Thread local Hive
itself instead of re-allocating.
Let me update the patch with this change and please review it.
cc [~pvary], [~daijy]
was (Author: sankarh):
[~maheshk114]
I think there is misunderstanding. There won't be any impact in current usage.
* Reference count of Hive object should be incremented only when some thread
sets it in Thread local. Not for every Hive.get() call.
* It will be decremented when the thread removes it from Thread local. If the
reference count is 0, then current thread will close the MS connection as well.
* If the Thread local is forcefully overwritten without removing the previous
Thread local properly, then the set() method ensures the previous one is
removed gracefully before overwriting with new one.
* So, just like current usage, it is expected to close the Hive object only
once within a thread after setting it in thread local.
Regarding, changes in sessionConf, we need to do the following.
* Reset sessionHive object using new sessionConf. But, the MS connection in
sessionHive object should be closed based on reference count as it might be
referred by some other async thread. All these methods are synchronised and so
no race conditions possible.
* Currently, sessionHive stores the sessionConf object reference and the same
sessionConf is updated for any configurations set within the session. This
makes isCompatible() to return true always even if there is MS related config
changes. So, to fix this, we need to recreate sessionConf object when there is
a set command.
Let me update the patch with this change and please review it.
cc [~pvary], [~daijy]
> Async query execution can potentially fail if shared sessionHive is closed by
> master thread.
>
>
> Key: HIVE-20682
> URL: https://issues.apache.org/jira/browse/HIVE-20682
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20682.01.patch, HIVE-20682.02.patch,
> HIVE-20682.03.patch, HIVE-20682.04.patch
>
>
> *Problem description:*
> The master thread initializes the *sessionHive* object in *HiveSessionImpl*
> class when we open a new session for a client connection and by default all
> queries from this connection shares the same sessionHive object.
> If the master thread executes a *synchronous* query, it closes the
> sessionHive object (referred via thread local hiveDb) if
> {{Hive.isCompatible}} returns false and sets new Hive object in thread local
> HiveDb but doesn't change the sessionHive object in the session. Whereas,
> *asynchronous* query execution via async threads never closes the sessionHive
> object and it just creates a new one if needed and sets it as their thread
> local hiveDb.
> So, the problem can happen in the case where an *asynchronous* query is being
> executed by async threads refers to sessionHive object and the master thread
> receives a *synchronous* query that closes the same sessionHive object.
> Also, each query execution overwrites the thread local hiveDb object to
> sessionHive object which potentially leaks a metastore connection if the
> previous synchronous query execution re-created the Hive object.
>
[jira] [Commented] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.
[
https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669666#comment-16669666
]
Sankar Hariappan commented on HIVE-20682:
-
[~maheshk114]
I think there is misunderstanding. There won't be any impact in current usage.
* Reference count of Hive object should be incremented only when some thread
sets it in Thread local. Not for every Hive.get() call.
* It will be decremented when the thread removes it from Thread local. If the
reference count is 0, then current thread will close the MS connection as well.
* If the Thread local is forcefully overwritten without removing the previous
Thread local properly, then the set() method ensures the previous one is
removed gracefully before overwriting with new one.
* So, just like current usage, it is expected to close the Hive object only
once within a thread after setting it in thread local.
Regarding, changes in sessionConf, we need to do the following.
* Reset sessionHive object using new sessionConf. But, the MS connection in
sessionHive object should be closed based on reference count as it might be
referred by some other async thread. All these methods are synchronised and so
no race conditions possible.
* Currently, sessionHive stores the sessionConf object reference and the same
sessionConf is updated for any configurations set within the session. This
makes isCompatible() to return true always even if there is MS related config
changes. So, to fix this, we need to recreate sessionConf object when there is
a set command.
Let me update the patch with this change and please review it.
cc [~pvary], [~daijy]
> Async query execution can potentially fail if shared sessionHive is closed by
> master thread.
>
>
> Key: HIVE-20682
> URL: https://issues.apache.org/jira/browse/HIVE-20682
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.1.0, 4.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-20682.01.patch, HIVE-20682.02.patch,
> HIVE-20682.03.patch, HIVE-20682.04.patch
>
>
> *Problem description:*
> The master thread initializes the *sessionHive* object in *HiveSessionImpl*
> class when we open a new session for a client connection and by default all
> queries from this connection shares the same sessionHive object.
> If the master thread executes a *synchronous* query, it closes the
> sessionHive object (referred via thread local hiveDb) if
> {{Hive.isCompatible}} returns false and sets new Hive object in thread local
> HiveDb but doesn't change the sessionHive object in the session. Whereas,
> *asynchronous* query execution via async threads never closes the sessionHive
> object and it just creates a new one if needed and sets it as their thread
> local hiveDb.
> So, the problem can happen in the case where an *asynchronous* query is being
> executed by async threads refers to sessionHive object and the master thread
> receives a *synchronous* query that closes the same sessionHive object.
> Also, each query execution overwrites the thread local hiveDb object to
> sessionHive object which potentially leaks a metastore connection if the
> previous synchronous query execution re-created the Hive object.
> *Possible Fix:*
> The *sessionHive* object could be shared my multiple threads and so it
> shouldn't be allowed to be closed by any query execution threads when they
> re-create the Hive object due to changes in Hive configurations. But the Hive
> objects created by query execution threads should be closed when the thread
> exits.
> So, it is proposed to have an *isAllowClose* flag (default: *true*) in Hive
> object which should be set to *false* for *sessionHive* and would be
> forcefully closed when the session is closed or released.
> cc [~pvary]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
