[jira] [Commented] (HIVE-18846) Query results cache: Allow queries to refer to the pending results of a query that has not finished yet
[ https://issues.apache.org/jira/browse/HIVE-18846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16401184#comment-16401184 ] Jason Dere commented on HIVE-18846: --- Minor update to patch, just updating testconfiguration.properties to add the test to minillaplocal.shared. This shouldn't affect compilation. > Query results cache: Allow queries to refer to the pending results of a query > that has not finished yet > --- > > Key: HIVE-18846 > URL: https://issues.apache.org/jira/browse/HIVE-18846 > Project: Hive > Issue Type: Sub-task > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18846.1.patch, HIVE-18846.2.patch, > HIVE-18846.3.patch > > > Currently, a query's results can only be looked up in the cache if the query > has completely finished execution. Allow new queries to use the results cache > to find queries that are still executing so they can re-use the results when > the query has finished. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18846) Query results cache: Allow queries to refer to the pending results of a query that has not finished yet
[ https://issues.apache.org/jira/browse/HIVE-18846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399955#comment-16399955 ] Gopal V commented on HIVE-18846: LGTM - +1 Tested with 1024 users and 1min expiry - this produces a neat sawtooth graph going from 5s -> 110ms and repeating. > Query results cache: Allow queries to refer to the pending results of a query > that has not finished yet > --- > > Key: HIVE-18846 > URL: https://issues.apache.org/jira/browse/HIVE-18846 > Project: Hive > Issue Type: Sub-task > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18846.1.patch, HIVE-18846.2.patch > > > Currently, a query's results can only be looked up in the cache if the query > has completely finished execution. Allow new queries to use the results cache > to find queries that are still executing so they can re-use the results when > the query has finished. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18846) Query results cache: Allow queries to refer to the pending results of a query that has not finished yet
[
https://issues.apache.org/jira/browse/HIVE-18846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394113#comment-16394113
]
Hive QA commented on HIVE-18846:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913837/HIVE-18846.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 12955 tests
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[nopart_insert.q,insert_into_with_schema.q,input41.q,having1.q,create_table_failure3.q,default_constraint_invalid_default_value.q,database_drop_not_empty_restrict.q,windowing_after_orderby.q,orderbysortby.q,subquery_select_distinct2.q,authorization_uri_alterpart_loc.q,udf_last_day_error_1.q,constraint_duplicate_name.q,create_table_failure4.q,alter_tableprops_external_with_notnull_constraint.q,semijoin5.q,udf_format_number_wrong4.q,deletejar.q,exim_11_nonpart_noncompat_sorting.q,show_tables_bad_db2.q,drop_func_nonexistent.q,nopart_load.q,alter_table_non_partitioned_table_cascade.q,load_wrong_fileformat.q,lockneg_try_db_lock_conflict.q,udf_field_wrong_args_len.q,create_table_failure2.q,create_with_fk_constraints_enforced.q,groupby2_map_skew_multi_distinct.q,udf_min.q,authorization_update_noupdatepriv.q,show_columns2.q,authorization_insert_noselectpriv.q,orc_replace_columns3_acid.q,compare_double_bigint.q,authorization_set_nonexistent_conf.q,alter_rename_partition_failure3.q,split_sample_wrong_format2.q,create_with_fk_pk_same_tab.q,compare_double_bigint_2.q,authorization_show_roles_no_admin.q,materialized_view_authorization_rebuild_no_grant.q,unionLimit.q,authorization_revoke_table_fail2.q,authorization_insert_noinspriv.q,duplicate_insert3.q,authorization_desc_table_nosel.q,stats_noscan_non_native.q,orc_change_serde_acid.q,create_or_replace_view7.q,exim_07_nonpart_noncompat_ifof.q,create_with_unique_constraints_enforced.q,udf_concat_ws_wrong2.q,fileformat_bad_class.q,merge_negative_2.q,exim_15_part_nonpart.q,authorization_not_owner_drop_view.q,external1.q,authorization_uri_insert.q,create_with_fk_wrong_ref.q,columnstats_tbllvl_incorrect_column.q,authorization_show_parts_nosel.q,authorization_not_owner_drop_tab.q,external2.q,authorization_deletejar.q,temp_table_create_like_partitions.q,udf_greatest_error_1.q,ptf_negative_AggrFuncsWithNoGBYNoPartDef.q,alter_view_as_select_not_exist.q,touch1.q,groupby3_map_skew_multi_distinct.q,insert_into_notnull_constraint.q,exchange_partition_neg_partition_missing.q,groupby_cube_multi_gby.q,columnstats_tbllvl.q,drop_invalid_constraint2.q,alter_table_add_partition.q,update_not_acid.q,archive5.q,alter_table_constraint_invalid_pk_col.q,ivyDownload.q,udf_instr_wrong_type.q,bad_sample_clause.q,authorization_not_owner_drop_tab2.q,authorization_alter_db_owner.q,show_columns1.q,orc_type_promotion3.q,create_view_failure8.q,strict_join.q,udf_add_months_error_1.q,groupby_cube2.q,groupby_cube1.q,groupby_rollup1.q,genericFileFormat.q,invalid_cast_from_binary_4.q,drop_invalid_constraint1.q,serde_regex.q,show_partitions1.q,invalid_cast_from_binary_6.q,create_with_multi_pk_constraint.q,udf_field_wrong_type.q,groupby_grouping_sets4.q,groupby_grouping_sets3.q,insertsel_fail.q,udf_locate_wrong_type.q,orc_type_promotion1_acid.q,set_table_property.q,create_or_replace_view2.q,groupby_grouping_sets2.q,alter_view_failure.q,distinct_windowing_failure1.q,invalid_t_alter2.q,alter_table_constraint_invalid_fk_col1.q,invalid_varchar_length_2.q,authorization_show_grant_otheruser_alltabs.q,subquery_windowing_corr.q,compact_non_acid_table.q,authorization_view_4.q,authorization_disallow_transform.q,materialized_view_authorization_rebuild_other.q,authorization_fail_4.q,dbtxnmgr_nodblock.q,set_hiveconf_internal_variable1.q,input_part0_neg.q,udf_printf_wrong3.q,load_orc_negative2.q,druid_buckets.q,archive2.q,authorization_addjar.q,invalid_sum_syntax.q,insert_into_with_schema1.q,udf_add_months_error_2.q,dyn_part_max_per_node.q,authorization_revoke_table_fail1.q,udf_printf_wrong2.q,archive_multi3.q,udf_printf_wrong1.q,subquery_subquery_chain.q,authorization_view_disable_cbo_4.q,no_matching_udf.q,create_view_failure7.q,drop_native_udf.q,truncate_column_list_bucketing.q,authorization_uri_add_partition.q,authorization_view_disable_cbo_3.q,bad_exec_hooks.q,authorization_view_disable_cbo_2.q,fetchtask_ioexception.q,char_pad_convert_fail2.q,authorization_set_role_neg1.q,serde_regex3.q,authorization_delete_nodeletepriv.q,materialized_view_delete.q,create_or_replace_view6.q,bucket_mapjoin_wrong_table_metadata_2.q,msck_repair_3.q,udf_sort_array_by_wrong2.q,local_mapred_error_cache.q,alter_external_acid.q,mm_concatenate.q,authorization_fail_3.q,set_hiveconf_internal_variable0.q,udf_last_day_error_2.q,alter_table_constraint_invalid_ref.q,create_table_wrong_regex.q,describe_x
[jira] [Commented] (HIVE-18846) Query results cache: Allow queries to refer to the pending results of a query that has not finished yet
[
https://issues.apache.org/jira/browse/HIVE-18846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16394104#comment-16394104
]
Hive QA commented on HIVE-18846:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 2 new + 574 unchanged - 0
fixed = 576 total (was 574) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9586/dev-support/hive-personality.sh
|
| git revision | master / 41afa4e |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9586/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9586/yetus/patch-asflicense-problems.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9586/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Query results cache: Allow queries to refer to the pending results of a query
> that has not finished yet
> ---
>
> Key: HIVE-18846
> URL: https://issues.apache.org/jira/browse/HIVE-18846
> Project: Hive
> Issue Type: Sub-task
> Components: Query Planning
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-18846.1.patch, HIVE-18846.2.patch
>
>
> Currently, a query's results can only be looked up in the cache if the query
> has completely finished execution. Allow new queries to use the results cache
> to find queries that are still executing so they can re-use the results when
> the query has finished.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18846) Query results cache: Allow queries to refer to the pending results of a query that has not finished yet
[ https://issues.apache.org/jira/browse/HIVE-18846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16393703#comment-16393703 ] Jason Dere commented on HIVE-18846: --- RB at https://reviews.apache.org/r/66005/ > Query results cache: Allow queries to refer to the pending results of a query > that has not finished yet > --- > > Key: HIVE-18846 > URL: https://issues.apache.org/jira/browse/HIVE-18846 > Project: Hive > Issue Type: Sub-task > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18846.1.patch, HIVE-18846.2.patch > > > Currently, a query's results can only be looked up in the cache if the query > has completely finished execution. Allow new queries to use the results cache > to find queries that are still executing so they can re-use the results when > the query has finished. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18846) Query results cache: Allow queries to refer to the pending results of a query that has not finished yet
[ https://issues.apache.org/jira/browse/HIVE-18846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16393700#comment-16393700 ] Jason Dere commented on HIVE-18846: --- Update patch to fix empty results case which came up during testing. > Query results cache: Allow queries to refer to the pending results of a query > that has not finished yet > --- > > Key: HIVE-18846 > URL: https://issues.apache.org/jira/browse/HIVE-18846 > Project: Hive > Issue Type: Sub-task > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18846.1.patch, HIVE-18846.2.patch > > > Currently, a query's results can only be looked up in the cache if the query > has completely finished execution. Allow new queries to use the results cache > to find queries that are still executing so they can re-use the results when > the query has finished. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18846) Query results cache: Allow queries to refer to the pending results of a query that has not finished yet
[ https://issues.apache.org/jira/browse/HIVE-18846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16387193#comment-16387193 ] Jason Dere commented on HIVE-18846: --- Work in progress patch, this passes the existing qfile tests but requires testing with concurrent queries to see if this will cause queries to wait on the executing query to finish. This patch causes the query to wait for the pending results by blocking during query compilation, not sure if that is the best approach or not. If HIVE-17626 gets committed then we would have another possible approach where the query could block during query execution (and cause a retryable failure if it turns out that the pending results did not result in a valid cacheable result). > Query results cache: Allow queries to refer to the pending results of a query > that has not finished yet > --- > > Key: HIVE-18846 > URL: https://issues.apache.org/jira/browse/HIVE-18846 > Project: Hive > Issue Type: Sub-task > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-18846.1.patch > > > Currently, a query's results can only be looked up in the cache if the query > has completely finished execution. Allow new queries to use the results cache > to find queries that are still executing so they can re-use the results when > the query has finished. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
