[jira] [Commented] (HIVE-18839) Implement incremental rebuild for materialized views (only insert operations in source tables)
[ https://issues.apache.org/jira/browse/HIVE-18839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431671#comment-16431671 ] Hive QA commented on HIVE-18839: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12918226/HIVE-18839.04.patch {color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 141 failed/errored test(s), 14040 tests executed *Failed tests:* {noformat} TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=253) TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed out) (batchId=247) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=253) TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed out) (batchId=247) TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=253) TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=253) TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed out) (batchId=217) TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely timed out) (batchId=247) TestTezPerfCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=253) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats] (batchId=54) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[statsoptimizer] (batchId=62) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] (batchId=153) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all] (batchId=152) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby_groupingset_bug] (batchId=174) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=169) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin] (batchId=168) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1] (batchId=171) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main] (batchId=160) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[update_access_time_non_current_db] (batchId=172) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_semijoin_reduction] (batchId=155) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat] (batchId=183) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5] (batchId=105) org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver (batchId=95) org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[alter_notnull_constraint_violation] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into_acid_notnull] (batchId=95) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into_notnull_constraint] (batchId=95) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_multi_into_notnull] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_overwrite_notnull_constraint] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_bucketmapjoin] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_mapjoin_14] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[sortmerge_mapjoin_mismatch_1] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_2] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_1] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_2] (batchId=95) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_in_agg] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_in_implicit_gby] (batchId=95) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_notin_implicit_gby] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_bucketed_column] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_column_list_bucketing] (batchId=95) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_column_seqfile] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_nonexistant_column] (batchId=95)
[jira] [Commented] (HIVE-18839) Implement incremental rebuild for materialized views (only insert operations in source tables)
[ https://issues.apache.org/jira/browse/HIVE-18839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431633#comment-16431633 ] Hive QA commented on HIVE-18839: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 1s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 46s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 29s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 23s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 41s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 54s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 57s{color} | {color:red} ql in the patch failed. {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 20s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 53s{color} | {color:red} ql: The patch generated 41 new + 1163 unchanged - 20 fixed = 1204 total (was 1183) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 33s{color} | {color:red} standalone-metastore: The patch generated 10 new + 1603 unchanged - 1 fixed = 1613 total (was 1604) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 28 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 40s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 25m 22s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-10103/dev-support/hive-personality.sh | | git revision | master / 55fb0a1 | | Default Java | 1.8.0_111 | | mvninstall | http://104.198.109.242/logs//PreCommit-HIVE-Build-10103/yetus/patch-mvninstall-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10103/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10103/yetus/diff-checkstyle-standalone-metastore.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-10103/yetus/whitespace-eol.txt | | modules | C: common itests ql standalone-metastore U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-10103/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Implement incremental rebuild for materialized views (only insert operations > in source tables) > -- > > Key: HIVE-18839 > URL: https://issues.apache.org/jira/browse/HIVE-18839 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: TODOC3.0 > Attachments: HIVE-18839.01.patch, HIVE-18839.02.patch, > HIVE-18839.03.patch, HIVE-18839.04.patch, HIVE-18839.patch > > > Implementation will follow current code path for full
[jira] [Commented] (HIVE-18839) Implement incremental rebuild for materialized views (only insert operations in source tables)
[ https://issues.apache.org/jira/browse/HIVE-18839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431062#comment-16431062 ] Jesus Camacho Rodriguez commented on HIVE-18839: [~ashutoshc], uploaded new patch that fixes both failures. Let's have another ptest run before pushing it. > Implement incremental rebuild for materialized views (only insert operations > in source tables) > -- > > Key: HIVE-18839 > URL: https://issues.apache.org/jira/browse/HIVE-18839 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: TODOC3.0 > Attachments: HIVE-18839.01.patch, HIVE-18839.02.patch, > HIVE-18839.03.patch, HIVE-18839.04.patch, HIVE-18839.patch > > > Implementation will follow current code path for full rebuild. > When the MV query plan is retrieved, if the MV contents are outdated because > there were insert operations in the source tables, we will introduce a filter > with a condition based on stored value of ValidWriteIdLists. For instance, > {{WRITE_ID < high_txn_id AND WRITE_ID NOT IN (x, y, ...)}}. Then the > rewriting will do the rest of the work by creating a partial rewriting, where > the contents of the MV are read as well as the new contents from the source > tables. > This mechanism will not work only for ALTER MV... REBUILD, but also for user > queries which will be able to benefit from using outdated MVs to compute part > of the needed results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18839) Implement incremental rebuild for materialized views (only insert operations in source tables)
[ https://issues.apache.org/jira/browse/HIVE-18839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430996#comment-16430996 ] Ashutosh Chauhan commented on HIVE-18839: - Failures TestMetaStoreMaterializationsCacheCleaner and TestUpdateDeleteSemanticAnalyzer looks related. > Implement incremental rebuild for materialized views (only insert operations > in source tables) > -- > > Key: HIVE-18839 > URL: https://issues.apache.org/jira/browse/HIVE-18839 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: TODOC3.0 > Attachments: HIVE-18839.01.patch, HIVE-18839.02.patch, > HIVE-18839.03.patch, HIVE-18839.patch > > > Implementation will follow current code path for full rebuild. > When the MV query plan is retrieved, if the MV contents are outdated because > there were insert operations in the source tables, we will introduce a filter > with a condition based on stored value of ValidWriteIdLists. For instance, > {{WRITE_ID < high_txn_id AND WRITE_ID NOT IN (x, y, ...)}}. Then the > rewriting will do the rest of the work by creating a partial rewriting, where > the contents of the MV are read as well as the new contents from the source > tables. > This mechanism will not work only for ALTER MV... REBUILD, but also for user > queries which will be able to benefit from using outdated MVs to compute part > of the needed results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18839) Implement incremental rebuild for materialized views (only insert operations in source tables)
[ https://issues.apache.org/jira/browse/HIVE-18839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430954#comment-16430954 ] Hive QA commented on HIVE-18839: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12918121/HIVE-18839.03.patch {color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 149 failed/errored test(s), 13566 tests executed *Failed tests:* {noformat} TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=253) TestCopyUtils - did not produce a TEST-*.xml file (likely timed out) (batchId=231) TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed out) (batchId=247) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=253) TestExportImport - did not produce a TEST-*.xml file (likely timed out) (batchId=231) TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed out) (batchId=247) TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=253) TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=253) TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=93) [infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q] TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=95)
[jira] [Commented] (HIVE-18839) Implement incremental rebuild for materialized views (only insert operations in source tables)
[ https://issues.apache.org/jira/browse/HIVE-18839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430846#comment-16430846 ] Hive QA commented on HIVE-18839: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 1s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 49s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 25s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 25s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 49s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 46s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 57s{color} | {color:red} ql in the patch failed. {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 23s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 54s{color} | {color:red} ql: The patch generated 41 new + 1161 unchanged - 20 fixed = 1202 total (was 1181) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 32s{color} | {color:red} standalone-metastore: The patch generated 10 new + 1603 unchanged - 1 fixed = 1613 total (was 1604) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s{color} | {color:red} The patch has 28 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 28s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 15s{color} | {color:red} The patch generated 1 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 26m 17s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-10094/dev-support/hive-personality.sh | | git revision | master / 959e772 | | Default Java | 1.8.0_111 | | mvninstall | http://104.198.109.242/logs//PreCommit-HIVE-Build-10094/yetus/patch-mvninstall-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10094/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10094/yetus/diff-checkstyle-standalone-metastore.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-10094/yetus/whitespace-eol.txt | | asflicense | http://104.198.109.242/logs//PreCommit-HIVE-Build-10094/yetus/patch-asflicense-problems.txt | | modules | C: common itests ql standalone-metastore U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-10094/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Implement incremental rebuild for materialized views (only insert operations > in source tables) > -- > > Key: HIVE-18839 > URL: https://issues.apache.org/jira/browse/HIVE-18839 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: TODOC3.0 > Attachments: HIVE-18839.01.patch, HIVE-18839.02.patch, > HIVE-18839.03.patch,
[jira] [Commented] (HIVE-18839) Implement incremental rebuild for materialized views (only insert operations in source tables)
[ https://issues.apache.org/jira/browse/HIVE-18839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430678#comment-16430678 ] Ashutosh Chauhan commented on HIVE-18839: - +1 pending tests > Implement incremental rebuild for materialized views (only insert operations > in source tables) > -- > > Key: HIVE-18839 > URL: https://issues.apache.org/jira/browse/HIVE-18839 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: TODOC3.0 > Attachments: HIVE-18839.01.patch, HIVE-18839.02.patch, > HIVE-18839.03.patch, HIVE-18839.patch > > > Implementation will follow current code path for full rebuild. > When the MV query plan is retrieved, if the MV contents are outdated because > there were insert operations in the source tables, we will introduce a filter > with a condition based on stored value of ValidWriteIdLists. For instance, > {{WRITE_ID < high_txn_id AND WRITE_ID NOT IN (x, y, ...)}}. Then the > rewriting will do the rest of the work by creating a partial rewriting, where > the contents of the MV are read as well as the new contents from the source > tables. > This mechanism will not work only for ALTER MV... REBUILD, but also for user > queries which will be able to benefit from using outdated MVs to compute part > of the needed results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18839) Implement incremental rebuild for materialized views (only insert operations in source tables)
[ https://issues.apache.org/jira/browse/HIVE-18839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428234#comment-16428234 ] Hive QA commented on HIVE-18839: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12917634/HIVE-18839.02.patch {color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 115 failed/errored test(s), 13089 tests executed *Failed tests:* {noformat} TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=253) TestBeeLineExceptionHandling - did not produce a TEST-*.xml file (likely timed out) (batchId=191) TestBeeLineHistory - did not produce a TEST-*.xml file (likely timed out) (batchId=191) TestBeelineArgParsing - did not produce a TEST-*.xml file (likely timed out) (batchId=191) TestClientCommandHookFactory - did not produce a TEST-*.xml file (likely timed out) (batchId=191) TestCopyUtils - did not produce a TEST-*.xml file (likely timed out) (batchId=231) TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed out) (batchId=247) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=253) TestExportImport - did not produce a TEST-*.xml file (likely timed out) (batchId=231) TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed out) (batchId=247) TestHiveCli - did not produce a TEST-*.xml file (likely timed out) (batchId=191) TestHiveSchemaTool - did not produce a TEST-*.xml file (likely timed out) (batchId=191) TestIncrementalRows - did not produce a TEST-*.xml file (likely timed out) (batchId=191) TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=253) TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=253) TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=93) [infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q] TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=95)
[jira] [Commented] (HIVE-18839) Implement incremental rebuild for materialized views (only insert operations in source tables)
[ https://issues.apache.org/jira/browse/HIVE-18839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428190#comment-16428190 ] Hive QA commented on HIVE-18839: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 1s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 49s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 5s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 25s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 45s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 50s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 56s{color} | {color:red} ql in the patch failed. {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 27s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 56s{color} | {color:red} ql: The patch generated 28 new + 1161 unchanged - 20 fixed = 1189 total (was 1181) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 34s{color} | {color:red} standalone-metastore: The patch generated 10 new + 1603 unchanged - 1 fixed = 1613 total (was 1604) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 28 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 33s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 16s{color} | {color:red} The patch generated 53 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 26m 4s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-10036/dev-support/hive-personality.sh | | git revision | master / 0a81e1e | | Default Java | 1.8.0_111 | | mvninstall | http://104.198.109.242/logs//PreCommit-HIVE-Build-10036/yetus/patch-mvninstall-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10036/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10036/yetus/diff-checkstyle-standalone-metastore.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-10036/yetus/whitespace-eol.txt | | asflicense | http://104.198.109.242/logs//PreCommit-HIVE-Build-10036/yetus/patch-asflicense-problems.txt | | modules | C: common itests ql standalone-metastore U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-10036/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Implement incremental rebuild for materialized views (only insert operations > in source tables) > -- > > Key: HIVE-18839 > URL: https://issues.apache.org/jira/browse/HIVE-18839 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: TODOC3.0 > Attachments: HIVE-18839.01.patch, HIVE-18839.02.patch, > HIVE-18839.patch > > >
[jira] [Commented] (HIVE-18839) Implement incremental rebuild for materialized views (only insert operations in source tables)
[ https://issues.apache.org/jira/browse/HIVE-18839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426416#comment-16426416 ] Jesus Camacho Rodriguez commented on HIVE-18839: [~ashutoshc], addressed comments and uploaded a new patch to RB and the issue. I left a couple of questions in RB. In turn, after moving one of the old rewriting tests to MiniLlapLocalDriver, rewriting is not being triggered for one query (last query in {{materialized_view_rewrite_4.q}}), which is weird because logical plan should be the same (I am thinking on a different configuration property value that is altering the plan, maybe). Issue does not seem related in any case, so I will trigger that in a follow-up if that is OK. > Implement incremental rebuild for materialized views (only insert operations > in source tables) > -- > > Key: HIVE-18839 > URL: https://issues.apache.org/jira/browse/HIVE-18839 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: TODOC3.0 > Attachments: HIVE-18839.01.patch, HIVE-18839.02.patch, > HIVE-18839.patch > > > Implementation will follow current code path for full rebuild. > When the MV query plan is retrieved, if the MV contents are outdated because > there were insert operations in the source tables, we will introduce a filter > with a condition based on stored value of ValidWriteIdLists. For instance, > {{WRITE_ID < high_txn_id AND WRITE_ID NOT IN (x, y, ...)}}. Then the > rewriting will do the rest of the work by creating a partial rewriting, where > the contents of the MV are read as well as the new contents from the source > tables. > This mechanism will not work only for ALTER MV... REBUILD, but also for user > queries which will be able to benefit from using outdated MVs to compute part > of the needed results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18839) Implement incremental rebuild for materialized views (only insert operations in source tables)
[ https://issues.apache.org/jira/browse/HIVE-18839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426405#comment-16426405 ] Hive QA commented on HIVE-18839: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12917425/HIVE-18839.01.patch {color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 211 failed/errored test(s), 13977 tests executed *Failed tests:* {noformat} TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=252) TestCopyUtils - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed out) (batchId=246) TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=252) TestExportImport - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed out) (batchId=246) TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=252) TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=252) TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed out) (batchId=216) TestReplicationOnHDFSEncryptedZones - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestReplicationScenarios - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestReplicationScenariosAcidTables - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestReplicationScenariosAcrossInstances - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely timed out) (batchId=246) TestTezPerfCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=252) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats] (batchId=54) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] (batchId=50) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_create_rewrite_time_window] (batchId=14) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[varchar_2] (batchId=67) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] (batchId=153) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez_empty] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby_groupingset_bug] (batchId=174) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=169) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_3] (batchId=165) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_rebuild_dummy] (batchId=160) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin] (batchId=168) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main] (batchId=160) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[update_access_time_non_current_db] (batchId=171) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_div0] (batchId=170) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_semijoin_reduction] (batchId=155) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3] (batchId=105) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5] (batchId=105) org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver (batchId=95) org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[alter_notnull_constraint_violation] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into_acid_notnull] (batchId=95) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_into_notnull_constraint] (batchId=95) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_multi_into_notnull] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[insert_overwrite_notnull_constraint] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[nopart_insert] (batchId=95) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_bucketmapjoin] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_mapjoin_14] (batchId=96) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[sortmerge_mapjoin_mismatch_1] (batchId=96)
[jira] [Commented] (HIVE-18839) Implement incremental rebuild for materialized views (only insert operations in source tables)
[ https://issues.apache.org/jira/browse/HIVE-18839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426382#comment-16426382 ] Hive QA commented on HIVE-18839: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 47s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 48s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 23s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 44s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 54s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 58s{color} | {color:red} ql in the patch failed. {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 31s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 54s{color} | {color:red} ql: The patch generated 23 new + 1161 unchanged - 20 fixed = 1184 total (was 1181) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 34s{color} | {color:red} standalone-metastore: The patch generated 10 new + 1608 unchanged - 1 fixed = 1618 total (was 1609) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 24 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 42s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 15s{color} | {color:red} The patch generated 53 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 25m 57s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-10001/dev-support/hive-personality.sh | | git revision | master / dc5a943 | | Default Java | 1.8.0_111 | | mvninstall | http://104.198.109.242/logs//PreCommit-HIVE-Build-10001/yetus/patch-mvninstall-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10001/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-10001/yetus/diff-checkstyle-standalone-metastore.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-10001/yetus/whitespace-eol.txt | | asflicense | http://104.198.109.242/logs//PreCommit-HIVE-Build-10001/yetus/patch-asflicense-problems.txt | | modules | C: common ql standalone-metastore U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-10001/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Implement incremental rebuild for materialized views (only insert operations > in source tables) > -- > > Key: HIVE-18839 > URL: https://issues.apache.org/jira/browse/HIVE-18839 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: TODOC3.0 > Attachments: HIVE-18839.01.patch, HIVE-18839.patch > > > Implementation will follow
[jira] [Commented] (HIVE-18839) Implement incremental rebuild for materialized views (only insert operations in source tables)
[ https://issues.apache.org/jira/browse/HIVE-18839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424467#comment-16424467 ] Jesus Camacho Rodriguez commented on HIVE-18839: Rebased patch. > Implement incremental rebuild for materialized views (only insert operations > in source tables) > -- > > Key: HIVE-18839 > URL: https://issues.apache.org/jira/browse/HIVE-18839 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: TODOC3.0 > Attachments: HIVE-18839.01.patch, HIVE-18839.patch > > > Implementation will follow current code path for full rebuild. > When the MV query plan is retrieved, if the MV contents are outdated because > there were insert operations in the source tables, we will introduce a filter > with a condition based on stored value of ValidWriteIdLists. For instance, > {{WRITE_ID < high_txn_id AND WRITE_ID NOT IN (x, y, ...)}}. Then the > rewriting will do the rest of the work by creating a partial rewriting, where > the contents of the MV are read as well as the new contents from the source > tables. > This mechanism will not work only for ALTER MV... REBUILD, but also for user > queries which will be able to benefit from using outdated MVs to compute part > of the needed results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18839) Implement incremental rebuild for materialized views (only insert operations in source tables)
[ https://issues.apache.org/jira/browse/HIVE-18839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424383#comment-16424383 ] Hive QA commented on HIVE-18839: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12916889/HIVE-18839.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9977/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9977/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9977/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2018-04-03 18:10:05.312 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-9977/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2018-04-03 18:10:05.315 + cd apache-github-source-source + git fetch origin >From https://github.com/apache/hive ec17339..fdc1e1f master -> origin/master + git reset --hard HEAD HEAD is now at ec17339 HIVE-18955: HoS: Unable to create Channel from class NioServerSocketChannel (Rui reviewed by Jesus Camacho Rodriguez and Sahil Takiar) + git clean -f -d + git checkout master Already on 'master' Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded. (use "git pull" to update your local branch) + git reset --hard origin/master HEAD is now at fdc1e1f HIVE-18814 - Support Add Partition For Acid tables (Eugene Koifman, reviewed by Alan Gates) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2018-04-03 18:10:09.444 + rm -rf ../yetus_PreCommit-HIVE-Build-9977 + mkdir ../yetus_PreCommit-HIVE-Build-9977 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-9977 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-9977/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/MaterializedViewTask.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManager.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveMaterializedViewsRegistry.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/MaterializedViewRebuildSemanticAnalyzer.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java: does not exist in index error: a/ql/src/test/queries/clientpositive/materialized_view_create_rewrite_4.q: does not exist in index error: a/ql/src/test/results/clientpositive/materialized_view_create_rewrite_3.q.out: does not exist in index error: a/ql/src/test/results/clientpositive/materialized_view_create_rewrite_4.q.out: does not exist in index error: a/standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp: does not exist in index error: a/standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h: does not exist in index error:
[jira] [Commented] (HIVE-18839) Implement incremental rebuild for materialized views (only insert operations in source tables)
[ https://issues.apache.org/jira/browse/HIVE-18839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423289#comment-16423289 ] Jesus Camacho Rodriguez commented on HIVE-18839: [~ashutoshc], could you take a look at this one? https://reviews.apache.org/r/66369/ Thanks > Implement incremental rebuild for materialized views (only insert operations > in source tables) > -- > > Key: HIVE-18839 > URL: https://issues.apache.org/jira/browse/HIVE-18839 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: TODOC3.0 > Attachments: HIVE-18839.patch > > > Implementation will follow current code path for full rebuild. > When the MV query plan is retrieved, if the MV contents are outdated because > there were insert operations in the source tables, we will introduce a filter > with a condition based on stored value of ValidWriteIdLists. For instance, > {{WRITE_ID < high_txn_id AND WRITE_ID NOT IN (x, y, ...)}}. Then the > rewriting will do the rest of the work by creating a partial rewriting, where > the contents of the MV are read as well as the new contents from the source > tables. > This mechanism will not work only for ALTER MV... REBUILD, but also for user > queries which will be able to benefit from using outdated MVs to compute part > of the needed results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)