[jira] [Commented] (HIVE-18387) Minimize time that REBUILD locks the materialized view
[ https://issues.apache.org/jira/browse/HIVE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16373165#comment-16373165 ] Jesus Camacho Rodriguez commented on HIVE-18387: [~kgyrtkirk], after investigation this was broken by the interaction of two issues that were pushed at the same time: this one and HIVE-17871. The issue will be solved as part of HIVE-18770. > Minimize time that REBUILD locks the materialized view > -- > > Key: HIVE-18387 > URL: https://issues.apache.org/jira/browse/HIVE-18387 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18387.01.patch, HIVE-18387.02.patch, > HIVE-18387.03.patch, HIVE-18387.04.patch, HIVE-18387.05.patch, > HIVE-18387.07.patch, HIVE-18387.08.patch, HIVE-18387.09.patch, > HIVE-18387.11.patch, HIVE-18387.patch > > > Currently, REBUILD will block the materialized view while the final move task > is being executed. The idea for this improvement is to create the new > materialization in a new folder (new version) and then just flip the pointer > to the folder in the MV definition in the metastore. REBUILD operations for a > given MV should get an exclusive lock though, i.e., they cannot be executed > concurrently. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18387) Minimize time that REBUILD locks the materialized view
[ https://issues.apache.org/jira/browse/HIVE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371671#comment-16371671 ] Jesus Camacho Rodriguez commented on HIVE-18387: Thanks [~kgyrtkirk], I am taking a look right now, I will open a new JIRA for it. > Minimize time that REBUILD locks the materialized view > -- > > Key: HIVE-18387 > URL: https://issues.apache.org/jira/browse/HIVE-18387 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18387.01.patch, HIVE-18387.02.patch, > HIVE-18387.03.patch, HIVE-18387.04.patch, HIVE-18387.05.patch, > HIVE-18387.07.patch, HIVE-18387.08.patch, HIVE-18387.09.patch, > HIVE-18387.11.patch, HIVE-18387.patch > > > Currently, REBUILD will block the materialized view while the final move task > is being executed. The idea for this improvement is to create the new > materialization in a new folder (new version) and then just flip the pointer > to the folder in the MV definition in the metastore. REBUILD operations for a > given MV should get an exclusive lock though, i.e., they cannot be executed > concurrently. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18387) Minimize time that REBUILD locks the materialized view
[ https://issues.apache.org/jira/browse/HIVE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371155#comment-16371155 ] Zoltan Haindrich commented on HIVE-18387: - [~jcamachorodriguez] it seems to me that [druidmini_mv test|https://builds.apache.org/job/PreCommit-HIVE-Build/9287/testReport/org.apache.hadoop.hive.cli/TestMiniDruidCliDriver/testCliDriver_druidmini_mv_/] is failing since this commit; there are some resultset differences as well; could you please take a look? > Minimize time that REBUILD locks the materialized view > -- > > Key: HIVE-18387 > URL: https://issues.apache.org/jira/browse/HIVE-18387 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18387.01.patch, HIVE-18387.02.patch, > HIVE-18387.03.patch, HIVE-18387.04.patch, HIVE-18387.05.patch, > HIVE-18387.07.patch, HIVE-18387.08.patch, HIVE-18387.09.patch, > HIVE-18387.11.patch, HIVE-18387.patch > > > Currently, REBUILD will block the materialized view while the final move task > is being executed. The idea for this improvement is to create the new > materialization in a new folder (new version) and then just flip the pointer > to the folder in the MV definition in the metastore. REBUILD operations for a > given MV should get an exclusive lock though, i.e., they cannot be executed > concurrently. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18387) Minimize time that REBUILD locks the materialized view
[ https://issues.apache.org/jira/browse/HIVE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366783#comment-16366783 ] Hive QA commented on HIVE-18387: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12910834/HIVE-18387.11.patch {color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 31 failed/errored test(s), 13783 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries] (batchId=240) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_update_status] (batchId=79) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=78) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl] (batchId=174) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_mv] (batchId=248) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] (batchId=151) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez_empty] (batchId=156) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=166) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] (batchId=170) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=161) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_3] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_rebuild_dummy] (batchId=158) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1] (batchId=167) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=160) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[materialized_view_authorization_rebuild_no_grant] (batchId=93) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[materialized_view_authorization_rebuild_other] (batchId=93) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] (batchId=121) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=221) org.apache.hadoop.hive.metastore.TestReplChangeManager.testRecycleNonPartTable (batchId=223) org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNullStorageDescriptorInNew[Embedded] (batchId=205) org.apache.hadoop.hive.ql.TestAcidOnTez.testGetSplitsLocks (batchId=224) org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=187) org.apache.hive.hcatalog.listener.TestDbNotificationListener.alterIndex (batchId=242) org.apache.hive.hcatalog.listener.TestDbNotificationListener.createIndex (batchId=242) org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropIndex (batchId=242) org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd (batchId=235) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9245/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9245/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9245/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 31 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12910834 - PreCommit-HIVE-Build > Minimize time that REBUILD locks the materialized view > -- > > Key: HIVE-18387 > URL: https://issues.apache.org/jira/browse/HIVE-18387 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18387.01.patch, HIVE-18387.02.patch, > HIVE-18387.03.patch, HIVE-18387.04.patch, HIVE-18387.05.patch, > HIVE-18387.07.patch, HIVE-18387.08.patch, HIVE-18387.09.patch, > HIVE-18387.11.patch, HIVE-18387.patch > > > Currently, REBUILD will block the materialized view while the fi
[jira] [Commented] (HIVE-18387) Minimize time that REBUILD locks the materialized view
[ https://issues.apache.org/jira/browse/HIVE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366769#comment-16366769 ] Hive QA commented on HIVE-18387: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 41s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 6s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 47s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 21s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m 5s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 47s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 21s{color} | {color:red} root: The patch generated 34 new + 3147 unchanged - 6 fixed = 3181 total (was 3153) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 22s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 630 unchanged - 1 fixed = 631 total (was 631) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 49s{color} | {color:red} ql: The patch generated 30 new + 1232 unchanged - 4 fixed = 1262 total (was 1236) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 27s{color} | {color:red} standalone-metastore: The patch generated 3 new + 1265 unchanged - 1 fixed = 1268 total (was 1266) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 39 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 46s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 58m 6s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense xml javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh | | git revision | master / 01f34e4 | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9245/yetus/diff-checkstyle-root.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9245/yetus/diff-checkstyle-itests_hive-unit.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9245/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9245/yetus/diff-checkstyle-standalone-metastore.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-9245/yetus/whitespace-eol.txt | | asflicense | http://104.198.109.242/logs//PreCommit-HIVE-Build-9245/yetus/patch-asflicense-problems.txt | | modules | C: . itests/hcatalog-unit itests/hive-unit ql standalone-metastore U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-9245/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Minimi
[jira] [Commented] (HIVE-18387) Minimize time that REBUILD locks the materialized view
[ https://issues.apache.org/jira/browse/HIVE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365058#comment-16365058 ] Hive QA commented on HIVE-18387: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 44s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 30s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 1s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 28s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m 12s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 6s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 33s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 43s{color} | {color:red} root: The patch generated 33 new + 3147 unchanged - 6 fixed = 3180 total (was 3153) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 24s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 630 unchanged - 1 fixed = 631 total (was 631) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 57s{color} | {color:red} ql: The patch generated 29 new + 1232 unchanged - 4 fixed = 1261 total (was 1236) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 30s{color} | {color:red} standalone-metastore: The patch generated 3 new + 1265 unchanged - 1 fixed = 1268 total (was 1266) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 39 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m 28s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 61m 27s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense xml javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh | | git revision | master / 71d6e16 | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9220/yetus/diff-checkstyle-root.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9220/yetus/diff-checkstyle-itests_hive-unit.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9220/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9220/yetus/diff-checkstyle-standalone-metastore.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-9220/yetus/whitespace-eol.txt | | asflicense | http://104.198.109.242/logs//PreCommit-HIVE-Build-9220/yetus/patch-asflicense-problems.txt | | modules | C: . itests/hcatalog-unit itests/hive-unit ql standalone-metastore U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-9220/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Minimi
[jira] [Commented] (HIVE-18387) Minimize time that REBUILD locks the materialized view
[ https://issues.apache.org/jira/browse/HIVE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364625#comment-16364625 ] Hive QA commented on HIVE-18387: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 45s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 22s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 13s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 38s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m 10s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 1s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 28s{color} | {color:red} root: The patch generated 33 new + 3147 unchanged - 4 fixed = 3180 total (was 3151) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 21s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 630 unchanged - 1 fixed = 631 total (was 631) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 49s{color} | {color:red} ql: The patch generated 29 new + 1232 unchanged - 2 fixed = 1261 total (was 1234) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 28s{color} | {color:red} standalone-metastore: The patch generated 3 new + 1265 unchanged - 1 fixed = 1268 total (was 1266) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 39 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 57s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 14s{color} | {color:red} The patch generated 49 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 60m 5s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh | | git revision | master / a2d22b4 | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9214/yetus/diff-checkstyle-root.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9214/yetus/diff-checkstyle-itests_hive-unit.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9214/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9214/yetus/diff-checkstyle-standalone-metastore.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-9214/yetus/whitespace-eol.txt | | asflicense | http://104.198.109.242/logs//PreCommit-HIVE-Build-9214/yetus/patch-asflicense-problems.txt | | modules | C: . itests/hcatalog-unit itests/hive-unit ql standalone-metastore U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-9214/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Minimize time that REBUILD locks the materialized view > -- > > Key: HIVE-18387 >
[jira] [Commented] (HIVE-18387) Minimize time that REBUILD locks the materialized view
[ https://issues.apache.org/jira/browse/HIVE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364388#comment-16364388 ] Hive QA commented on HIVE-18387: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12910483/HIVE-18387.05.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9212/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9212/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9212/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2018-02-14 16:41:27.348 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-9212/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2018-02-14 16:41:27.351 + cd apache-github-source-source + git fetch origin >From https://github.com/apache/hive b0d3cb4..b98fb1f master -> origin/master + git reset --hard HEAD HEAD is now at b0d3cb4 HIVE-18448: Drop Support For Indexes From Apache Hive (Zoltan Haindrich reviewed by Ashutosh Chauhan) + git clean -f -d + git checkout master Already on 'master' Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded. (use "git pull" to update your local branch) + git reset --hard origin/master HEAD is now at b98fb1f HIVE-18604: DropDatabase cascade fails when there is an index in the DB (Adam Szita, reviewed by Peter Vary) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2018-02-14 16:41:28.667 + rm -rf ../yetus + mkdir ../yetus + git gc + cp -R . ../yetus + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-9212/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: a/itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java: does not exist in index error: a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenarios.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/QueryLifeTimeHookRunner.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/plan/ImportTableDesc.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsTask.java: does not exist in index error: a/ql/src/test/queries/clientpositive/druidmini_mv.q: does not exist in index error: a/ql/src/test/queries/clientpositive/materialized_view_create_rewrite_4.q: does not exist in index error: a/ql/src/test/results/clientpositive/druid/druidmini_mv.q.out: does not exist in index error: a/ql/src/test/results/clientpositive/materialized_view_create_re
[jira] [Commented] (HIVE-18387) Minimize time that REBUILD locks the materialized view
[ https://issues.apache.org/jira/browse/HIVE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363344#comment-16363344 ] Jesus Camacho Rodriguez commented on HIVE-18387: Applied changes based on review comments and rebased patch. Uploaded for a new ptests run. > Minimize time that REBUILD locks the materialized view > -- > > Key: HIVE-18387 > URL: https://issues.apache.org/jira/browse/HIVE-18387 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18387.01.patch, HIVE-18387.02.patch, > HIVE-18387.03.patch, HIVE-18387.04.patch, HIVE-18387.05.patch, > HIVE-18387.patch > > > Currently, REBUILD will block the materialized view while the final move task > is being executed. The idea for this improvement is to create the new > materialization in a new folder (new version) and then just flip the pointer > to the folder in the MV definition in the metastore. REBUILD operations for a > given MV should get an exclusive lock though, i.e., they cannot be executed > concurrently. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18387) Minimize time that REBUILD locks the materialized view
[ https://issues.apache.org/jira/browse/HIVE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363049#comment-16363049 ] Ashutosh Chauhan commented on HIVE-18387: - +1 some minor comments on RB. > Minimize time that REBUILD locks the materialized view > -- > > Key: HIVE-18387 > URL: https://issues.apache.org/jira/browse/HIVE-18387 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18387.01.patch, HIVE-18387.02.patch, > HIVE-18387.03.patch, HIVE-18387.04.patch, HIVE-18387.patch > > > Currently, REBUILD will block the materialized view while the final move task > is being executed. The idea for this improvement is to create the new > materialization in a new folder (new version) and then just flip the pointer > to the folder in the MV definition in the metastore. REBUILD operations for a > given MV should get an exclusive lock though, i.e., they cannot be executed > concurrently. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18387) Minimize time that REBUILD locks the materialized view
[ https://issues.apache.org/jira/browse/HIVE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359228#comment-16359228 ] Hive QA commented on HIVE-18387: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 35s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 7s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 45s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 22s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m 11s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 53s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 27s{color} | {color:red} standalone-metastore: The patch generated 3 new + 1266 unchanged - 1 fixed = 1269 total (was 1267) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 54s{color} | {color:red} ql: The patch generated 30 new + 1265 unchanged - 2 fixed = 1295 total (was 1267) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 23s{color} | {color:red} root: The patch generated 34 new + 3181 unchanged - 4 fixed = 3215 total (was 3185) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 22s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 630 unchanged - 1 fixed = 631 total (was 631) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 39 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m 5s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 58m 26s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh | | git revision | master / 89e75c7 | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9130/yetus/diff-checkstyle-standalone-metastore.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9130/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9130/yetus/diff-checkstyle-root.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9130/yetus/diff-checkstyle-itests_hive-unit.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-9130/yetus/whitespace-eol.txt | | modules | C: standalone-metastore ql . itests/hcatalog-unit itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-9130/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Minimize time that REBUILD locks the materialized view > -- > > Key: HIVE-18387 > URL: https://issues.apache.org/jira/browse/HIVE-18387 > Project: Hive >
[jira] [Commented] (HIVE-18387) Minimize time that REBUILD locks the materialized view
[ https://issues.apache.org/jira/browse/HIVE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352062#comment-16352062 ] Hive QA commented on HIVE-18387: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 40s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 52s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 54s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 26s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 26s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 12s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 52s{color} | {color:red} ql: The patch generated 12 new + 1030 unchanged - 9 fixed = 1042 total (was 1039) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 55s{color} | {color:red} root: The patch generated 13 new + 2041 unchanged - 10 fixed = 2054 total (was 2051) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 19s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 635 unchanged - 1 fixed = 636 total (was 636) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 29s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 54m 39s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh | | git revision | master / 0a328f0 | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9015/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9015/yetus/diff-checkstyle-root.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-9015/yetus/diff-checkstyle-itests_hive-unit.txt | | modules | C: common standalone-metastore ql . itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-9015/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Minimize time that REBUILD locks the materialized view > -- > > Key: HIVE-18387 > URL: https://issues.apache.org/jira/browse/HIVE-18387 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18387.01.patch, HIVE-18387.02.patch, > HIVE-18387.03.patch, HIVE-18387.patch > > > Currently, REBUILD will block the materialized view while the final move task > is being executed. The idea for this improvement is to create the new > materialization in a new folder (new vers
[jira] [Commented] (HIVE-18387) Minimize time that REBUILD locks the materialized view
[ https://issues.apache.org/jira/browse/HIVE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352030#comment-16352030 ] Jesus Camacho Rodriguez commented on HIVE-18387: [~ashutoshc], I have rebased this patch after HIVE-18546 went in, this might go in before HIVE-18387. Could you take a look? https://reviews.apache.org/r/65250/ I created HIVE-18621 as a follow-up to this issue. Replication does not work properly for views and materialized views yet, hence I think it can be tackled there. > Minimize time that REBUILD locks the materialized view > -- > > Key: HIVE-18387 > URL: https://issues.apache.org/jira/browse/HIVE-18387 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18387.01.patch, HIVE-18387.02.patch, > HIVE-18387.03.patch, HIVE-18387.patch > > > Currently, REBUILD will block the materialized view while the final move task > is being executed. The idea for this improvement is to create the new > materialization in a new folder (new version) and then just flip the pointer > to the folder in the MV definition in the metastore. REBUILD operations for a > given MV should get an exclusive lock though, i.e., they cannot be executed > concurrently. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18387) Minimize time that REBUILD locks the materialized view
[ https://issues.apache.org/jira/browse/HIVE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338312#comment-16338312 ] Hive QA commented on HIVE-18387: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12907382/HIVE-18387.02.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 29 failed/errored test(s), 11658 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite] (batchId=246) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] (batchId=48) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] (batchId=12) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl] (batchId=173) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_mv] (batchId=246) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] (batchId=151) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1] (batchId=170) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=165) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] (batchId=169) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=160) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create] (batchId=167) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite] (batchId=152) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_3] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_dummy] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_multi_db] (batchId=166) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_rebuild_dummy] (batchId=157) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_describe] (batchId=166) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=160) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes] (batchId=162) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part] (batchId=94) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1] (batchId=94) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] (batchId=121) org.apache.hadoop.hive.ql.hooks.TestHooks.org.apache.hadoop.hive.ql.hooks.TestHooks (batchId=289) org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=254) org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=186) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=232) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=232) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=232) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8817/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8817/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8817/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 29 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12907382 - PreCommit-HIVE-Build > Minimize time that REBUILD locks the materialized view > -- > > Key: HIVE-18387 > URL: https://issues.apache.org/jira/browse/HIVE-18387 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18387.01.patch, HIVE-18387.02.patch, > HIVE-18387.patch > > > Currently, REBUILD will block the materialized view while the final move task > is being executed. The idea for this improvement is to create the new > materialization in a new folder (new version) and then just flip the pointer > to the folder in the MV definition in the metastore. REBUILD operations for a >
[jira] [Commented] (HIVE-18387) Minimize time that REBUILD locks the materialized view
[ https://issues.apache.org/jira/browse/HIVE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338275#comment-16338275 ] Hive QA commented on HIVE-18387: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 39s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 58s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 4s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 50s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 26s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 53s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 46s{color} | {color:red} ql: The patch generated 12 new + 1025 unchanged - 5 fixed = 1037 total (was 1030) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 58s{color} | {color:red} root: The patch generated 12 new + 1027 unchanged - 5 fixed = 1039 total (was 1032) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 35s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 11s{color} | {color:red} The patch generated 6 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 46m 15s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh | | git revision | master / 042698c | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-8817/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-8817/yetus/diff-checkstyle-root.txt | | asflicense | http://104.198.109.242/logs//PreCommit-HIVE-Build-8817/yetus/patch-asflicense-problems.txt | | modules | C: common ql . U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-8817/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Minimize time that REBUILD locks the materialized view > -- > > Key: HIVE-18387 > URL: https://issues.apache.org/jira/browse/HIVE-18387 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18387.01.patch, HIVE-18387.02.patch, > HIVE-18387.patch > > > Currently, REBUILD will block the materialized view while the final move task > is being executed. The idea for this improvement is to create the new > materialization in a new folder (new version) and then just flip the pointer > to the folder in the MV definition in the metastore. REBUILD operations for a > given MV should get an exclusive lock though, i.e., they cannot be executed > concurrently. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18387) Minimize time that REBUILD locks the materialized view
[ https://issues.apache.org/jira/browse/HIVE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16336496#comment-16336496 ] Jesus Camacho Rodriguez commented on HIVE-18387: Rebasing patch as it was not applying cleanly anymore. > Minimize time that REBUILD locks the materialized view > -- > > Key: HIVE-18387 > URL: https://issues.apache.org/jira/browse/HIVE-18387 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18387.01.patch, HIVE-18387.02.patch, > HIVE-18387.patch > > > Currently, REBUILD will block the materialized view while the final move task > is being executed. The idea for this improvement is to create the new > materialization in a new folder (new version) and then just flip the pointer > to the folder in the MV definition in the metastore. REBUILD operations for a > given MV should get an exclusive lock though, i.e., they cannot be executed > concurrently. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18387) Minimize time that REBUILD locks the materialized view
[ https://issues.apache.org/jira/browse/HIVE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16336491#comment-16336491 ] Hive QA commented on HIVE-18387: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12907190/HIVE-18387.01.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8783/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8783/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8783/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2018-01-23 22:41:17.242 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-8783/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2018-01-23 22:41:17.246 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at fff86f3 HIVE-17833: Publish split generation counters (Prasanth Jayachandran reviewed by Sergey Shelukhin) + git clean -f -d + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at fff86f3 HIVE-17833: Publish split generation counters (Prasanth Jayachandran reviewed by Sergey Shelukhin) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2018-01-23 22:41:19.499 + rm -rf ../yetus + mkdir ../yetus + git gc + cp -R . ../yetus + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-8783/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/plan/ImportTableDesc.java: does not exist in index error: a/ql/src/test/queries/clientpositive/druidmini_mv.q: does not exist in index error: a/ql/src/test/results/clientpositive/druid/druidmini_mv.q.out: does not exist in index error: a/ql/src/test/results/clientpositive/materialized_view_create_rewrite_3.q.out: does not exist in index error: a/ql/src/test/results/clientpositive/materialized_view_create_rewrite_4.q.out: does not exist in index error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java:5135 error: repository lacks the necessary blob to fall back on 3-way merge. error: ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java: patch does not apply error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:12550 error: repository lacks the necessary blob to fall back on 3-way merge. error: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java: patch does not apply error: patch failed: ql/src/test/results/clientpositive/materialized_view_create_rewrite_3.q.out:542 error: repository lacks the necessary blob to fall back on 3-way merge. error: ql/src/test/results/clientpositive/materialized_view_create_rewrite_3.q.out: patch does not apply error: patch failed: ql/src/test/results/clientpositive/materialized_view_create_rewrite_4.q.out:629 error: repository lacks the necessary blob to fall back on 3-way merge. error: ql/src/test/results/clientpositive/materialized_view_create_rewrite_4.q.out: patch does not apply error: src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java: does not exist in index error: src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java: does not exist in index error: src/java/org/apache/hadoop/h
[jira] [Commented] (HIVE-18387) Minimize time that REBUILD locks the materialized view
[ https://issues.apache.org/jira/browse/HIVE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16335060#comment-16335060 ] Jesus Camacho Rodriguez commented on HIVE-18387: [~ashutoshc], I include HIVE-18321 within same patch, as this patch only required minor changes to make it all work. Could you take a look? https://reviews.apache.org/r/65250/ Thanks > Minimize time that REBUILD locks the materialized view > -- > > Key: HIVE-18387 > URL: https://issues.apache.org/jira/browse/HIVE-18387 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18387.01.patch, HIVE-18387.patch > > > Currently, REBUILD will block the materialized view while the final move task > is being executed. The idea for this improvement is to create the new > materialization in a new folder (new version) and then just flip the pointer > to the folder in the MV definition in the metastore. REBUILD operations for a > given MV should get an exclusive lock though, i.e., they cannot be executed > concurrently. -- This message was sent by Atlassian JIRA (v7.6.3#76005)