[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16274944#comment-16274944 ] Sergey Shelukhin commented on HIVE-17856: - [~ekoifman] [~steveyeom2017] is this ready to be committed? > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch, > HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch, > HIVE-17856.14.patch, HIVE-17856.15.patch, HIVE-17856.16.patch, > HIVE-17856.17.patch, HIVE-17856.2.patch, HIVE-17856.3.patch, > HIVE-17856.4.patch, HIVE-17856.5.patch, HIVE-17856.6.patch, > HIVE-17856.7.patch, HIVE-17856.8.patch, HIVE-17856.9.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272964#comment-16272964 ] Steve Yeom commented on HIVE-17856: --- It looks like all pre-commit test failures are gone with the last patch 17. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch, > HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch, > HIVE-17856.14.patch, HIVE-17856.15.patch, HIVE-17856.16.patch, > HIVE-17856.17.patch, HIVE-17856.2.patch, HIVE-17856.3.patch, > HIVE-17856.4.patch, HIVE-17856.5.patch, HIVE-17856.6.patch, > HIVE-17856.7.patch, HIVE-17856.8.patch, HIVE-17856.9.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272549#comment-16272549 ] Hive QA commented on HIVE-17856: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12899958/HIVE-17856.17.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11491 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_complex_join] (batchId=44) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2] (batchId=157) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=165) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=160) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=227) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=233) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=233) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=233) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8054/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8054/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8054/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12899958 - PreCommit-HIVE-Build > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch, > HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch, > HIVE-17856.14.patch, HIVE-17856.15.patch, HIVE-17856.16.patch, > HIVE-17856.17.patch, HIVE-17856.2.patch, HIVE-17856.3.patch, > HIVE-17856.4.patch, HIVE-17856.5.patch, HIVE-17856.6.patch, > HIVE-17856.7.patch, HIVE-17856.8.patch, HIVE-17856.9.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16272509#comment-16272509 ] Hive QA commented on HIVE-17856: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 1s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 12s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 6s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 20s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 58s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 19s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s{color} | {color:red} common: The patch generated 2 new + 8 unchanged - 1 fixed = 10 total (was 9) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 48s{color} | {color:red} ql: The patch generated 39 new + 1756 unchanged - 7 fixed = 1795 total (was 1763) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 9 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch 6 line(s) with tabs. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 15m 58s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh | | git revision | master / 2efb7d3 | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-8054/yetus/diff-checkstyle-common.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-8054/yetus/diff-checkstyle-ql.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-8054/yetus/whitespace-eol.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-8054/yetus/whitespace-tabs.txt | | modules | C: common ql U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-8054/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch, > HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch, > HIVE-17856.14.patch, HIVE-17856.15.patch, HIVE-17856.16.patch, > HIVE-17856.17.patch, HIVE-17856.2.patch, HIVE-17856.3.patch, > HIVE-17856.4.patch, HIVE-17856.5.patch, HIVE-17856.6.patch, > HIVE-17856.7.patch,
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271504#comment-16271504 ] Steve Yeom commented on HIVE-17856: --- Working on a fix with minimum changes on the dp_counter_mm.q failure. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch, > HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch, > HIVE-17856.14.patch, HIVE-17856.15.patch, HIVE-17856.16.patch, > HIVE-17856.2.patch, HIVE-17856.3.patch, HIVE-17856.4.patch, > HIVE-17856.5.patch, HIVE-17856.6.patch, HIVE-17856.7.patch, > HIVE-17856.8.patch, HIVE-17856.9.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269645#comment-16269645 ] Hive QA commented on HIVE-17856: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12899665/HIVE-17856.16.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 11423 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_remove_25] (batchId=86) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dp_counter_mm] (batchId=150) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2] (batchId=154) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=157) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan] (batchId=160) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=157) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=224) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230) org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementParallel (batchId=228) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8029/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8029/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8029/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 12 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12899665 - PreCommit-HIVE-Build > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch, > HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch, > HIVE-17856.14.patch, HIVE-17856.15.patch, HIVE-17856.16.patch, > HIVE-17856.2.patch, HIVE-17856.3.patch, HIVE-17856.4.patch, > HIVE-17856.5.patch, HIVE-17856.6.patch, HIVE-17856.7.patch, > HIVE-17856.8.patch, HIVE-17856.9.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269611#comment-16269611 ] Hive QA commented on HIVE-17856: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 1s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 5s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 7s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 41s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 45s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 34s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 40s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s{color} | {color:red} common: The patch generated 2 new + 8 unchanged - 1 fixed = 10 total (was 9) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 50s{color} | {color:red} ql: The patch generated 41 new + 1756 unchanged - 7 fixed = 1797 total (was 1763) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m 3s{color} | {color:red} root: The patch generated 43 new + 1764 unchanged - 8 fixed = 1807 total (was 1772) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s{color} | {color:red} The patch has 9 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s{color} | {color:red} The patch 6 line(s) with tabs. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 29s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 44m 15s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh | | git revision | master / 5add3a7 | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-8029/yetus/diff-checkstyle-common.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-8029/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-8029/yetus/diff-checkstyle-root.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-8029/yetus/whitespace-eol.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-8029/yetus/whitespace-tabs.txt | | modules | C: common ql . U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-8029/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch,
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268672#comment-16268672 ] Hive QA commented on HIVE-17856: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12899538/HIVE-17856.15.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 11422 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] (batchId=47) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dp_counter_mm] (batchId=150) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2] (batchId=154) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan] (batchId=159) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testQueueing (batchId=282) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=224) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8019/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8019/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8019/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 10 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12899538 - PreCommit-HIVE-Build > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch, > HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch, > HIVE-17856.14.patch, HIVE-17856.15.patch, HIVE-17856.2.patch, > HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17856.5.patch, > HIVE-17856.6.patch, HIVE-17856.7.patch, HIVE-17856.8.patch, HIVE-17856.9.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268635#comment-16268635 ] Hive QA commented on HIVE-17856: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 7s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 51s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 20s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 54s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 19s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 10s{color} | {color:red} common: The patch generated 2 new + 8 unchanged - 1 fixed = 10 total (was 9) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 46s{color} | {color:red} ql: The patch generated 41 new + 1756 unchanged - 7 fixed = 1797 total (was 1763) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 9 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch 6 line(s) with tabs. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 12s{color} | {color:red} The patch generated 5 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 15m 15s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh | | git revision | master / a21742e | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-8019/yetus/diff-checkstyle-common.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-8019/yetus/diff-checkstyle-ql.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-8019/yetus/whitespace-eol.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-8019/yetus/whitespace-tabs.txt | | asflicense | http://104.198.109.242/logs//PreCommit-HIVE-Build-8019/yetus/patch-asflicense-problems.txt | | modules | C: common ql U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-8019/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch, > HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch, > HIVE-17856.14.patch, HIVE-17856.15.patch, HIVE-17856.2.patch, > HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17856.5.patch, >
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268151#comment-16268151 ] Steve Yeom commented on HIVE-17856: --- Hey Sergey.. FYI, I have been closely looking at dp_counter_mm.q failure, which only happens during pre-commit tests (not on Mac laptop or our cn105 CentOS cluster). Finally I found out the root cause of the problem which is related to "union all" so will add the fix of that failure shortly but NOT a big change. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch, > HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch, > HIVE-17856.14.patch, HIVE-17856.15.patch, HIVE-17856.2.patch, > HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17856.5.patch, > HIVE-17856.6.patch, HIVE-17856.7.patch, HIVE-17856.8.patch, HIVE-17856.9.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268111#comment-16268111 ] Eugene Koifman commented on HIVE-17856: --- [~sershe], could you hold off checking this in - I'd like to look at it tomorrow. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch, > HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch, > HIVE-17856.14.patch, HIVE-17856.15.patch, HIVE-17856.2.patch, > HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17856.5.patch, > HIVE-17856.6.patch, HIVE-17856.7.patch, HIVE-17856.8.patch, HIVE-17856.9.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268069#comment-16268069 ] Sergey Shelukhin commented on HIVE-17856: - +1 one small nit, I can fix on commit if it's ok. Is dp_counter_mm failure related? The results have expired so I cannot tell what it was. Let me reattach for another run to dbl check... > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch, > HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch, > HIVE-17856.14.patch, HIVE-17856.2.patch, HIVE-17856.3.patch, > HIVE-17856.4.patch, HIVE-17856.5.patch, HIVE-17856.6.patch, > HIVE-17856.7.patch, HIVE-17856.8.patch, HIVE-17856.9.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16262910#comment-16262910 ] Hive QA commented on HIVE-17856: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12898808/HIVE-17856.14.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 11420 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] (batchId=77) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] (batchId=146) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dp_counter_mm] (batchId=150) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2] (batchId=154) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan] (batchId=165) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=102) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=224) org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema[0] (batchId=188) org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema[5] (batchId=188) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7969/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7969/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7969/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 13 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12898808 - PreCommit-HIVE-Build > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch, > HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch, > HIVE-17856.14.patch, HIVE-17856.2.patch, HIVE-17856.3.patch, > HIVE-17856.4.patch, HIVE-17856.5.patch, HIVE-17856.6.patch, > HIVE-17856.7.patch, HIVE-17856.8.patch, HIVE-17856.9.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16262854#comment-16262854 ] Hive QA commented on HIVE-17856: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 1s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 55s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 52s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 18s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 9s{color} | {color:red} common: The patch generated 2 new + 8 unchanged - 1 fixed = 10 total (was 9) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 47s{color} | {color:red} ql: The patch generated 41 new + 1755 unchanged - 7 fixed = 1796 total (was 1762) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 9 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch 6 line(s) with tabs. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 14m 54s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh | | git revision | master / c7aa7b6 | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-7969/yetus/diff-checkstyle-common.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-7969/yetus/diff-checkstyle-ql.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-7969/yetus/whitespace-eol.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-7969/yetus/whitespace-tabs.txt | | asflicense | http://104.198.109.242/logs//PreCommit-HIVE-Build-7969/yetus/patch-asflicense-problems.txt | | modules | C: common ql U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-7969/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch, > HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch, > HIVE-17856.14.patch, HIVE-17856.2.patch, HIVE-17856.3.patch, > HIVE-17856.4.patch, HIVE-17856.5.patch, HIVE-17856.6.patch, >
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259946#comment-16259946 ] Hive QA commented on HIVE-17856: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12898513/HIVE-17856.13.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 11397 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] (batchId=47) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] (batchId=77) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_range_multiorder] (batchId=7) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] (batchId=146) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dp_counter_mm] (batchId=150) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=162) org.apache.hadoop.hive.ql.TestTxnCommandsForOrcMmTable.testMmTableCompaction (batchId=272) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=276) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=223) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7925/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7925/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7925/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 9 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12898513 - PreCommit-HIVE-Build > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch, > HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch, > HIVE-17856.2.patch, HIVE-17856.3.patch, HIVE-17856.4.patch, > HIVE-17856.5.patch, HIVE-17856.6.patch, HIVE-17856.7.patch, > HIVE-17856.8.patch, HIVE-17856.9.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16259038#comment-16259038 ] Hive QA commented on HIVE-17856: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12898435/HIVE-17856.12.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 11397 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] (batchId=77) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all] (batchId=147) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] (batchId=146) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dp_counter_mm] (batchId=150) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=162) org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testInsertOverwriteWithDynamicPartition (batchId=254) org.apache.hadoop.hive.ql.TestTxnCommandsForOrcMmTable.testInsertOverwriteWithDynamicPartition (batchId=272) org.apache.hadoop.hive.ql.TestTxnCommandsForOrcMmTable.testMmTableCompaction (batchId=272) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=276) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=223) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7917/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7917/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7917/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 10 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12898435 - PreCommit-HIVE-Build > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch, > HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.2.patch, > HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17856.5.patch, > HIVE-17856.6.patch, HIVE-17856.7.patch, HIVE-17856.8.patch, HIVE-17856.9.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258304#comment-16258304 ] Steve Yeom commented on HIVE-17856: --- Most of the test failures should be related to code merge of the patch and the current code base. Working on. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch, > HIVE-17856.11.patch, HIVE-17856.2.patch, HIVE-17856.3.patch, > HIVE-17856.4.patch, HIVE-17856.5.patch, HIVE-17856.6.patch, > HIVE-17856.7.patch, HIVE-17856.8.patch, HIVE-17856.9.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258251#comment-16258251 ] Hive QA commented on HIVE-17856: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12898363/HIVE-17856.11.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 11395 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] (batchId=77) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=66) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all] (batchId=147) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] (batchId=146) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dp_counter_mm] (batchId=150) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=162) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=102) org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testInsertOverwriteForMmTable (batchId=254) org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testInsertOverwriteForPartitionedMmTable (batchId=254) org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testInsertOverwriteWithDynamicPartition (batchId=254) org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testInsertOverwriteWithUnionAll (batchId=254) org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testMultiInsertOverwriteForMmTable (batchId=254) org.apache.hadoop.hive.ql.TestTxnCommandsForOrcMmTable.testInsertOverwriteForMmTable (batchId=272) org.apache.hadoop.hive.ql.TestTxnCommandsForOrcMmTable.testInsertOverwriteForPartitionedMmTable (batchId=272) org.apache.hadoop.hive.ql.TestTxnCommandsForOrcMmTable.testInsertOverwriteWithDynamicPartition (batchId=272) org.apache.hadoop.hive.ql.TestTxnCommandsForOrcMmTable.testInsertOverwriteWithUnionAll (batchId=272) org.apache.hadoop.hive.ql.TestTxnCommandsForOrcMmTable.testMmTableCompaction (batchId=272) org.apache.hadoop.hive.ql.TestTxnCommandsForOrcMmTable.testMultiInsertOverwriteForMmTable (batchId=272) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=276) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=223) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7910/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7910/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7910/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 20 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12898363 - PreCommit-HIVE-Build > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch, > HIVE-17856.11.patch, HIVE-17856.2.patch, HIVE-17856.3.patch, > HIVE-17856.4.patch, HIVE-17856.5.patch, HIVE-17856.6.patch, > HIVE-17856.7.patch, HIVE-17856.8.patch, HIVE-17856.9.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258233#comment-16258233 ] Steve Yeom commented on HIVE-17856: --- The patch 10 has code merging issues with the current code set, Patch 11 is submitted with no compilation error. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch, > HIVE-17856.2.patch, HIVE-17856.3.patch, HIVE-17856.4.patch, > HIVE-17856.5.patch, HIVE-17856.6.patch, HIVE-17856.7.patch, > HIVE-17856.8.patch, HIVE-17856.9.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258223#comment-16258223 ] Hive QA commented on HIVE-17856: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12898359/HIVE-17856.10.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7908/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7908/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7908/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2017-11-18 21:23:36.238 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-7908/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2017-11-18 21:23:36.246 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at f1698b6 HIVE-18056: CachedStore: Have a whitelist/blacklist config to allow selective caching of tables/partitions and allow read while prewarming (Vaibhav Gumashta, Daniel Dai, reviewed by Thejas Nair, Sergey Shelukhin) + git clean -f -d Removing ql/src/test/queries/clientpositive/unionDistinct_3.q Removing ql/src/test/results/clientpositive/llap/explainanalyze_2.q.out Removing ql/src/test/results/clientpositive/llap/unionDistinct_3.q.out + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at f1698b6 HIVE-18056: CachedStore: Have a whitelist/blacklist config to allow selective caching of tables/partitions and allow read while prewarming (Vaibhav Gumashta, Daniel Dai, reviewed by Thejas Nair, Sergey Shelukhin) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2017-11-18 21:23:37.595 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch Going to apply patch with: patch -p0 patching file common/src/java/org/apache/hadoop/hive/common/JavaUtils.java patching file ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java patching file ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java patching file ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java patching file ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java patching file ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java patching file ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java patching file ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java patching file ql/src/java/org/apache/hadoop/hive/ql/parse/QBParseInfo.java patching file ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java patching file ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java patching file ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForOrcMmTable.java patching file ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java patching file ql/src/test/org/apache/hadoop/hive/ql/exec/TestUtilities.java patching file ql/src/test/results/clientpositive/mm_all.q.out + [[ maven == \m\a\v\e\n ]] + rm -rf /data/hiveptest/working/maven/org/apache/hive + mvn -B clean install -DskipTests -T 4 -q -Dmaven.repo.local=/data/hiveptest/working/maven protoc-jar: protoc version: 250, detected platform: linux/amd64 protoc-jar: executing: [/tmp/protoc3196949482387587835.exe, -I/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore, --java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources,
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16258216#comment-16258216 ] Steve Yeom commented on HIVE-17856: --- Submitted patch # 10, which is supposed to clear/fix all the unit and q tests. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch, > HIVE-17856.2.patch, HIVE-17856.3.patch, HIVE-17856.4.patch, > HIVE-17856.5.patch, HIVE-17856.6.patch, HIVE-17856.7.patch, > HIVE-17856.8.patch, HIVE-17856.9.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16256387#comment-16256387 ] Steve Yeom commented on HIVE-17856: --- Yes, Sergey.. Especially one is a multiple IOW statement and another is a statement with a regular insert and a IOW on the same table. I had to figure out which is the correct result. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.2.patch, > HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17856.5.patch, > HIVE-17856.6.patch, HIVE-17856.7.patch, HIVE-17856.8.patch, HIVE-17856.9.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16256376#comment-16256376 ] Sergey Shelukhin commented on HIVE-17856: - Which ones were incorrect? Were those the IOW tests? That would make sense > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.2.patch, > HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17856.5.patch, > HIVE-17856.6.patch, HIVE-17856.7.patch, HIVE-17856.8.patch, HIVE-17856.9.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16256212#comment-16256212 ] Steve Yeom commented on HIVE-17856: --- mm_all.q tests were taking a bit more time since the correct behavior should be verified: in order words, some of the existing expected results are not correct. And I think we have jiras related to IOW other than this jira. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.2.patch, > HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17856.5.patch, > HIVE-17856.6.patch, HIVE-17856.7.patch, HIVE-17856.8.patch, HIVE-17856.9.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254682#comment-16254682 ] Steve Yeom commented on HIVE-17856: --- I have been working on a new version of patch, which will reflect on your review and also will clear all (the consistently occurring) mm_all.q and mm_loaddata.q failures during pre-commit test. I am in the last phase. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.2.patch, > HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17856.5.patch, > HIVE-17856.6.patch, HIVE-17856.7.patch, HIVE-17856.8.patch, HIVE-17856.9.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251038#comment-16251038 ] Steve Yeom commented on HIVE-17856: --- Again.. using a unit test context seems to make it easier to clarify the errors of mm_all.q and mm_loaddata.q tests failures, regarding whether one is a failure or an indicator of fixed bug. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.2.patch, > HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17856.5.patch, > HIVE-17856.6.patch, HIVE-17856.7.patch, HIVE-17856.8.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250160#comment-16250160 ] Steve Yeom commented on HIVE-17856: --- Hey, Sergey. Great! Thanks for the info! Steve. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.2.patch, > HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17856.5.patch, > HIVE-17856.6.patch, HIVE-17856.7.patch, HIVE-17856.8.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250157#comment-16250157 ] Sergey Shelukhin commented on HIVE-17856: - [~steveyeom2017] setting Utilities.FILE_OP_LOGGER level to trace and running with just the problematic statement makes testing somewhat better for q files, it allows one to see what MM stuff is actually doing > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.2.patch, > HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17856.5.patch, > HIVE-17856.6.patch, HIVE-17856.7.patch, HIVE-17856.8.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249894#comment-16249894 ] Steve Yeom commented on HIVE-17856: --- Working on mm_all.q test failures for the latest pre-commit tests by adding the same test case scenarios into unit tests which provides easier an debugging environment. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.2.patch, > HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17856.5.patch, > HIVE-17856.6.patch, HIVE-17856.7.patch, HIVE-17856.8.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248538#comment-16248538 ] Hive QA commented on HIVE-17856: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12897174/HIVE-17856.8.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 11384 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] (batchId=47) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] (batchId=77) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=66) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_loaddata] (batchId=45) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_null] (batchId=80) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all] (batchId=147) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] (batchId=146) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dp_counter_mm] (batchId=150) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=156) org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc] (batchId=94) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi] (batchId=111) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=206) org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testInsertOverwriteWithDynamicPartition (batchId=254) org.apache.hadoop.hive.ql.TestTxnCommandsForOrcMmTable.testInsertOverwriteWithDynamicPartition (batchId=272) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=223) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7780/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7780/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7780/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 16 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12897174 - PreCommit-HIVE-Build > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.2.patch, > HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17856.5.patch, > HIVE-17856.6.patch, HIVE-17856.7.patch, HIVE-17856.8.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true",
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248513#comment-16248513 ] Hive QA commented on HIVE-17856: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12897174/HIVE-17856.8.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 11384 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] (batchId=77) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=66) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_loaddata] (batchId=45) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all] (batchId=147) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] (batchId=146) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dp_counter_mm] (batchId=150) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=157) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=156) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=102) org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc] (batchId=94) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi] (batchId=111) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=206) org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testInsertOverwriteWithDynamicPartition (batchId=254) org.apache.hadoop.hive.ql.TestTxnCommandsForOrcMmTable.testInsertOverwriteWithDynamicPartition (batchId=272) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=223) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7779/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7779/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7779/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 16 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12897174 - PreCommit-HIVE-Build > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.2.patch, > HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17856.5.patch, > HIVE-17856.6.patch, HIVE-17856.7.patch, HIVE-17856.8.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties >
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248356#comment-16248356 ] Hive QA commented on HIVE-17856: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12897144/HIVE-17856.6.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 11384 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] (batchId=47) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] (batchId=77) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=66) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_loaddata] (batchId=45) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_ppd_varchar] (batchId=11) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all] (batchId=147) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] (batchId=146) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dp_counter_mm] (batchId=150) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=156) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=102) org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc] (batchId=94) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi] (batchId=111) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=206) org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testInsertOverwriteForPartitionedMmTable (batchId=254) org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testInsertOverwriteWithDynamicPartition (batchId=254) org.apache.hadoop.hive.ql.TestTxnCommandsForOrcMmTable.testInsertOverwriteForPartitionedMmTable (batchId=272) org.apache.hadoop.hive.ql.TestTxnCommandsForOrcMmTable.testInsertOverwriteWithDynamicPartition (batchId=272) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=223) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7772/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7772/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7772/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 19 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12897144 - PreCommit-HIVE-Build > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.2.patch, > HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17856.5.patch, > HIVE-17856.6.patch, HIVE-17856.7.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key +
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16248057#comment-16248057 ] Steve Yeom commented on HIVE-17856: --- Wow the system seems to have dropped my patch 5 submitted yesterday early afternoon. Reattaching the same patch as version number 6. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.2.patch, > HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17856.5.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246762#comment-16246762 ] Steve Yeom commented on HIVE-17856: --- Right now working on mm_all.q failures in detail from the pre-commit tests. I think the run I did couple of days ago was not correct (which was successful). > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.2.patch, > HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17856.5.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246757#comment-16246757 ] Sergey Shelukhin commented on HIVE-17856: - Left some comments on RB, mostly minor. Looks like mm_all had result change last time it ran... let's see for the recent patch > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.2.patch, > HIVE-17856.3.patch, HIVE-17856.4.patch, HIVE-17856.5.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246192#comment-16246192 ] Steve Yeom commented on HIVE-17856: --- FYI, actively working on the pre-commit test failures.. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.2.patch, > HIVE-17856.3.patch, HIVE-17856.4.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246071#comment-16246071 ] Hive QA commented on HIVE-17856: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12896813/HIVE-17856.4.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 11382 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] (batchId=77) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_sort_1_23] (batchId=76) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=62) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=66) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_loaddata] (batchId=45) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all] (batchId=147) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] (batchId=146) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dp_counter_mm] (batchId=150) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=156) org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc] (batchId=94) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi] (batchId=111) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query64] (batchId=243) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=206) org.apache.hadoop.hive.ql.TestTxnCommands.testNonAcidToAcidConversion01 (batchId=288) org.apache.hadoop.hive.ql.TestTxnCommands.testTimeOutReaper (batchId=288) org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testInsertOverwriteWithDynamicPartition (batchId=254) org.apache.hadoop.hive.ql.TestTxnCommandsForOrcMmTable.testInsertOverwriteWithDynamicPartition (batchId=272) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=276) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testInsertToAcidWithUnionRemove (batchId=276) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testNoBuckets (batchId=276) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testNonAcidToAcidVectorzied (batchId=276) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testToAcidConversion02 (batchId=276) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testToAcidConversionMultiBucket (batchId=276) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=223) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7737/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7737/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7737/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 24 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12896813 - PreCommit-HIVE-Build > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.2.patch, > HIVE-17856.3.patch, HIVE-17856.4.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245561#comment-16245561 ] Hive QA commented on HIVE-17856: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12896813/HIVE-17856.4.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 11382 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks] (batchId=77) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=62) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=66) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_loaddata] (batchId=45) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all] (batchId=147) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] (batchId=146) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dp_counter_mm] (batchId=150) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=157) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=156) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=102) org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc] (batchId=94) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi] (batchId=111) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=206) org.apache.hadoop.hive.ql.TestTxnCommands.testNonAcidToAcidConversion01 (batchId=288) org.apache.hadoop.hive.ql.TestTxnCommands.testTimeOutReaper (batchId=288) org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testInsertOverwriteWithDynamicPartition (batchId=254) org.apache.hadoop.hive.ql.TestTxnCommandsForOrcMmTable.testInsertOverwriteWithDynamicPartition (batchId=272) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCTAS (batchId=276) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testInsertToAcidWithUnionRemove (batchId=276) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testNoBuckets (batchId=276) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testNonAcidToAcidVectorzied (batchId=276) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testToAcidConversion02 (batchId=276) org.apache.hadoop.hive.ql.TestTxnNoBuckets.testToAcidConversionMultiBucket (batchId=276) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=223) org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout (batchId=233) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7732/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7732/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7732/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 25 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12896813 - PreCommit-HIVE-Build > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.2.patch, > HIVE-17856.3.patch, HIVE-17856.4.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244709#comment-16244709 ] Sergey Shelukhin commented on HIVE-17856: - [~steveyeom2017] looks like the patch doesn't apply. Can you also update RB? Thanks > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.2.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244361#comment-16244361 ] Hive QA commented on HIVE-17856: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12896560/HIVE-17856.2.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7709/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7709/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7709/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2017-11-08 17:23:27.416 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-7709/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2017-11-08 17:23:27.419 + cd apache-github-source-source + git fetch origin >From https://github.com/apache/hive 1cb3f05..6f4388f master -> origin/master + git reset --hard HEAD HEAD is now at 1cb3f05 HIVE-18006: Optimize memory footprint of HLLDenseRegister (Prasanth Jayachandran reviewed by Gopal V) + git clean -f -d + git checkout master Already on 'master' Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded. (use "git pull" to update your local branch) + git reset --hard origin/master HEAD is now at 6f4388f HIVE-18007 : Address maven warnings (Ashutosh Chauhan via Alan Gates) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2017-11-08 17:23:33.563 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:1577 error: ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java: patch does not apply The patch does not appear to apply with p0, p1, or p2 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12896560 - PreCommit-HIVE-Build > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch, HIVE-17856.2.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241344#comment-16241344 ] Steve Yeom commented on HIVE-17856: --- 4) is actually a good comment. The current patch 1 has not considered IOW union all case yet. So I will check and possibly add/modify the code. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241341#comment-16241341 ] Eugene Koifman commented on HIVE-17856: --- 2) Everything in Acid relies on AcidUtils.getAcidState() - it has the logic to skip deltas that are subsumed by a base as well as paying attention to ValidTxnList. From what I've seen MM usually tries to use it's own logic for figuring out the list of dirs it should read. One caveat is that for full acid the structure is always table/partition/delta|base/buckets so it wouldn't handle the Union subdirs anyway. 4b) good question - I suspect IOW + Union All may not work for full acid > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240929#comment-16240929 ] Steve Yeom commented on HIVE-17856: --- I am also carefully checking all the test failures to find any possible code problem. 1. mm_all qtest was successful from my environment. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240925#comment-16240925 ] Steve Yeom commented on HIVE-17856: --- Hey [~sershe] Thank you for the great review. I can create a review board with the changes for 1) (that you suggested). I have added comments for each item. 1) I will check and change if relevant for this patch. Esp., patch version 2. 2) The processForWriteIds() with the "// Find the abse, created for IOW" has a call to AcidUtils.getAcidState(), which returns "dirInfo" and this class returns only valid deltas via "dirinfo.getCurrentDirectories()". Obsolete ones are in "obsolete" list from dirInfo, which are cleaned by the Compactor cleaner later. 3) I have run mm_all in my environment and it was clear and successful. 4) I will check further the items and come back with the possible code changes for item 1). > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240887#comment-16240887 ] Sergey Shelukhin commented on HIVE-17856: - Is it possible to post a review board review (https://reviews.apache.org/r/new/, project hive-git) or a pull request (whichever you prefer)? I have some minor comments that I can post there. The ones though, 1) It should be possible to remove the code that deletes "other" deltas with this patch, and probably to remove "isMatch" argument from IdPathFilter, since it was only used for that. 2) On the read side, where it adds the base (" // Find the base, created for IOW."), shouldn't it also do something to skip the deltas before the last committed base? Maybe there's logic in ACID somewhere; it may be hard to reuse directly given a different layout, but perhaps it can be duplicated or modified cc [~ekoifman] 3) Tests appear to have failed (mm_all). 4) getValidPartitionsInPath passes in false, although it is possible to have dynamic partitioning with IOW. 4a) In general though, one txn can only create either base or delta in each given location (or multiple of one type), as far as I understand. So it may be possible to change the filter/etc. to find both by txn number, instead of passing the flag and only finding one type. Then, we can error out if both are present at once. 4b) Actually [~ekoifman], that makes me wonder, about IOW with union work for ACID? ACID union insert creates two deltas; does ACID IOW with union create two base directories? > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240628#comment-16240628 ] Steve Yeom commented on HIVE-17856: --- Hi [~sershe] can you review the 01.patch? Thank you, Steve. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240483#comment-16240483 ] Hive QA commented on HIVE-17856: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12896162/HIVE-17856.1.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 11361 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[index_auto_mult_tables] (batchId=84) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=62) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=66) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all] (batchId=147) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1] (batchId=146) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=156) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=102) org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc] (batchId=94) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi] (batchId=111) org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut (batchId=206) org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion02 (batchId=274) org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion3 (batchId=274) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion02 (batchId=284) org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion3 (batchId=284) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testAmPoolInteractions (batchId=281) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testApplyPlanQpChanges (batchId=281) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testApplyPlanUserMapping (batchId=281) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testAsyncSessionInitFailures (batchId=281) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testClusterFractions (batchId=281) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testDestroyAndReturn (batchId=281) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testQueueing (batchId=281) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testReopen (batchId=281) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testReuse (batchId=281) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testReuseWithDifferentPool (batchId=281) org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testReuseWithQueueing (batchId=281) org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints (batchId=223) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7660/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7660/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7660/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 26 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12896162 - PreCommit-HIVE-Build > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240017#comment-16240017 ] Steve Yeom commented on HIVE-17856: --- patch01 has implemented the three items above for a regular and partitioned Micro-Managed (MM) table. Also tested with ORC file format and default text format as two representatives of the all the formats. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > Attachments: HIVE-17856.1.patch > > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236159#comment-16236159 ] Eugene Koifman commented on HIVE-17856: --- It looks like LOAD DATA with OVERWRITE for MM also does the non-acid delete + write operation. I think it would make sense to make it behave the same way as IOW and use a base_N directory. I filed HIVE-17970 to track it. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom >Priority: Major > Labels: mm-gap-1 > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16219558#comment-16219558 ] Steve Yeom commented on HIVE-17856: --- Thanks for the info. Eugene. FYI, The above comments by Eugene is an item of the current plan for MM for this jira : 1. Keep delta dirs (for the open transaction reader to keep access the delta dirs). 2. base dir processing 2.1 Writer for IOW statement creates base dir for new rows. 2.2 Subsequent reader for a MM table after successful IOW commit shall use base dir only when it encounters a complete base dir. 3. Cleaner of the compactor cleans as in the case of Full ACID tables. I am actively working on the items. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16219526#comment-16219526 ] Eugene Koifman commented on HIVE-17856: --- The general idea of how full Acid IOW works is given Insert into T select * from Foo we create a new /warehouse/db/T/base_N where N is the current txn id in which the above SQL is running. Then, the every acid reader uses AcidUtils.getAcidState(Path rootOfT,) to create a AcidUtils.Directory object which will have base_N/ files but will put all other files in Directory.getObsolete(). This way it will only see what is written to base_N. All Acid readers use getAcidState() to filter the input file list wrt ValidTxnList for each operation. It would make sense for MM to follow the same approach. I don't think the current MM code is quite there yet. This would also make the Cleaner understand that deltas from before N can be removed (but only after all the readers for them went away) > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215702#comment-16215702 ] Steve Yeom commented on HIVE-17856: --- FYI. As [~sershe] mentioned, I am now focusing on IOW on MM table with the test cases used for ACID IOW patch, checking the comments and applying possible fixes. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213569#comment-16213569 ] Steve Yeom commented on HIVE-17856: --- Talked with Sergey. Sergey indicated that the current IOW execution code/path for MM tables is not ACID compliant because first it deletes directories as mentioned above by an IOW writer/updater. For one case, to support Snapshot isolation by concurrent open reader transactions, we should not delete them with the writers when possibly open transactions may exist, but through a mechanism which is sync with open transactions, as in the case of ACID tables, where delta directories are merged/deleted by compactors with TxnList with open and aborted xacts. > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[ https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16213453#comment-16213453 ] Sergey Shelukhin commented on HIVE-17856: - Ok I noticed this while looking at the union bug. I think the reason IOW "worked for me" in some of the tests and works for [~steveyeom2017] when he runs the above test is that it's still using old MM logic that is not ACID compliant. This call in Hive.java: {noformat} deleteOldPathForReplace(newPartPath, oldPartPath, getConf(), isAutoPurge, new JavaUtils.IdPathFilter(txnId, stmtId, false, true), true, tbl.isStoredAsSubDirectories() ? tbl.getSkewedColNames().size() : 0); {noformat} Deletes old delta directories by means of IdPathFilter with 3rd arg being false, which means, return ALL delta directories that don't match txnId. So, all the other data in the table gets nuked. This is the implementation is incorrect for ACID integration. So, delete for MM table codepath needs to be removed (easy to find by looking where IdPathFilter is used with isMatch == false, meaning "find every txn except this one"). Then, IOW will probably stop working w.r.t. "overwrite" because old deltas will stick around. After that Eugene can comment on where and how ACID uses base directories to implement IOW. I suspect that in IOW case, it will be as simple as instead of creating delta_ dir for output, creating base_ directory; and that would be enough for all the logic shared with ACID, e.g. compactor, to handle it correctly. The code that finds what to read in HiveInputFormat would also need to be updated to take committed base-s into account. But I am not familiar with ACID IOW, so it may not be as simple. cc [~hagleitn] [~steveyeom2017] [~ekoifman] > MM tables - IOW is not ACID compliant > - > > Key: HIVE-17856 > URL: https://issues.apache.org/jira/browse/HIVE-17856 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Steve Yeom > Labels: mm-gap-1 > > The following tests were removed from mm_all during "integration"... I should > have never allowed such manner of intergration. > MM logic should have been kept intact until ACID logic could catch up. Alas, > here we are. > {noformat} > drop table iow0_mm; > create table iow0_mm(key int) tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow0_mm select key from intermediate; > insert into table iow0_mm select key + 1 from intermediate; > select * from iow0_mm order by key; > insert overwrite table iow0_mm select key + 2 from intermediate; > select * from iow0_mm order by key; > drop table iow0_mm; > drop table iow1_mm; > create table iow1_mm(key int) partitioned by (key2 int) > tblproperties("transactional"="true", > "transactional_properties"="insert_only"); > insert overwrite table iow1_mm partition (key2) > select key as k1, key from intermediate union all select key as k1, key from > intermediate; > insert into table iow1_mm partition (key2) > select key + 1 as k1, key from intermediate union all select key as k1, key > from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key from intermediate union all select key + 4 as k1, > key from intermediate; > select * from iow1_mm order by key, key2; > insert overwrite table iow1_mm partition (key2) > select key + 3 as k1, key + 3 from intermediate union all select key + 2 as > k1, key + 2 from intermediate; > select * from iow1_mm order by key, key2; > drop table iow1_mm; > {noformat} > {noformat} > drop table simple_mm; > create table simple_mm(key int) stored as orc tblproperties > ("transactional"="true", "transactional_properties"="insert_only"); > insert into table simple_mm select key from intermediate; > -insert overwrite table simple_mm select key from intermediate; > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)