[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15131437#comment-15131437 ] Eugene Koifman commented on HIVE-11716: --- branch-1: caf4b516b74201633e25d99bfcf5d69eacc8a617 > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Fix For: 2.1.0 > > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch, > HIVE-11716.3.patch, HIVE-11716.4.patch, HIVE-11716.5.patch, > HIVE-11716.6.patch, HIVE-11716.branch-1.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15131030#comment-15131030 ] Eugene Koifman commented on HIVE-11716: --- committed to master https://github.com/apache/hive/commit/f2e9edbc2d81e32f5da700cd7024959184aeaebf > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch, > HIVE-11716.3.patch, HIVE-11716.4.patch, HIVE-11716.5.patch, HIVE-11716.6.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15130958#comment-15130958 ] Eugene Koifman commented on HIVE-11716: --- [~wzheng] could you make a branch-1 patch > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch, > HIVE-11716.3.patch, HIVE-11716.4.patch, HIVE-11716.5.patch, HIVE-11716.6.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15130854#comment-15130854 ] Wei Zheng commented on HIVE-11716: -- TestHiveMetaStorePartitionSpecs passed locally for me without any failures. My change shouldn't have impact on that test. > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch, > HIVE-11716.3.patch, HIVE-11716.4.patch, HIVE-11716.5.patch, HIVE-11716.6.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15130736#comment-15130736 ] Hive QA commented on HIVE-11716: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12785860/HIVE-11716.6.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10034 tests executed *Failed tests:* {noformat} TestMiniTezCliDriver-vector_decimal_2.q-schema_evol_text_fetchwork_table.q-vector_null_projection.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping org.apache.hive.jdbc.TestSSL.testSSLVersion org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6852/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6852/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6852/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12785860 - PreCommit-HIVE-TRUNK-Build > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch, > HIVE-11716.3.patch, HIVE-11716.4.patch, HIVE-11716.5.patch, HIVE-11716.6.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129337#comment-15129337 ] Eugene Koifman commented on HIVE-11716: --- +1 pending tests > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch, > HIVE-11716.3.patch, HIVE-11716.4.patch, HIVE-11716.5.patch, HIVE-11716.6.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15127702#comment-15127702 ] Wei Zheng commented on HIVE-11716: -- Good advice. Will follow up with new patch. > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch, > HIVE-11716.3.patch, HIVE-11716.4.patch, HIVE-11716.5.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15127452#comment-15127452 ] Eugene Koifman commented on HIVE-11716: --- ErrorMsg.TXNMGR_NOT_ACID - this should be a parametrized message. At minimum, it should include db_name.table_name. Perhaps it should also include first 10 or 20 chars of the command itself to make easier to read the logs down the road. I don't think SessionState.initTxnMgr() works long term. What if we add BetterTxnManger later? I would, in addition to txnMgr also add txnMgrName to remember the last property name or perhaps better add HiveTxnManager.getTxnManagerName() so that you can query currently installed manager for its name. Nit: In TestDbTxnManager2.testDummyTxnManagerOnAcidTable(). Instead of 10265, etc could you use ErrorMsg.TXNMGR_NOT_ACID, etc > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch, > HIVE-11716.3.patch, HIVE-11716.4.patch, HIVE-11716.5.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15126748#comment-15126748 ] Wei Zheng commented on HIVE-11716: -- Mismatches are unrelated > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch, > HIVE-11716.3.patch, HIVE-11716.4.patch, HIVE-11716.5.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15125238#comment-15125238 ] Hive QA commented on HIVE-11716: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12785122/HIVE-11716.5.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10032 tests executed *Failed tests:* {noformat} TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery org.apache.hive.jdbc.TestSSL.testSSLVersion org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6809/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6809/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6809/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12785122 - PreCommit-HIVE-TRUNK-Build > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch, > HIVE-11716.3.patch, HIVE-11716.4.patch, HIVE-11716.5.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123118#comment-15123118 ] Wei Zheng commented on HIVE-11716: -- [~alangates] You're right. The mismatch is caused by my change, but in a good way :) Basically the test result mismatch is like this: {code} < FAILED: SemanticException [Error 10297]: Attempt to do update or delete on table default.acid_notbucketed that does not use an AcidOutputFormat or is not bucketed --- > FAILED: SemanticException [Error 10297]: Attempt to do update or delete on > table acid_notbucketed that does not use an AcidOutputFormat or is not > bucketed {code} Without patch the error was thrown by SemanticAnalyzer#checkAcidConstraints, there we have {code} if (table.getNumBuckets() < 1) { throw new SemanticException(ErrorMsg.ACID_OP_ON_NONACID_TABLE, table.getTableName()); } {code} where table.getTableName() just returns tablename acid_notbucketed. With patch the error was caught earlier and thrown by SemanticAnalyzer#getMetaData() {code} // Disallow update and delete on non-acid tables if ((updating() || deleting()) && !isAcid && isTableWrittenTo) { //isTableWrittenTo: delete from acidTbl where a in (select id from nonAcidTable) //so only assert this if we are actually writing to this table // isAcidTable above also checks for whether we are using an acid compliant // transaction manager. But that has already been caught in // UpdateDeleteSemanticAnalyzer, so if we are updating or deleting and getting nonAcid // here, it means the table itself doesn't support it. throw new SemanticException(ErrorMsg.ACID_OP_ON_NONACID_TABLE, tab_name); } {code} where tab_name is default.acid_notbucketed Here isAcid flag was true without patch, which is wrong, because isAcidTable() has a bug as it only checked 'transactional'='true', but didn't check whether the table is bucketed or whether it's ORC. The patch fixed isAcidTable(), so isAcid is false now, thus we can trigger this logic. > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch, > HIVE-11716.3.patch, HIVE-11716.4.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122676#comment-15122676 ] Alan Gates commented on HIVE-11716: --- Those tests don't generally fail and given that you're working exactly in that space I wouldn't ignore those failures. > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch, > HIVE-11716.3.patch, HIVE-11716.4.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122616#comment-15122616 ] Wei Zheng commented on HIVE-11716: -- Both delete_not_bucketed.q and update_not_bucketed.q passed locally for me. > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch, > HIVE-11716.3.patch, HIVE-11716.4.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122531#comment-15122531 ] Hive QA commented on HIVE-11716: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12784780/HIVE-11716.4.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10022 tests executed *Failed tests:* {noformat} TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_delete_not_bucketed org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_update_not_bucketed org.apache.hive.jdbc.TestSSL.testSSLVersion {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6782/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6782/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6782/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12784780 - PreCommit-HIVE-TRUNK-Build > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch, > HIVE-11716.3.patch, HIVE-11716.4.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120102#comment-15120102 ] Hive QA commented on HIVE-11716: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12784545/HIVE-11716.3.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10035 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import org.apache.hive.jdbc.TestSSL.testSSLVersion {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6762/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6762/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6762/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12784545 - PreCommit-HIVE-TRUNK-Build > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch, > HIVE-11716.3.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15118194#comment-15118194 ] Eugene Koifman commented on HIVE-11716: --- in that case you are right, you should keep track of hive.txn.manager field > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15118183#comment-15118183 ] Wei Zheng commented on HIVE-11716: -- But I think set hive.txn.manager= ... will make changes to conf, and we will only know this if we do a comparison between confs > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15118166#comment-15118166 ] Eugene Koifman commented on HIVE-11716: --- I think it should only initialized if it is not already set or if there is explicit "set hive.txn.manager..." command. In the later case, the previous instance must be shutdown first. > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15118165#comment-15118165 ] Wei Zheng commented on HIVE-11716: -- Unless I compare the original conf and the passed-in conf, and only initialize a new TxnManger if hive.txn.manager field is changed? > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15118160#comment-15118160 ] Wei Zheng commented on HIVE-11716: -- Sounds good. Now users can change the TxnManager for real. But one issue is, initialize a TxnManager for every query might be a little slower than reusing the same one in a session (because we will check conf every time and init a TxnManager) > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15118150#comment-15118150 ] Wei Zheng commented on HIVE-11716: -- How about "This command is not allowed on an ACID table with a non-ACID transaction manager"? > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15118134#comment-15118134 ] Eugene Koifman commented on HIVE-11716: --- Nit: "Cannot query an ACID table using a non-ACID transaction manager" - maybe should read "This command is not allowed on an ACID " > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15118097#comment-15118097 ] Eugene Koifman commented on HIVE-11716: --- one other comment: now that you've implemented logic to raise an error when Acid table is accessed from non-acid session, it's seems like it's safe to allow user to change TxnManager within a Session. Since it can no longer do any harm. So maybe using HIV_IN_TEST isn't needed... > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15118091#comment-15118091 ] Eugene Koifman commented on HIVE-11716: --- I think you need to use AcidUtil.isTablePropertyTransactional() to see if table is transactional rather than AcidUtils.isAcidTable(tab). The later will return false for table that was non-acid-to-acid upgraded but w/o major compaction having been run. Maybe even better to define a new isAcidTable(Table t) method which checks that it's bucketed, implements AcidInput/AcidOutputFormat and has the transactional=true. Also, SemanticAnalyzer.isAcidTable() should delegate to new method (or be removed) or at minimum not check the which TxnManager is installed > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15117995#comment-15117995 ] Hive QA commented on HIVE-11716: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12784367/HIVE-11716.1.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6749/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6749/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6749/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]] + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6749/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + cd apache-github-source-source + git fetch origin >From https://github.com/apache/hive eb4a164..27172bc master -> origin/master + git reset --hard HEAD HEAD is now at eb4a164 HIVE-12905 : Issue with mapjoin in tez under certain conditions (Vikram Dixit K, reviewed by Sergey Shelukhin) + git clean -f -d Removing metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetChangeVersionRequest.java Removing metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetChangeVersionResult.java Removing metastore/src/model/org/apache/hadoop/hive/metastore/model/MChangeVersion.java + git checkout master Already on 'master' Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded. + git reset --hard origin/master HEAD is now at 27172bc HIVE-12889: Support COUNT(DISTINCT) for partitioning query. (Aihua Xu, reviewed by Szehon Ho) + git merge --ff-only origin/master Already up-to-date. + git gc + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12784367 - PreCommit-HIVE-TRUNK-Build > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15116837#comment-15116837 ] Wei Zheng commented on HIVE-11716: -- I will follow up with HIVE-12441 once this one is done > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15116835#comment-15116835 ] Wei Zheng commented on HIVE-11716: -- This JIRA solves the problems below: * Explicitly disallow switching hive.txn.manager (This was not allowed always, but we didn't throw out error, and SET command gives people impression that this entry can be changed because it does show the new value, which is misleading) * For queries against ACID table with a transaction manager that doesn't support ACID, we raise errors. This applies to SELECT, INSERT INTO, INSERT OVERWRITE, UPDATE and DELETE. > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > Attachments: HIVE-11716.1.patch > > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[ https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15113039#comment-15113039 ] Eugene Koifman commented on HIVE-11716: --- HIVE-12441 is related since in both cases we need to walk the plan and see if there are any ACID tables in it (read or written) > Reading ACID table from non-acid session should raise an error > -- > > Key: HIVE-11716 > URL: https://issues.apache.org/jira/browse/HIVE-11716 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Critical > > if someone performs a delete using DbTxnManager and than starts a new session > with DummyTxnManager, then previously successfully deleted data will show up. > Once the logic to identify all acid tables being read is there, make sure that > Driver.recordValidTxns() called iff there ACID tables in the query. > We should also not allow writing to ACID table from non-acid session -- This message was sent by Atlassian JIRA (v6.3.4#6332)