[jira] [Commented] (HIVE-15077) Acid LockManager is unfair
[
https://issues.apache.org/jira/browse/HIVE-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374883#comment-16374883
]
Alan Gates commented on HIVE-15077:
---
+1. The previous code was trying to save time by only comparing against some
of the locks, but since the sort or was changed in HIVE-10242 that no longer
works.
> Acid LockManager is unfair
> --
>
> Key: HIVE-15077
> URL: https://issues.apache.org/jira/browse/HIVE-15077
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-15077.02.patch
>
>
> HIVE-10242 made the acid LM unfair.
> In TxnHandler.checkLock(), suppose we are trying to acquire SR5 (the number
> is extLockId).
> Then
> LockInfo[] locks = lockSet.toArray(new LockInfo[lockSet.size()]);
> may look like this (all explicitly listed locks are in Waiting state)
> {, SR5 SW3 X4}
> So the algorithm will find SR5 in the list and start looking backwards (to
> the left).
> According to IDs, SR5 should wait for X4 to be granted but X4 won't even be
> examined and so SR5 may be granted.
> Theoretically, this could cause starvation.
> The query that generates the list already has
> query.append(" and hl_lock_ext_id <= ").append(extLockId);
> but it should use "<" rather than "<=" to exclude the locks being checked
> from "locks" list which will make the algorithm look at all locks "in front"
> of a given lock.
> Here is an example (add to TestDbTxnManager2)
> {noformat}
> @Test
> public void testFairness2() throws Exception {
> dropTable(new String[]{"T7"});
> CommandProcessorResponse cpr = driver.run("create table if not exists T7
> (a int) partitioned by (p int) stored as orc TBLPROPERTIES
> ('transactional'='true')");
> checkCmdOnDriver(cpr);
> checkCmdOnDriver(driver.run("insert into T7 partition(p)
> values(1,1),(1,2)"));//create 2 partitions
> cpr = driver.compileAndRespond("select a from T7 ");
> checkCmdOnDriver(cpr);
> txnMgr.acquireLocks(driver.getPlan(), ctx, "Fifer");//gets S lock on T7
> HiveTxnManager txnMgr2 =
> TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
> swapTxnManager(txnMgr2);
> cpr = driver.compileAndRespond("alter table T7 drop partition (p=1)");
> checkCmdOnDriver(cpr);
> //tries to get X lock on T7.p=1 and gets Waiting state
> LockState lockState = ((DbTxnManager)
> txnMgr2).acquireLocks(driver.getPlan(), ctx, "Fiddler", false);
> List locks = getLocks();
> Assert.assertEquals("Unexpected lock count", 4, locks.size());
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> null, locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=1", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=2", locks);
> checkLock(LockType.EXCLUSIVE, LockState.WAITING, "default", "T7", "p=1",
> locks);
> HiveTxnManager txnMgr3 =
> TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
> swapTxnManager(txnMgr3);
> //this should block behind the X lock on T7.p=1
> cpr = driver.compileAndRespond("select a from T7");
> checkCmdOnDriver(cpr);
> txnMgr3.acquireLocks(driver.getPlan(), ctx, "Fifer");//gets S lock on T6
> locks = getLocks();
> Assert.assertEquals("Unexpected lock count", 7, locks.size());
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> null, locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=1", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=2", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> null, locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=1", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=2", locks);
> checkLock(LockType.EXCLUSIVE, LockState.WAITING, "default", "T7", "p=1",
> locks);
> }
> {noformat}
> The 2nd {{locks = getLocks();}} output shows that all locks for the 2nd
> {{select * from T7}} are all acquired while they should block behind the X
> lock to be fair.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-15077) Acid LockManager is unfair
[
https://issues.apache.org/jira/browse/HIVE-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373307#comment-16373307
]
Eugene Koifman commented on HIVE-15077:
---
I did make it unfair in HIVE-10242 but that was not intentional. (To ensure
fairness the requested lock is only checked against locks with smaller
extLockId. The problem was that HIVE-10242 made it such that not all locks
with smaller extLockId that should have been checked, were checked.) I had to
sort locks by weight in HIVE-10242 because the jumpTable doesn't have any info
about the resource and so the logic that says "if looking to get S lock and see
acquired S lock in front, acquire" doesn't work because the S lock in front may
be on a different resource. The issue this caused is demonstrated in
\{{testFairness2()}} in the Description of this ticket.
So I'm aiming for a lock manager that is fair, correct and not more strict than
necessary. The last part is a work in progress.
What I think we really need is the use of Intention locks. That way what you
are suggesting is possible. Right now we just "infer" a lock (which is not
physically there) up/down the resource hierarchy based on the lock that is
actually asked for (and of the same type). This way you'd only have to compare
locks on resources with the same path. This is a bigger change.
> Acid LockManager is unfair
> --
>
> Key: HIVE-15077
> URL: https://issues.apache.org/jira/browse/HIVE-15077
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-15077.02.patch
>
>
> HIVE-10242 made the acid LM unfair.
> In TxnHandler.checkLock(), suppose we are trying to acquire SR5 (the number
> is extLockId).
> Then
> LockInfo[] locks = lockSet.toArray(new LockInfo[lockSet.size()]);
> may look like this (all explicitly listed locks are in Waiting state)
> {, SR5 SW3 X4}
> So the algorithm will find SR5 in the list and start looking backwards (to
> the left).
> According to IDs, SR5 should wait for X4 to be granted but X4 won't even be
> examined and so SR5 may be granted.
> Theoretically, this could cause starvation.
> The query that generates the list already has
> query.append(" and hl_lock_ext_id <= ").append(extLockId);
> but it should use "<" rather than "<=" to exclude the locks being checked
> from "locks" list which will make the algorithm look at all locks "in front"
> of a given lock.
> Here is an example (add to TestDbTxnManager2)
> {noformat}
> @Test
> public void testFairness2() throws Exception {
> dropTable(new String[]{"T7"});
> CommandProcessorResponse cpr = driver.run("create table if not exists T7
> (a int) partitioned by (p int) stored as orc TBLPROPERTIES
> ('transactional'='true')");
> checkCmdOnDriver(cpr);
> checkCmdOnDriver(driver.run("insert into T7 partition(p)
> values(1,1),(1,2)"));//create 2 partitions
> cpr = driver.compileAndRespond("select a from T7 ");
> checkCmdOnDriver(cpr);
> txnMgr.acquireLocks(driver.getPlan(), ctx, "Fifer");//gets S lock on T7
> HiveTxnManager txnMgr2 =
> TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
> swapTxnManager(txnMgr2);
> cpr = driver.compileAndRespond("alter table T7 drop partition (p=1)");
> checkCmdOnDriver(cpr);
> //tries to get X lock on T7.p=1 and gets Waiting state
> LockState lockState = ((DbTxnManager)
> txnMgr2).acquireLocks(driver.getPlan(), ctx, "Fiddler", false);
> List locks = getLocks();
> Assert.assertEquals("Unexpected lock count", 4, locks.size());
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> null, locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=1", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=2", locks);
> checkLock(LockType.EXCLUSIVE, LockState.WAITING, "default", "T7", "p=1",
> locks);
> HiveTxnManager txnMgr3 =
> TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
> swapTxnManager(txnMgr3);
> //this should block behind the X lock on T7.p=1
> cpr = driver.compileAndRespond("select a from T7");
> checkCmdOnDriver(cpr);
> txnMgr3.acquireLocks(driver.getPlan(), ctx, "Fifer");//gets S lock on T6
> locks = getLocks();
> Assert.assertEquals("Unexpected lock count", 7, locks.size());
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> null, locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=1", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=2", locks);
> checkLock(LockType.SHARED_READ,
[jira] [Commented] (HIVE-15077) Acid LockManager is unfair
[
https://issues.apache.org/jira/browse/HIVE-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373155#comment-16373155
]
Alan Gates commented on HIVE-15077:
---
I haven't fully understood this patch yet, but I'm confused by its intent. So
some questions to clarify that first.
Pre HIVE-10242 the lock manager was fair but too restrictive. To solve that
you made it unfair in favor of shared locks. There are reasonable arguments
for either option. But the choice seems binary. What middle ground are you
aiming for?
One thing I can see that would work is changing the locking mechanism to
understand the database.table.partition hierarchy. In your 'insert overwrite'
example it seems to me that what is broken is that the xlock on DB.T1 extends
up to DB. It should be a xlock on T1 and a slock on DB.
> Acid LockManager is unfair
> --
>
> Key: HIVE-15077
> URL: https://issues.apache.org/jira/browse/HIVE-15077
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-15077.02.patch
>
>
> HIVE-10242 made the acid LM unfair.
> In TxnHandler.checkLock(), suppose we are trying to acquire SR5 (the number
> is extLockId).
> Then
> LockInfo[] locks = lockSet.toArray(new LockInfo[lockSet.size()]);
> may look like this (all explicitly listed locks are in Waiting state)
> {, SR5 SW3 X4}
> So the algorithm will find SR5 in the list and start looking backwards (to
> the left).
> According to IDs, SR5 should wait for X4 to be granted but X4 won't even be
> examined and so SR5 may be granted.
> Theoretically, this could cause starvation.
> The query that generates the list already has
> query.append(" and hl_lock_ext_id <= ").append(extLockId);
> but it should use "<" rather than "<=" to exclude the locks being checked
> from "locks" list which will make the algorithm look at all locks "in front"
> of a given lock.
> Here is an example (add to TestDbTxnManager2)
> {noformat}
> @Test
> public void testFairness2() throws Exception {
> dropTable(new String[]{"T7"});
> CommandProcessorResponse cpr = driver.run("create table if not exists T7
> (a int) partitioned by (p int) stored as orc TBLPROPERTIES
> ('transactional'='true')");
> checkCmdOnDriver(cpr);
> checkCmdOnDriver(driver.run("insert into T7 partition(p)
> values(1,1),(1,2)"));//create 2 partitions
> cpr = driver.compileAndRespond("select a from T7 ");
> checkCmdOnDriver(cpr);
> txnMgr.acquireLocks(driver.getPlan(), ctx, "Fifer");//gets S lock on T7
> HiveTxnManager txnMgr2 =
> TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
> swapTxnManager(txnMgr2);
> cpr = driver.compileAndRespond("alter table T7 drop partition (p=1)");
> checkCmdOnDriver(cpr);
> //tries to get X lock on T7.p=1 and gets Waiting state
> LockState lockState = ((DbTxnManager)
> txnMgr2).acquireLocks(driver.getPlan(), ctx, "Fiddler", false);
> List locks = getLocks();
> Assert.assertEquals("Unexpected lock count", 4, locks.size());
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> null, locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=1", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=2", locks);
> checkLock(LockType.EXCLUSIVE, LockState.WAITING, "default", "T7", "p=1",
> locks);
> HiveTxnManager txnMgr3 =
> TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
> swapTxnManager(txnMgr3);
> //this should block behind the X lock on T7.p=1
> cpr = driver.compileAndRespond("select a from T7");
> checkCmdOnDriver(cpr);
> txnMgr3.acquireLocks(driver.getPlan(), ctx, "Fifer");//gets S lock on T6
> locks = getLocks();
> Assert.assertEquals("Unexpected lock count", 7, locks.size());
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> null, locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=1", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=2", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> null, locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=1", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=2", locks);
> checkLock(LockType.EXCLUSIVE, LockState.WAITING, "default", "T7", "p=1",
> locks);
> }
> {noformat}
> The 2nd {{locks = getLocks();}} output shows that all locks for the 2nd
> {{select * from T7}} are all acquired while they should block behind the X
> lock to be fair.
--
This message was sent by
[jira] [Commented] (HIVE-15077) Acid LockManager is unfair
[
https://issues.apache.org/jira/browse/HIVE-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366255#comment-16366255
]
Eugene Koifman commented on HIVE-15077:
---
no related failures
[~alangates] could you review please
> Acid LockManager is unfair
> --
>
> Key: HIVE-15077
> URL: https://issues.apache.org/jira/browse/HIVE-15077
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-15077.02.patch
>
>
> HIVE-10242 made the acid LM unfair.
> In TxnHandler.checkLock(), suppose we are trying to acquire SR5 (the number
> is extLockId).
> Then
> LockInfo[] locks = lockSet.toArray(new LockInfo[lockSet.size()]);
> may look like this (all explicitly listed locks are in Waiting state)
> {, SR5 SW3 X4}
> So the algorithm will find SR5 in the list and start looking backwards (to
> the left).
> According to IDs, SR5 should wait for X4 to be granted but X4 won't even be
> examined and so SR5 may be granted.
> Theoretically, this could cause starvation.
> The query that generates the list already has
> query.append(" and hl_lock_ext_id <= ").append(extLockId);
> but it should use "<" rather than "<=" to exclude the locks being checked
> from "locks" list which will make the algorithm look at all locks "in front"
> of a given lock.
> Here is an example (add to TestDbTxnManager2)
> {noformat}
> @Test
> public void testFairness2() throws Exception {
> dropTable(new String[]{"T7"});
> CommandProcessorResponse cpr = driver.run("create table if not exists T7
> (a int) partitioned by (p int) stored as orc TBLPROPERTIES
> ('transactional'='true')");
> checkCmdOnDriver(cpr);
> checkCmdOnDriver(driver.run("insert into T7 partition(p)
> values(1,1),(1,2)"));//create 2 partitions
> cpr = driver.compileAndRespond("select a from T7 ");
> checkCmdOnDriver(cpr);
> txnMgr.acquireLocks(driver.getPlan(), ctx, "Fifer");//gets S lock on T7
> HiveTxnManager txnMgr2 =
> TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
> swapTxnManager(txnMgr2);
> cpr = driver.compileAndRespond("alter table T7 drop partition (p=1)");
> checkCmdOnDriver(cpr);
> //tries to get X lock on T7.p=1 and gets Waiting state
> LockState lockState = ((DbTxnManager)
> txnMgr2).acquireLocks(driver.getPlan(), ctx, "Fiddler", false);
> List locks = getLocks();
> Assert.assertEquals("Unexpected lock count", 4, locks.size());
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> null, locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=1", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=2", locks);
> checkLock(LockType.EXCLUSIVE, LockState.WAITING, "default", "T7", "p=1",
> locks);
> HiveTxnManager txnMgr3 =
> TxnManagerFactory.getTxnManagerFactory().getTxnManager(conf);
> swapTxnManager(txnMgr3);
> //this should block behind the X lock on T7.p=1
> cpr = driver.compileAndRespond("select a from T7");
> checkCmdOnDriver(cpr);
> txnMgr3.acquireLocks(driver.getPlan(), ctx, "Fifer");//gets S lock on T6
> locks = getLocks();
> Assert.assertEquals("Unexpected lock count", 7, locks.size());
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> null, locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=1", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=2", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> null, locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=1", locks);
> checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T7",
> "p=2", locks);
> checkLock(LockType.EXCLUSIVE, LockState.WAITING, "default", "T7", "p=1",
> locks);
> }
> {noformat}
> The 2nd {{locks = getLocks();}} output shows that all locks for the 2nd
> {{select * from T7}} are all acquired while they should block behind the X
> lock to be fair.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-15077) Acid LockManager is unfair
[
https://issues.apache.org/jira/browse/HIVE-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365017#comment-16365017
]
Hive QA commented on HIVE-15077:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12910632/HIVE-15077.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 13101 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=78)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_opt_shuffle_serde]
(batchId=179)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=121)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query1]
(batchId=250)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.TestAcidTableSetup.testTransactionalValidation
(batchId=223)
org.apache.hadoop.hive.metastore.client.TestAddPartitionsFromPartSpec.testAddPartitionSpecChangeRootPathToNull[Embedded]
(batchId=205)
org.apache.hadoop.hive.ql.TestAcidOnTez.testGetSplitsLocks (batchId=224)
org.apache.hadoop.hive.ql.metadata.TestHive.testTable (batchId=278)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=187)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.alterIndex
(batchId=242)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.createIndex
(batchId=242)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropIndex
(batchId=242)
org.apache.hive.hcatalog.pig.TestSequenceFileHCatStorer.testWriteSmallint
(batchId=192)
org.apache.hive.jdbc.TestJdbcWithMiniLlap.testLlapInputFormatEndToEnd
(batchId=235)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9219/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9219/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9219/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 27 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12910632 - PreCommit-HIVE-Build
> Acid LockManager is unfair
> --
>
> Key: HIVE-15077
> URL: https://issues.apache.org/jira/browse/HIVE-15077
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-15077.02.patch
>
>
> HIVE-10242 made the acid LM unfair.
> In TxnHandler.checkLock(), suppose we are trying to acquire SR5 (the number
> is extLockId).
> Then
> LockInfo[] locks = lockSet.toArray(new LockInfo[lockSet.size()]);
> may look like this (all explicitly listed locks are in Waiting state)
> {, SR5 SW3 X4}
> So the algorithm will find SR5 in the list and start looking backwards (to
> the left).
> According to IDs, SR5 should wait for X4 to be granted but X4 won't even be
> examined and so SR5 may be granted.
> Theoretically, this could cause starvation.
> The query that generates the list already has
> query.append(" and hl_lock_ext_id <= ").append(extLockId);
> but it should use "<" rather than "<=" to exclude the locks being checked
> from "locks" list which will make the algorithm look at all locks "in front"
> of a given lock.
> Here is an example (add to TestDbTxnManager2)
>
[jira] [Commented] (HIVE-15077) Acid LockManager is unfair
[
https://issues.apache.org/jira/browse/HIVE-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364990#comment-16364990
]
Hive QA commented on HIVE-15077:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 7 new + 426 unchanged - 0
fixed = 433 total (was 426) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} standalone-metastore: The patch generated 2 new + 510
unchanged - 3 fixed = 512 total (was 513) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a2d22b4 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9219/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9219/yetus/diff-checkstyle-standalone-metastore.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9219/yetus/patch-asflicense-problems.txt
|
| modules | C: ql standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-9219/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Acid LockManager is unfair
> --
>
> Key: HIVE-15077
> URL: https://issues.apache.org/jira/browse/HIVE-15077
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-15077.02.patch
>
>
> HIVE-10242 made the acid LM unfair.
> In TxnHandler.checkLock(), suppose we are trying to acquire SR5 (the number
> is extLockId).
> Then
> LockInfo[] locks = lockSet.toArray(new LockInfo[lockSet.size()]);
> may look like this (all explicitly listed locks are in Waiting state)
> {, SR5 SW3 X4}
> So the algorithm will find SR5 in the list and start looking backwards (to
> the left).
> According to IDs, SR5 should wait for X4 to be granted but X4 won't even be
> examined and so SR5 may be granted.
> Theoretically, this could cause starvation.
> The query that generates
[jira] [Commented] (HIVE-15077) Acid LockManager is unfair
[
https://issues.apache.org/jira/browse/HIVE-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609622#comment-15609622
]
Eugene Koifman commented on HIVE-15077:
---
a side note: change in HIVE-10483 to skip conflicts in the same txn should
probably be moved to before
if (locks[index].state == LockState.ACQUIRED) {
> Acid LockManager is unfair
> --
>
> Key: HIVE-15077
> URL: https://issues.apache.org/jira/browse/HIVE-15077
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> HIVE-10242 made the acid LM unfair.
> In TxnHandler.checkLock(), suppose we are trying to acquire SR5 (the number
> is extLockId).
> Then
> LockInfo[] locks = lockSet.toArray(new LockInfo[lockSet.size()]);
> may look like this (all explicitly listed locks are in Waiting state)
> {, SR5 SW3 X4}
> So the algorithm will find SR5 in the list and start looking backwards.
> According to IDs, SR5 should wait for X4 to be granted but X4 won't even be
> examined and so SR5 may be granted.
> Theoretically, this could cause starvation.
> The query that generates the list already has
> query.append(" and hl_lock_ext_id <= ").append(extLockId);
> but it should use "<" rather than "<=" to exclude the locks being checked
> from "locks" list which will make the algorithm look at all locks "in front"
> of a given lock.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
