[jira] [Commented] (HIVE-19369) Locks: Add new lock implementations for always zero-wait readers
[ https://issues.apache.org/jira/browse/HIVE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091263#comment-17091263 ] Denys Kuzmenko commented on HIVE-19369: --- Pushed to master. Thank you [~pvary] for the review! > Locks: Add new lock implementations for always zero-wait readers > > > Key: HIVE-19369 > URL: https://issues.apache.org/jira/browse/HIVE-19369 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal Vijayaraghavan >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-19369.1.patch, HIVE-19369.2.patch, > HIVE-19369.3.patch, HIVE-19369.4.patch, HIVE-19369.5.patch, > HIVE-19369.6.patch, HIVE-19369.7.patch, HIVE-19369.8.patch, > HIVE-19369.9.patch, HIVE-19369.9.patch > > > Hive Locking with Micro-managed and full-ACID tables needs a better locking > implementation which allows for no-wait readers always. > EXCL_DROP > EXCL_WRITE > SHARED_WRITE > SHARED_READ > Short write-up > EXCL_DROP is a "drop partition" or "drop table" and waits for all others to > exit > EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to > exit. > SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an > EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different > threads). > SHARED_READ does not wait for any lock - it fails fast for a pending > EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, > there's no semantic reason to wait for them to succeed before going ahead > with a SHARED_WRITE. > a select * => SHARED_READ > an insert into => SHARED_WRITE > an insert overwrite or MERGE => EXCL_WRITE > a drop table => EXCL_DROP > TODO: > The fate of the compactor needs to be added to this before it is a complete > description. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-19369) Locks: Add new lock implementations for always zero-wait readers
[
https://issues.apache.org/jira/browse/HIVE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091167#comment-17091167
]
Hive QA commented on HIVE-19369:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13000993/HIVE-19369.9.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17137 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21907/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21907/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21907/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13000993 - PreCommit-HIVE-Build
> Locks: Add new lock implementations for always zero-wait readers
>
>
> Key: HIVE-19369
> URL: https://issues.apache.org/jira/browse/HIVE-19369
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal Vijayaraghavan
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-19369.1.patch, HIVE-19369.2.patch,
> HIVE-19369.3.patch, HIVE-19369.4.patch, HIVE-19369.5.patch,
> HIVE-19369.6.patch, HIVE-19369.7.patch, HIVE-19369.8.patch,
> HIVE-19369.9.patch, HIVE-19369.9.patch
>
>
> Hive Locking with Micro-managed and full-ACID tables needs a better locking
> implementation which allows for no-wait readers always.
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
> Short write-up
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to
> exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to
> exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an
> EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different
> threads).
> SHARED_READ does not wait for any lock - it fails fast for a pending
> EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending,
> there's no semantic reason to wait for them to succeed before going ahead
> with a SHARED_WRITE.
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
> TODO:
> The fate of the compactor needs to be added to this before it is a complete
> description.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-19369) Locks: Add new lock implementations for always zero-wait readers
[
https://issues.apache.org/jira/browse/HIVE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091160#comment-17091160
]
Hive QA commented on HIVE-19369:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
42s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
18s{color} | {color:blue} standalone-metastore/metastore-server in master has
190 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
1s{color} | {color:blue} ql in master has 1530 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
36s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} standalone-metastore/metastore-common: The patch
generated 0 new + 3 unchanged - 1 fixed = 3 total (was 4) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
24s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 5 new + 504 unchanged - 33 fixed = 509 total (was 537) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 21 new + 231 unchanged - 122
fixed = 252 total (was 353) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
53s{color} | {color:green} metastore-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} standalone-metastore/metastore-server generated 0
new + 189 unchanged - 1 fixed = 189 total (was 190) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
5s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21907/dev-support/hive-personality.sh
|
| git
[jira] [Commented] (HIVE-19369) Locks: Add new lock implementations for always zero-wait readers
[
https://issues.apache.org/jira/browse/HIVE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090353#comment-17090353
]
Hive QA commented on HIVE-19369:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13000866/HIVE-19369.8.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17137 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.txn.compactor.TestCrudCompactorOnTez.testCompactionWithSchemaEvolutionAndBuckets
(batchId=181)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21866/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21866/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21866/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13000866 - PreCommit-HIVE-Build
> Locks: Add new lock implementations for always zero-wait readers
>
>
> Key: HIVE-19369
> URL: https://issues.apache.org/jira/browse/HIVE-19369
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal Vijayaraghavan
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-19369.1.patch, HIVE-19369.2.patch,
> HIVE-19369.3.patch, HIVE-19369.4.patch, HIVE-19369.5.patch,
> HIVE-19369.6.patch, HIVE-19369.7.patch, HIVE-19369.8.patch
>
>
> Hive Locking with Micro-managed and full-ACID tables needs a better locking
> implementation which allows for no-wait readers always.
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
> Short write-up
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to
> exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to
> exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an
> EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different
> threads).
> SHARED_READ does not wait for any lock - it fails fast for a pending
> EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending,
> there's no semantic reason to wait for them to succeed before going ahead
> with a SHARED_WRITE.
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
> TODO:
> The fate of the compactor needs to be added to this before it is a complete
> description.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-19369) Locks: Add new lock implementations for always zero-wait readers
[
https://issues.apache.org/jira/browse/HIVE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090342#comment-17090342
]
Hive QA commented on HIVE-19369:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
40s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
18s{color} | {color:blue} standalone-metastore/metastore-server in master has
190 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
0s{color} | {color:blue} ql in master has 1530 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} standalone-metastore/metastore-common: The patch
generated 0 new + 3 unchanged - 1 fixed = 3 total (was 4) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 5 new + 504 unchanged - 33 fixed = 509 total (was 537) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 21 new + 231 unchanged - 122
fixed = 252 total (was 353) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
47s{color} | {color:green} metastore-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} standalone-metastore/metastore-server generated 0
new + 189 unchanged - 1 fixed = 189 total (was 190) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
1s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21866/dev-support/hive-personality.sh
|
| git
[jira] [Commented] (HIVE-19369) Locks: Add new lock implementations for always zero-wait readers
[
https://issues.apache.org/jira/browse/HIVE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089271#comment-17089271
]
Hive QA commented on HIVE-19369:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13000698/HIVE-19369.7.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 17146 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamptz_2]
(batchId=37)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testCompletedTxnComponents
(batchId=300)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testDynamicPartitionInsert
(batchId=300)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testDynamicPartitionInsertSharedWrite
(batchId=300)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testMerge3Way (batchId=300)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testMerge3WayConflict
(batchId=300)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testMerge3WayConflictSharedWrite
(batchId=300)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testMergePartitioned
(batchId=300)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testMergePartitionedConflict
(batchId=300)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testMergePartitionedConflictSharedWrite
(batchId=300)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testMergeUnpartitioned
(batchId=300)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testMergeUnpartitionedConflict
(batchId=300)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testMergeUnpartitionedConflictSharedWrite
(batchId=300)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testMultiInsert
(batchId=300)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testMultiInsertOnDynamicallyPartitionedMmTable
(batchId=300)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking10
(batchId=300)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking11
(batchId=300)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking7
(batchId=300)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking8
(batchId=300)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking9
(batchId=300)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21840/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21840/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21840/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13000698 - PreCommit-HIVE-Build
> Locks: Add new lock implementations for always zero-wait readers
>
>
> Key: HIVE-19369
> URL: https://issues.apache.org/jira/browse/HIVE-19369
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal Vijayaraghavan
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-19369.1.patch, HIVE-19369.2.patch,
> HIVE-19369.3.patch, HIVE-19369.4.patch, HIVE-19369.5.patch,
> HIVE-19369.6.patch, HIVE-19369.7.patch
>
>
> Hive Locking with Micro-managed and full-ACID tables needs a better locking
> implementation which allows for no-wait readers always.
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
> Short write-up
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to
> exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to
> exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an
> EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different
> threads).
> SHARED_READ does not wait for any lock - it fails fast for a pending
> EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending,
> there's no semantic reason to wait for them to succeed before going ahead
> with a SHARED_WRITE.
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
> TODO:
> The fate of the compactor needs to be added to this before it is a complete
> description.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-19369) Locks: Add new lock implementations for always zero-wait readers
[
https://issues.apache.org/jira/browse/HIVE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089267#comment-17089267
]
Hive QA commented on HIVE-19369:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
46s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
21s{color} | {color:blue} standalone-metastore/metastore-server in master has
190 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
59s{color} | {color:blue} ql in master has 1530 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
42s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} standalone-metastore/metastore-common: The patch
generated 0 new + 3 unchanged - 1 fixed = 3 total (was 4) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
24s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 5 new + 504 unchanged - 33 fixed = 509 total (was 537) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
48s{color} | {color:red} ql: The patch generated 23 new + 231 unchanged - 122
fixed = 254 total (was 353) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
55s{color} | {color:green} metastore-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s{color} | {color:green} standalone-metastore/metastore-server generated 0
new + 189 unchanged - 1 fixed = 189 total (was 190) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
6s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21840/dev-support/hive-personality.sh
|
| git
[jira] [Commented] (HIVE-19369) Locks: Add new lock implementations for always zero-wait readers
[
https://issues.apache.org/jira/browse/HIVE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17088616#comment-17088616
]
Hive QA commented on HIVE-19369:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13000613/HIVE-19369.6.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17146 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query11]
(batchId=228)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21820/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21820/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21820/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13000613 - PreCommit-HIVE-Build
> Locks: Add new lock implementations for always zero-wait readers
>
>
> Key: HIVE-19369
> URL: https://issues.apache.org/jira/browse/HIVE-19369
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal Vijayaraghavan
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-19369.1.patch, HIVE-19369.2.patch,
> HIVE-19369.3.patch, HIVE-19369.4.patch, HIVE-19369.5.patch, HIVE-19369.6.patch
>
>
> Hive Locking with Micro-managed and full-ACID tables needs a better locking
> implementation which allows for no-wait readers always.
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
> Short write-up
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to
> exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to
> exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an
> EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different
> threads).
> SHARED_READ does not wait for any lock - it fails fast for a pending
> EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending,
> there's no semantic reason to wait for them to succeed before going ahead
> with a SHARED_WRITE.
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
> TODO:
> The fate of the compactor needs to be added to this before it is a complete
> description.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-19369) Locks: Add new lock implementations for always zero-wait readers
[
https://issues.apache.org/jira/browse/HIVE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17088608#comment-17088608
]
Hive QA commented on HIVE-19369:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
41s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
15s{color} | {color:blue} standalone-metastore/metastore-server in master has
190 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
48s{color} | {color:blue} ql in master has 1530 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} standalone-metastore/metastore-common: The patch
generated 0 new + 3 unchanged - 1 fixed = 3 total (was 4) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 5 new + 504 unchanged - 33 fixed = 509 total (was 537) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 5 new + 233 unchanged - 120
fixed = 238 total (was 353) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
28s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 189 unchanged - 1 fixed = 190 total (was 190) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | org.apache.hadoop.hive.metastore.txn.TxnHandler$LockInfoComparator
implements Comparator but not Serializable At TxnHandler.java:Serializable At
TxnHandler.java:[lines 4085-4117] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21820/dev-support/hive-personality.sh
|
| git revision | master / 2f7caa0 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Commented] (HIVE-19369) Locks: Add new lock implementations for always zero-wait readers
[
https://issues.apache.org/jira/browse/HIVE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086861#comment-17086861
]
Hive QA commented on HIVE-19369:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13000402/HIVE-19369.5.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17153 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21767/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21767/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21767/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13000402 - PreCommit-HIVE-Build
> Locks: Add new lock implementations for always zero-wait readers
>
>
> Key: HIVE-19369
> URL: https://issues.apache.org/jira/browse/HIVE-19369
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal Vijayaraghavan
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-19369.1.patch, HIVE-19369.2.patch,
> HIVE-19369.3.patch, HIVE-19369.4.patch, HIVE-19369.5.patch
>
>
> Hive Locking with Micro-managed and full-ACID tables needs a better locking
> implementation which allows for no-wait readers always.
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
> Short write-up
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to
> exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to
> exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an
> EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different
> threads).
> SHARED_READ does not wait for any lock - it fails fast for a pending
> EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending,
> there's no semantic reason to wait for them to succeed before going ahead
> with a SHARED_WRITE.
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
> TODO:
> The fate of the compactor needs to be added to this before it is a complete
> description.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-19369) Locks: Add new lock implementations for always zero-wait readers
[
https://issues.apache.org/jira/browse/HIVE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086854#comment-17086854
]
Hive QA commented on HIVE-19369:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
38s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
20s{color} | {color:blue} standalone-metastore/metastore-server in master has
191 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
1s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} standalone-metastore/metastore-common: The patch
generated 0 new + 3 unchanged - 1 fixed = 3 total (was 4) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} common: The patch generated 1 new + 378 unchanged - 0
fixed = 379 total (was 378) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
24s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 5 new + 506 unchanged - 33 fixed = 511 total (was 539) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 26 new + 234 unchanged - 119
fixed = 260 total (was 353) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 21 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
51s{color} | {color:red} standalone-metastore/metastore-common generated 1 new
+ 35 unchanged - 0 fixed = 36 total (was 35) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
25s{color} | {color:green} standalone-metastore/metastore-server generated 0
new + 190 unchanged - 1 fixed = 190 total (was 191) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
3s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-common |
| | org.apache.hadoop.hive.metastore.LockTypeComparator implements Comparator
but not Serializable At
[jira] [Commented] (HIVE-19369) Locks: Add new lock implementations for always zero-wait readers
[
https://issues.apache.org/jira/browse/HIVE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086747#comment-17086747
]
Hive QA commented on HIVE-19369:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13000397/HIVE-19369.4.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17153 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query34]
(batchId=227)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21764/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21764/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21764/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13000397 - PreCommit-HIVE-Build
> Locks: Add new lock implementations for always zero-wait readers
>
>
> Key: HIVE-19369
> URL: https://issues.apache.org/jira/browse/HIVE-19369
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal Vijayaraghavan
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-19369.1.patch, HIVE-19369.2.patch,
> HIVE-19369.3.patch, HIVE-19369.4.patch
>
>
> Hive Locking with Micro-managed and full-ACID tables needs a better locking
> implementation which allows for no-wait readers always.
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
> Short write-up
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to
> exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to
> exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an
> EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different
> threads).
> SHARED_READ does not wait for any lock - it fails fast for a pending
> EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending,
> there's no semantic reason to wait for them to succeed before going ahead
> with a SHARED_WRITE.
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
> TODO:
> The fate of the compactor needs to be added to this before it is a complete
> description.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-19369) Locks: Add new lock implementations for always zero-wait readers
[
https://issues.apache.org/jira/browse/HIVE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086745#comment-17086745
]
Hive QA commented on HIVE-19369:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
43s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
20s{color} | {color:blue} standalone-metastore/metastore-server in master has
191 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
0s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
37s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} standalone-metastore/metastore-common: The patch
generated 0 new + 3 unchanged - 1 fixed = 3 total (was 4) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} common: The patch generated 1 new + 378 unchanged - 0
fixed = 379 total (was 378) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 5 new + 506 unchanged - 33 fixed = 511 total (was 539) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 26 new + 234 unchanged - 119
fixed = 260 total (was 353) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 21 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
56s{color} | {color:red} standalone-metastore/metastore-common generated 1 new
+ 35 unchanged - 0 fixed = 36 total (was 35) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s{color} | {color:green} standalone-metastore/metastore-server generated 0
new + 190 unchanged - 1 fixed = 190 total (was 191) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
11s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-common |
| | org.apache.hadoop.hive.metastore.LockTypeComparator implements Comparator
but not Serializable At
[jira] [Commented] (HIVE-19369) Locks: Add new lock implementations for always zero-wait readers
[
https://issues.apache.org/jira/browse/HIVE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086708#comment-17086708
]
Hive QA commented on HIVE-19369:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13000380/HIVE-19369.3.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17154 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.service.auth.TestImproperTrustDomainAuthenticationBinary.org.apache.hive.service.auth.TestImproperTrustDomainAuthenticationBinary
(batchId=209)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21760/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21760/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21760/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13000380 - PreCommit-HIVE-Build
> Locks: Add new lock implementations for always zero-wait readers
>
>
> Key: HIVE-19369
> URL: https://issues.apache.org/jira/browse/HIVE-19369
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal Vijayaraghavan
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-19369.1.patch, HIVE-19369.2.patch,
> HIVE-19369.3.patch
>
>
> Hive Locking with Micro-managed and full-ACID tables needs a better locking
> implementation which allows for no-wait readers always.
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
> Short write-up
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to
> exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to
> exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an
> EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different
> threads).
> SHARED_READ does not wait for any lock - it fails fast for a pending
> EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending,
> there's no semantic reason to wait for them to succeed before going ahead
> with a SHARED_WRITE.
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
> TODO:
> The fate of the compactor needs to be added to this before it is a complete
> description.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-19369) Locks: Add new lock implementations for always zero-wait readers
[
https://issues.apache.org/jira/browse/HIVE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086707#comment-17086707
]
Hive QA commented on HIVE-19369:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
44s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
22s{color} | {color:blue} standalone-metastore/metastore-server in master has
191 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
58s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} standalone-metastore/metastore-common: The patch
generated 0 new + 3 unchanged - 1 fixed = 3 total (was 4) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} common: The patch generated 1 new + 378 unchanged - 0
fixed = 379 total (was 378) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 5 new + 506 unchanged - 33 fixed = 511 total (was 539) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
47s{color} | {color:red} ql: The patch generated 26 new + 234 unchanged - 119
fixed = 260 total (was 353) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 21 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
55s{color} | {color:red} standalone-metastore/metastore-common generated 1 new
+ 35 unchanged - 0 fixed = 36 total (was 35) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} standalone-metastore/metastore-server generated 0
new + 190 unchanged - 1 fixed = 190 total (was 191) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
13s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-common |
| | org.apache.hadoop.hive.metastore.LockTypeComparator implements Comparator
but not Serializable At
[jira] [Commented] (HIVE-19369) Locks: Add new lock implementations for always zero-wait readers
[
https://issues.apache.org/jira/browse/HIVE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086541#comment-17086541
]
Hive QA commented on HIVE-19369:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13000368/HIVE-19369.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 17140 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explain_locks]
(batchId=36)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testCheckLockAcquireAfterWaiting
(batchId=243)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testLockSWSWSR (batchId=243)
org.apache.hadoop.hive.metastore.txn.TestTxnHandler.testLockSWSWSW (batchId=243)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testDynamicPartitionInsert
(batchId=298)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testShowLocksFilterOptions
(batchId=298)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21753/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21753/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21753/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13000368 - PreCommit-HIVE-Build
> Locks: Add new lock implementations for always zero-wait readers
>
>
> Key: HIVE-19369
> URL: https://issues.apache.org/jira/browse/HIVE-19369
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal Vijayaraghavan
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-19369.1.patch, HIVE-19369.2.patch
>
>
> Hive Locking with Micro-managed and full-ACID tables needs a better locking
> implementation which allows for no-wait readers always.
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
> Short write-up
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to
> exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to
> exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an
> EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different
> threads).
> SHARED_READ does not wait for any lock - it fails fast for a pending
> EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending,
> there's no semantic reason to wait for them to succeed before going ahead
> with a SHARED_WRITE.
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
> TODO:
> The fate of the compactor needs to be added to this before it is a complete
> description.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-19369) Locks: Add new lock implementations for always zero-wait readers
[
https://issues.apache.org/jira/browse/HIVE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086536#comment-17086536
]
Hive QA commented on HIVE-19369:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
42s{color} | {color:blue} standalone-metastore/metastore-common in master has
35 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
22s{color} | {color:blue} standalone-metastore/metastore-server in master has
191 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
6s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
39s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} standalone-metastore/metastore-common: The patch
generated 3 new + 4 unchanged - 0 fixed = 7 total (was 4) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} common: The patch generated 1 new + 378 unchanged - 0
fixed = 379 total (was 378) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
26s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 23 new + 511 unchanged - 13 fixed = 534 total (was 524) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
47s{color} | {color:red} ql: The patch generated 25 new + 202 unchanged - 118
fixed = 227 total (was 320) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 21 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
1s{color} | {color:red} standalone-metastore/metastore-common generated 1 new +
35 unchanged - 0 fixed = 36 total (was 35) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s{color} | {color:green} standalone-metastore/metastore-server generated 0
new + 190 unchanged - 1 fixed = 190 total (was 191) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
22s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-common |
| | org.apache.hadoop.hive.metastore.LockTypeComparator implements Comparator
but not Serializable At
[jira] [Commented] (HIVE-19369) Locks: Add new lock implementations for always zero-wait readers
[
https://issues.apache.org/jira/browse/HIVE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019564#comment-17019564
]
Denys Kuzmenko commented on HIVE-19369:
---
Hi [~gopalv], if there was a schema change (i.e. EXCL_DROP) - all
blocked/dependant locks (any type, not only SHARED_READ) should fail fast,
right?
Following scenario (current logic):
{code}alter table acid add columns (b);{code} - acquires EXCLUSIVE lock;
{code}insert into table acid values (1,2,3);{code} - SHARED_READ, waits for
EXCLUSIVE to be released and later throws IndexOutOfBoundsException: Index: 3,
Size: 3
> Locks: Add new lock implementations for always zero-wait readers
>
>
> Key: HIVE-19369
> URL: https://issues.apache.org/jira/browse/HIVE-19369
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal Vijayaraghavan
>Assignee: Denys Kuzmenko
>Priority: Major
>
> Hive Locking with Micro-managed and full-ACID tables needs a better locking
> implementation which allows for no-wait readers always.
> EXCL_DROP
> EXCL_WRITE
> SHARED_WRITE
> SHARED_READ
> Short write-up
> EXCL_DROP is a "drop partition" or "drop table" and waits for all others to
> exit
> EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to
> exit.
> SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an
> EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different
> threads).
> SHARED_READ does not wait for any lock - it fails fast for a pending
> EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending,
> there's no semantic reason to wait for them to succeed before going ahead
> with a SHARED_WRITE.
> a select * => SHARED_READ
> an insert into => SHARED_WRITE
> an insert overwrite or MERGE => EXCL_WRITE
> a drop table => EXCL_DROP
> TODO:
> The fate of the compactor needs to be added to this before it is a complete
> description.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-19369) Locks: Add new lock implementations for always zero-wait readers
[ https://issues.apache.org/jira/browse/HIVE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459938#comment-16459938 ] Gopal V commented on HIVE-19369: That means the 2nd part of that is a codepath which already exists (i.e drop_excl -> fail readers), which further reduces the code that is necessary for this ticket. > Locks: Add new lock implementations for always zero-wait readers > > > Key: HIVE-19369 > URL: https://issues.apache.org/jira/browse/HIVE-19369 > Project: Hive > Issue Type: Improvement >Reporter: Gopal V >Priority: Major > > Hive Locking with Micro-managed and full-ACID tables needs a better locking > implementation which allows for no-wait readers always. > EXCL_DROP > EXCL_WRITE > SHARED_WRITE > SHARED_READ > Short write-up > EXCL_DROP is a "drop partition" or "drop table" and waits for all others to > exit > EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to > exit. > SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an > EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different > threads). > SHARED_READ does not wait for any lock - it fails fast for a pending > EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, > there's no semantic reason to wait for them to succeed before going ahead > with a SHARED_WRITE. > a select * => SHARED_READ > an insert into => SHARED_WRITE > an insert overwrite or MERGE => EXCL_WRITE > a drop table => EXCL_DROP > TODO: > The fate of the compactor needs to be added to this before it is a complete > description. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19369) Locks: Add new lock implementations for always zero-wait readers
[ https://issues.apache.org/jira/browse/HIVE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459921#comment-16459921 ] Eugene Koifman commented on HIVE-19369: --- if you want readers to die rather than wait you can just set the lock acquisition retry count to 0 > Locks: Add new lock implementations for always zero-wait readers > > > Key: HIVE-19369 > URL: https://issues.apache.org/jira/browse/HIVE-19369 > Project: Hive > Issue Type: Improvement >Reporter: Gopal V >Priority: Major > > Hive Locking with Micro-managed and full-ACID tables needs a better locking > implementation which allows for no-wait readers always. > EXCL_DROP > EXCL_WRITE > SHARED_WRITE > SHARED_READ > Short write-up > EXCL_DROP is a "drop partition" or "drop table" and waits for all others to > exit > EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to > exit. > SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an > EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different > threads). > SHARED_READ does not wait for any lock - it fails fast for a pending > EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, > there's no semantic reason to wait for them to succeed before going ahead > with a SHARED_WRITE. > a select * => SHARED_READ > an insert into => SHARED_WRITE > an insert overwrite or MERGE => EXCL_WRITE > a drop table => EXCL_DROP > TODO: > The fate of the compactor needs to be added to this before it is a complete > description. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19369) Locks: Add new lock implementations for always zero-wait readers
[ https://issues.apache.org/jira/browse/HIVE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459865#comment-16459865 ] Gopal V commented on HIVE-19369: Yes, despite the big description above - that new lock is 90% of this JIRA adds, rest of this is just documentation Here's the current LockType SHARED_READ(1), SHARED_WRITE(2), EXCLUSIVE(3); We already have 3 of the above asks, just needs to add a new one (the one which you mentioned) SHARED_READ SHARED_WRITE EXCLUSIVE | EXCL_DROP + EXCL_WRITE And the only other behaviour change when EXCLUSIVE gets more granularity is that the EXCL_DROP is no-wait for the readers, while current EXCLUSIVE is a wait for the readers as well. > Locks: Add new lock implementations for always zero-wait readers > > > Key: HIVE-19369 > URL: https://issues.apache.org/jira/browse/HIVE-19369 > Project: Hive > Issue Type: Improvement >Reporter: Gopal V >Priority: Major > > Hive Locking with Micro-managed and full-ACID tables needs a better locking > implementation which allows for no-wait readers always. > EXCL_DROP > EXCL_WRITE > SHARED_WRITE > SHARED_READ > Short write-up > EXCL_DROP is a "drop partition" or "drop table" and waits for all others to > exit > EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to > exit. > SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an > EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different > threads). > SHARED_READ does not wait for any lock - it fails fast for a pending > EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, > there's no semantic reason to wait for them to succeed before going ahead > with a SHARED_WRITE. > a select * => SHARED_READ > an insert into => SHARED_WRITE > an insert overwrite or MERGE => EXCL_WRITE > a drop table => EXCL_DROP > TODO: > The fate of the compactor needs to be added to this before it is a complete > description. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19369) Locks: Add new lock implementations for always zero-wait readers
[ https://issues.apache.org/jira/browse/HIVE-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459377#comment-16459377 ] Sergey Shelukhin commented on HIVE-19369: - Can we start with one that would just exclude other writers? > Locks: Add new lock implementations for always zero-wait readers > > > Key: HIVE-19369 > URL: https://issues.apache.org/jira/browse/HIVE-19369 > Project: Hive > Issue Type: Improvement >Reporter: Gopal V >Priority: Major > > Hive Locking with Micro-managed and full-ACID tables needs a better locking > implementation which allows for no-wait readers always. > EXCL_DROP > EXCL_WRITE > SHARED_WRITE > SHARED_READ > Short write-up > EXCL_DROP is a "drop partition" or "drop table" and waits for all others to > exit > EXCL_WRITE excludes all writes and will wait for all existing SHARED_WRITE to > exit. > SHARED_WRITE allows all SHARED_WRITES to go through, but will wait for an > EXCL_WRITE & EXCL_DROP (waiting so that you can do drop + insert in different > threads). > SHARED_READ does not wait for any lock - it fails fast for a pending > EXCL_DROP, because even if there is an EXCL_WRITE or SHARED_WRITE pending, > there's no semantic reason to wait for them to succeed before going ahead > with a SHARED_WRITE. > a select * => SHARED_READ > an insert into => SHARED_WRITE > an insert overwrite or MERGE => EXCL_WRITE > a drop table => EXCL_DROP > TODO: > The fate of the compactor needs to be added to this before it is a complete > description. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
