[jira] [Commented] (HBASE-14947) WALProcedureStore improvements
[ https://issues.apache.org/jira/browse/HBASE-14947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059078#comment-15059078 ] Hudson commented on HBASE-14947: FAILURE: Integrated in HBase-1.3-IT #376 (See [https://builds.apache.org/job/HBase-1.3-IT/376/]) HBASE-14947 WALProcedureStore improvements (matteo.bertozzi: rev bf7c36fccac5477d08e296ad93671d2c30ae2dc8) * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/ProcedureStoreTracker.java * hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/TestWALProcedureStore.java * hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/TestProcedureRecovery.java * hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/TestProcedureStoreTracker.java > WALProcedureStore improvements > -- > > Key: HBASE-14947 > URL: https://issues.apache.org/jira/browse/HBASE-14947 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Ashu Pachauri >Assignee: Matteo Bertozzi >Priority: Blocker > Fix For: 2.0.0, 1.2.0, 1.1.3 > > Attachments: HBASE-14947-v0.patch, HBASE-14947-v1.patch > > > We ended up with a deadlock in HBASE-14943, with the storeTracker and lock > acquired in reverse order by syncLoop() and insert/update/delete. In the > syncLoop() with don't need the lock when we try to roll or removeInactive. > also we can move the insert/update/delete tracker check in the syncLoop > avoiding to the extra lock operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14947) WALProcedureStore improvements
[ https://issues.apache.org/jira/browse/HBASE-14947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059135#comment-15059135 ] Hudson commented on HBASE-14947: SUCCESS: Integrated in HBase-1.1-JDK8 #1708 (See [https://builds.apache.org/job/HBase-1.1-JDK8/1708/]) HBASE-14947 WALProcedureStore improvements (matteo.bertozzi: rev 20b6dba997acd973fb8bd74666c42165bc715dbc) * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/ProcedureStoreTracker.java * hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/TestWALProcedureStore.java * hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/TestProcedureStoreTracker.java * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java > WALProcedureStore improvements > -- > > Key: HBASE-14947 > URL: https://issues.apache.org/jira/browse/HBASE-14947 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Ashu Pachauri >Assignee: Matteo Bertozzi >Priority: Blocker > Fix For: 2.0.0, 1.2.0, 1.1.3 > > Attachments: HBASE-14947-v0.patch, HBASE-14947-v1.patch > > > We ended up with a deadlock in HBASE-14943, with the storeTracker and lock > acquired in reverse order by syncLoop() and insert/update/delete. In the > syncLoop() with don't need the lock when we try to roll or removeInactive. > also we can move the insert/update/delete tracker check in the syncLoop > avoiding to the extra lock operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14947) WALProcedureStore improvements
[ https://issues.apache.org/jira/browse/HBASE-14947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059199#comment-15059199 ] Hudson commented on HBASE-14947: FAILURE: Integrated in HBase-Trunk_matrix #556 (See [https://builds.apache.org/job/HBase-Trunk_matrix/556/]) HBASE-14947 WALProcedureStore improvements (matteo.bertozzi: rev 60d33ce34191533bb858852584bd9bddfeb16a23) * hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/TestProcedureStoreTracker.java * hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/TestProcedureRecovery.java * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/ProcedureStoreTracker.java * hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/TestWALProcedureStore.java * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java > WALProcedureStore improvements > -- > > Key: HBASE-14947 > URL: https://issues.apache.org/jira/browse/HBASE-14947 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Ashu Pachauri >Assignee: Matteo Bertozzi >Priority: Blocker > Fix For: 2.0.0, 1.2.0, 1.1.3 > > Attachments: HBASE-14947-v0.patch, HBASE-14947-v1.patch > > > We ended up with a deadlock in HBASE-14943, with the storeTracker and lock > acquired in reverse order by syncLoop() and insert/update/delete. In the > syncLoop() with don't need the lock when we try to roll or removeInactive. > also we can move the insert/update/delete tracker check in the syncLoop > avoiding to the extra lock operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14947) WALProcedureStore improvements
[ https://issues.apache.org/jira/browse/HBASE-14947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059115#comment-15059115 ] Hudson commented on HBASE-14947: FAILURE: Integrated in HBase-1.1-JDK7 #1620 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1620/]) HBASE-14947 WALProcedureStore improvements (matteo.bertozzi: rev 20b6dba997acd973fb8bd74666c42165bc715dbc) * hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/TestWALProcedureStore.java * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/ProcedureStoreTracker.java * hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/TestProcedureStoreTracker.java > WALProcedureStore improvements > -- > > Key: HBASE-14947 > URL: https://issues.apache.org/jira/browse/HBASE-14947 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Ashu Pachauri >Assignee: Matteo Bertozzi >Priority: Blocker > Fix For: 2.0.0, 1.2.0, 1.1.3 > > Attachments: HBASE-14947-v0.patch, HBASE-14947-v1.patch > > > We ended up with a deadlock in HBASE-14943, with the storeTracker and lock > acquired in reverse order by syncLoop() and insert/update/delete. In the > syncLoop() with don't need the lock when we try to roll or removeInactive. > also we can move the insert/update/delete tracker check in the syncLoop > avoiding to the extra lock operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14947) WALProcedureStore improvements
[ https://issues.apache.org/jira/browse/HBASE-14947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059157#comment-15059157 ] Hudson commented on HBASE-14947: SUCCESS: Integrated in HBase-1.2-IT #342 (See [https://builds.apache.org/job/HBase-1.2-IT/342/]) HBASE-14947 WALProcedureStore improvements (matteo.bertozzi: rev e8698da2a4b6f7b419478485617ec22933f22f3c) * hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/TestWALProcedureStore.java * hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/TestProcedureRecovery.java * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/ProcedureStoreTracker.java * hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/TestProcedureStoreTracker.java > WALProcedureStore improvements > -- > > Key: HBASE-14947 > URL: https://issues.apache.org/jira/browse/HBASE-14947 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Ashu Pachauri >Assignee: Matteo Bertozzi >Priority: Blocker > Fix For: 2.0.0, 1.2.0, 1.1.3 > > Attachments: HBASE-14947-v0.patch, HBASE-14947-v1.patch > > > We ended up with a deadlock in HBASE-14943, with the storeTracker and lock > acquired in reverse order by syncLoop() and insert/update/delete. In the > syncLoop() with don't need the lock when we try to roll or removeInactive. > also we can move the insert/update/delete tracker check in the syncLoop > avoiding to the extra lock operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14947) WALProcedureStore improvements
[ https://issues.apache.org/jira/browse/HBASE-14947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059506#comment-15059506 ] Hudson commented on HBASE-14947: FAILURE: Integrated in HBase-1.3 #439 (See [https://builds.apache.org/job/HBase-1.3/439/]) HBASE-14947 WALProcedureStore improvements (matteo.bertozzi: rev bf7c36fccac5477d08e296ad93671d2c30ae2dc8) * hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/TestProcedureRecovery.java * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/ProcedureStoreTracker.java * hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/TestWALProcedureStore.java * hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/TestProcedureStoreTracker.java * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java > WALProcedureStore improvements > -- > > Key: HBASE-14947 > URL: https://issues.apache.org/jira/browse/HBASE-14947 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Ashu Pachauri >Assignee: Matteo Bertozzi >Priority: Blocker > Fix For: 2.0.0, 1.2.0, 1.1.3 > > Attachments: HBASE-14947-v0.patch, HBASE-14947-v1.patch > > > We ended up with a deadlock in HBASE-14943, with the storeTracker and lock > acquired in reverse order by syncLoop() and insert/update/delete. In the > syncLoop() with don't need the lock when we try to roll or removeInactive. > also we can move the insert/update/delete tracker check in the syncLoop > avoiding to the extra lock operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14947) WALProcedureStore improvements
[ https://issues.apache.org/jira/browse/HBASE-14947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059342#comment-15059342 ] Hudson commented on HBASE-14947: SUCCESS: Integrated in HBase-1.2 #445 (See [https://builds.apache.org/job/HBase-1.2/445/]) HBASE-14947 WALProcedureStore improvements (matteo.bertozzi: rev e8698da2a4b6f7b419478485617ec22933f22f3c) * hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/TestProcedureStoreTracker.java * hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/TestWALProcedureStore.java * hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/TestProcedureRecovery.java * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java * hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/ProcedureStoreTracker.java > WALProcedureStore improvements > -- > > Key: HBASE-14947 > URL: https://issues.apache.org/jira/browse/HBASE-14947 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Ashu Pachauri >Assignee: Matteo Bertozzi >Priority: Blocker > Fix For: 2.0.0, 1.2.0, 1.1.3 > > Attachments: HBASE-14947-v0.patch, HBASE-14947-v1.patch > > > We ended up with a deadlock in HBASE-14943, with the storeTracker and lock > acquired in reverse order by syncLoop() and insert/update/delete. In the > syncLoop() with don't need the lock when we try to roll or removeInactive. > also we can move the insert/update/delete tracker check in the syncLoop > avoiding to the extra lock operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14947) WALProcedureStore improvements
[ https://issues.apache.org/jira/browse/HBASE-14947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15056980#comment-15056980 ] Matteo Bertozzi commented on HBASE-14947: - ping [~syuanjiang] > WALProcedureStore improvements > -- > > Key: HBASE-14947 > URL: https://issues.apache.org/jira/browse/HBASE-14947 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Ashu Pachauri >Assignee: Matteo Bertozzi >Priority: Blocker > Attachments: HBASE-14947-v0.patch, HBASE-14947-v1.patch > > > We ended up with a deadlock in HBASE-14943, with the storeTracker and lock > acquired in reverse order by syncLoop() and insert/update/delete. In the > syncLoop() with don't need the lock when we try to roll or removeInactive. > also we can move the insert/update/delete tracker check in the syncLoop > avoiding to the extra lock operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14947) WALProcedureStore improvements
[ https://issues.apache.org/jira/browse/HBASE-14947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15057021#comment-15057021 ] Ashu Pachauri commented on HBASE-14947: --- [~mbertozzi] I had a look at the patch, looks good to me. +1 > WALProcedureStore improvements > -- > > Key: HBASE-14947 > URL: https://issues.apache.org/jira/browse/HBASE-14947 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Ashu Pachauri >Assignee: Matteo Bertozzi >Priority: Blocker > Attachments: HBASE-14947-v0.patch, HBASE-14947-v1.patch > > > We ended up with a deadlock in HBASE-14943, with the storeTracker and lock > acquired in reverse order by syncLoop() and insert/update/delete. In the > syncLoop() with don't need the lock when we try to roll or removeInactive. > also we can move the insert/update/delete tracker check in the syncLoop > avoiding to the extra lock operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14947) WALProcedureStore improvements
[
https://issues.apache.org/jira/browse/HBASE-14947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048267#comment-15048267
]
Hadoop QA commented on HBASE-14947:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12776484/HBASE-14947-v1.patch
against master branch at commit 7bfbb6a3c9af4b0e2853b5ea2580a05bb471211b.
ATTACHMENT ID: 12776484
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not generate new
checkstyle errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16806//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16806//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16806//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16806//console
This message is automatically generated.
> WALProcedureStore improvements
> --
>
> Key: HBASE-14947
> URL: https://issues.apache.org/jira/browse/HBASE-14947
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2
>Reporter: Ashu Pachauri
>Assignee: Matteo Bertozzi
>Priority: Minor
> Attachments: HBASE-14947-v0.patch, HBASE-14947-v1.patch
>
>
> We ended up with a deadlock in HBASE-14943, with the storeTracker and lock
> acquired in reverse order by syncLoop() and insert/update/delete. In the
> syncLoop() with don't need the lock when we try to roll or removeInactive.
> also we can move the insert/update/delete tracker check in the syncLoop
> avoiding to the extra lock operation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14947) WALProcedureStore improvements
[
https://issues.apache.org/jira/browse/HBASE-14947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15047864#comment-15047864
]
Hadoop QA commented on HBASE-14947:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12776404/HBASE-14947-v0.patch
against master branch at commit 2430efc2b2f2ec43e6ae1f88b6ebcb04f4e116c4.
ATTACHMENT ID: 12776404
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
new checkstyle errors. Check build console for list of new errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16798//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16798//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16798//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16798//console
This message is automatically generated.
> WALProcedureStore improvements
> --
>
> Key: HBASE-14947
> URL: https://issues.apache.org/jira/browse/HBASE-14947
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2
>Reporter: Ashu Pachauri
>Assignee: Matteo Bertozzi
>Priority: Minor
> Attachments: HBASE-14947-v0.patch
>
>
> We ended up with a deadlock in HBASE-14943, with the storeTracker and lock
> acquired in reverse order by syncLoop() and insert/update/delete. In the
> syncLoop() with don't need the lock when we try to roll or removeInactive.
> also we can move the insert/update/delete tracker check in the syncLoop
> avoiding to the extra lock operation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14947) WALProcedureStore improvements
[ https://issues.apache.org/jira/browse/HBASE-14947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046139#comment-15046139 ] Ashu Pachauri commented on HBASE-14947: --- This is a follow up from jira. > WALProcedureStore improvements > -- > > Key: HBASE-14947 > URL: https://issues.apache.org/jira/browse/HBASE-14947 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Reporter: Ashu Pachauri >Priority: Minor > > Currently we rely on locking at every operation level in WALProcedureStore, > this requires use of multiple locks to eliminate potential race conditions. > But, use of multiple locks is a recipe for potential deadlocks. > Investigate if there is a better way to deal with this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
