[jira] [Commented] (HBASE-16372) References to previous cell in read path should be avoided
[ https://issues.apache.org/jira/browse/HBASE-16372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15551723#comment-15551723 ] Hudson commented on HBASE-16372: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1736 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1736/]) HBASE-16372 References to previous cell in read path should be avoided (ramkrishna: rev 58e843dae22963e119d6bdc5563921a3e1712812) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/RowColBloomContext.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/BloomFilterWriter.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAvoidCellReferencesIntoShippedBlocks.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/ColumnTracker.java * (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/ScanWildcardColumnTracker.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/ExplicitColumnTracker.java * (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestPut.java * (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/exceptions/TestClientExceptionsUtil.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CompoundBloomFilterWriter.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/BloomContext.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/RowBloomContext.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ShipperListener.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFlusher.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/querymatcher/ScanQueryMatcher.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AbstractMultiFileWriter.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterImpl.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CellSink.java > References to previous cell in read path should be avoided > -- > > Key: HBASE-16372 > URL: https://issues.apache.org/jira/browse/HBASE-16372 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16372_1.patch, HBASE-16372_10.patch, > HBASE-16372_2.patch, HBASE-16372_3.patch, HBASE-16372_5_withfactory.patch, > HBASE-16372_6_withfactory.patch, HBASE-16372_7_withfactory.patch, > HBASE-16372_9.patch, HBASE-16372_testcase.patch, HBASE-16372_testcase_1.patch > > > Came as part of review discussion in HBASE-15554. If there are references > kept to previous cells in the read path, with the Ref count based eviction > mechanism in trunk, then chances are there to evict a block backing the > previous cell but the read path still does some operations on that garbage > collected previous cell leading to incorrect results. > Areas to target > -> Storescanner > -> Bloom filters (particularly in compaction path) > Thanks to [~anoop.hbase] to point out this in bloomfilter path. But we found > it could be in other areas also. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16372) References to previous cell in read path should be avoided
[ https://issues.apache.org/jira/browse/HBASE-16372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15551355#comment-15551355 ] ramkrishna.s.vasudevan commented on HBASE-16372: Will commit this patch. Removed the assert in StorefileWriter as after the latest patch all Writers are of type ShipperListener. > References to previous cell in read path should be avoided > -- > > Key: HBASE-16372 > URL: https://issues.apache.org/jira/browse/HBASE-16372 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-16372_1.patch, HBASE-16372_2.patch, > HBASE-16372_3.patch, HBASE-16372_5_withfactory.patch, > HBASE-16372_6_withfactory.patch, HBASE-16372_7_withfactory.patch, > HBASE-16372_9.patch, HBASE-16372_testcase.patch, HBASE-16372_testcase_1.patch > > > Came as part of review discussion in HBASE-15554. If there are references > kept to previous cells in the read path, with the Ref count based eviction > mechanism in trunk, then chances are there to evict a block backing the > previous cell but the read path still does some operations on that garbage > collected previous cell leading to incorrect results. > Areas to target > -> Storescanner > -> Bloom filters (particularly in compaction path) > Thanks to [~anoop.hbase] to point out this in bloomfilter path. But we found > it could be in other areas also. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16372) References to previous cell in read path should be avoided
[
https://issues.apache.org/jira/browse/HBASE-16372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15551339#comment-15551339
]
Hadoop QA commented on HBASE-16372:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 23s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 49s
{color} | {color:red} hbase-server generated 2 new + 0 unchanged - 0 fixed = 2
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 38s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 57s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 9s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 132m 20s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | instanceof will always return true for all non-null values in
org.apache.hadoop.hbase.regionserver.StoreFileWriter.beforeShipped(), since all
org.apache.hadoop.hbase.io.hfile.HFile$Writer are instances of
org.apache.hadoop.hbase.regionserver.ShipperListener At
StoreFileWriter.java:for all non-null values in
org.apache.hadoop.hbase.regionserver.StoreFileWriter.beforeShipped(), since all
org.apache.hadoop.hbase.io.hfile.HFile$Writer are instances of
org.apache.hadoop.hbase.regionserver.ShipperListener At
StoreFileWriter.java:[line 266] |
| | instanceof will always return true for all non-null values in
org.apache.hadoop.hbase.regionserver.StoreFileWriter.beforeShipped(), since all
org.apache.hadoop.hbase.util.BloomFilterWriter are instances of
org.apache.hadoop.hbase.regionserver.ShipperListener At
StoreFileWriter.java:for all non-null values in
[jira] [Commented] (HBASE-16372) References to previous cell in read path should be avoided
[
https://issues.apache.org/jira/browse/HBASE-16372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15548445#comment-15548445
]
Hadoop QA commented on HBASE-16372:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 28s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 22s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 42s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 56s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 28s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 134m 4s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.procedure.TestDispatchMergingRegionsProcedure |
| Timed out junit tests |
org.apache.hadoop.hbase.master.procedure.TestEnableTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestCloneSnapshotProcedure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12831712/HBASE-16372_7_withfactory.patch
|
| JIRA Issue | HBASE-16372 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 95ed2ea10549 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 617dfe1 |
| Default Java |
[jira] [Commented] (HBASE-16372) References to previous cell in read path should be avoided
[
https://issues.apache.org/jira/browse/HBASE-16372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15548054#comment-15548054
]
Hadoop QA commented on HBASE-16372:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 36s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 40s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 54s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 12s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 137m 4s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.procedure.TestDispatchMergingRegionsProcedure |
| | hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush |
| | hadoop.hbase.master.procedure.TestMasterProcedureWalLease |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient |
| | org.apache.hadoop.hbase.wal.TestWALSplitCompressed |
| | org.apache.hadoop.hbase.TestZooKeeper |
| | org.apache.hadoop.hbase.TestNamespace |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.0 Server=1.12.0 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12831681/HBASE-16372_6_withfactory.patch
|
| JIRA Issue | HBASE-16372 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux e8d2011fbd08 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64
[jira] [Commented] (HBASE-16372) References to previous cell in read path should be avoided
[
https://issues.apache.org/jira/browse/HBASE-16372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546382#comment-15546382
]
Hadoop QA commented on HBASE-16372:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 32s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 19s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 55s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 13s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 63m 20s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.TestCheckTestClasses |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.0 Server=1.12.0 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12831583/HBASE-16372_5_withfactory.patch
|
| JIRA Issue | HBASE-16372 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 8dad4b4466c9 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 34ad965 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/3817/artifact/patchprocess/whitespace-eol.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3817/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
[jira] [Commented] (HBASE-16372) References to previous cell in read path should be avoided
[
https://issues.apache.org/jira/browse/HBASE-16372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15535857#comment-15535857
]
Hadoop QA commented on HBASE-16372:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
1s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 23s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 11s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 119m 52s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
| |
org.apache.hadoop.hbase.master.procedure.TestDispatchMergingRegionsProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestRestoreSnapshotProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestTruncateTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestMasterProcedureWalLease |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12831097/HBASE-16372_3.patch |
| JIRA Issue | HBASE-16372 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux b06edd54d5ea 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 3757da6 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3781/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3781/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3781/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
[jira] [Commented] (HBASE-16372) References to previous cell in read path should be avoided
[ https://issues.apache.org/jira/browse/HBASE-16372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503767#comment-15503767 ] Anoop Sam John commented on HBASE-16372: Will check this by tomorrow. > References to previous cell in read path should be avoided > -- > > Key: HBASE-16372 > URL: https://issues.apache.org/jira/browse/HBASE-16372 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16372_1.patch, HBASE-16372_2.patch, > HBASE-16372_testcase.patch, HBASE-16372_testcase_1.patch > > > Came as part of review discussion in HBASE-15554. If there are references > kept to previous cells in the read path, with the Ref count based eviction > mechanism in trunk, then chances are there to evict a block backing the > previous cell but the read path still does some operations on that garbage > collected previous cell leading to incorrect results. > Areas to target > -> Storescanner > -> Bloom filters (particularly in compaction path) > Thanks to [~anoop.hbase] to point out this in bloomfilter path. But we found > it could be in other areas also. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16372) References to previous cell in read path should be avoided
[ https://issues.apache.org/jira/browse/HBASE-16372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503186#comment-15503186 ] ramkrishna.s.vasudevan commented on HBASE-16372: Ping for reviews [~anoop.hbase] > References to previous cell in read path should be avoided > -- > > Key: HBASE-16372 > URL: https://issues.apache.org/jira/browse/HBASE-16372 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16372_1.patch, HBASE-16372_2.patch, > HBASE-16372_testcase.patch, HBASE-16372_testcase_1.patch > > > Came as part of review discussion in HBASE-15554. If there are references > kept to previous cells in the read path, with the Ref count based eviction > mechanism in trunk, then chances are there to evict a block backing the > previous cell but the read path still does some operations on that garbage > collected previous cell leading to incorrect results. > Areas to target > -> Storescanner > -> Bloom filters (particularly in compaction path) > Thanks to [~anoop.hbase] to point out this in bloomfilter path. But we found > it could be in other areas also. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16372) References to previous cell in read path should be avoided
[ https://issues.apache.org/jira/browse/HBASE-16372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15484940#comment-15484940 ] ramkrishna.s.vasudevan commented on HBASE-16372: [~anoop.hbase] Any chance of review here? > References to previous cell in read path should be avoided > -- > > Key: HBASE-16372 > URL: https://issues.apache.org/jira/browse/HBASE-16372 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16372_1.patch, HBASE-16372_2.patch, > HBASE-16372_testcase.patch, HBASE-16372_testcase_1.patch > > > Came as part of review discussion in HBASE-15554. If there are references > kept to previous cells in the read path, with the Ref count based eviction > mechanism in trunk, then chances are there to evict a block backing the > previous cell but the read path still does some operations on that garbage > collected previous cell leading to incorrect results. > Areas to target > -> Storescanner > -> Bloom filters (particularly in compaction path) > Thanks to [~anoop.hbase] to point out this in bloomfilter path. But we found > it could be in other areas also. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16372) References to previous cell in read path should be avoided
[ https://issues.apache.org/jira/browse/HBASE-16372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438565#comment-15438565 ] ramkrishna.s.vasudevan commented on HBASE-16372: Any chance of reviews here!!! > References to previous cell in read path should be avoided > -- > > Key: HBASE-16372 > URL: https://issues.apache.org/jira/browse/HBASE-16372 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16372_1.patch, HBASE-16372_2.patch, > HBASE-16372_testcase.patch, HBASE-16372_testcase_1.patch > > > Came as part of review discussion in HBASE-15554. If there are references > kept to previous cells in the read path, with the Ref count based eviction > mechanism in trunk, then chances are there to evict a block backing the > previous cell but the read path still does some operations on that garbage > collected previous cell leading to incorrect results. > Areas to target > -> Storescanner > -> Bloom filters (particularly in compaction path) > Thanks to [~anoop.hbase] to point out this in bloomfilter path. But we found > it could be in other areas also. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16372) References to previous cell in read path should be avoided
[ https://issues.apache.org/jira/browse/HBASE-16372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432199#comment-15432199 ] ramkrishna.s.vasudevan commented on HBASE-16372: Test failure seems unrelated. > References to previous cell in read path should be avoided > -- > > Key: HBASE-16372 > URL: https://issues.apache.org/jira/browse/HBASE-16372 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16372_1.patch, HBASE-16372_2.patch, > HBASE-16372_testcase.patch, HBASE-16372_testcase_1.patch > > > Came as part of review discussion in HBASE-15554. If there are references > kept to previous cells in the read path, with the Ref count based eviction > mechanism in trunk, then chances are there to evict a block backing the > previous cell but the read path still does some operations on that garbage > collected previous cell leading to incorrect results. > Areas to target > -> Storescanner > -> Bloom filters (particularly in compaction path) > Thanks to [~anoop.hbase] to point out this in bloomfilter path. But we found > it could be in other areas also. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16372) References to previous cell in read path should be avoided
[
https://issues.apache.org/jira/browse/HBASE-16372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431358#comment-15431358
]
Hadoop QA commented on HBASE-16372:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 49s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 52s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 134m 7s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.security.token.TestGenerateDelegationToken |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-08-22 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824880/HBASE-16372_2.patch |
| JIRA Issue | HBASE-16372 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 0264eff708fa 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision |
[jira] [Commented] (HBASE-16372) References to previous cell in read path should be avoided
[
https://issues.apache.org/jira/browse/HBASE-16372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430919#comment-15430919
]
Hadoop QA commented on HBASE-16372:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
6s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 30s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
39s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 28s
{color} | {color:red} hbase-server-jdk1.8.0_101 with JDK v1.8.0_101 generated 1
new + 1 unchanged - 0 fixed = 2 total (was 1) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 36s
{color} | {color:red} hbase-server-jdk1.7.0_101 with JDK v1.7.0_101 generated 1
new + 1 unchanged - 0 fixed = 2 total (was 1) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 101m 5s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 146m 5s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mob.compactions.TestMobCompactor |
| | hadoop.hbase.mob.compactions.TestPartitionedMobCompactor |
| | hadoop.hbase.regionserver.TestDefaultMemStore |
| | hadoop.hbase.regionserver.TestCompactingMemStore |
| | hadoop.hbase.regionserver.TestMobStoreCompaction |
| | hadoop.hbase.regionserver.TestStoreScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-08-22 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824824/HBASE-16372_1.patch |
| JIRA
[jira] [Commented] (HBASE-16372) References to previous cell in read path should be avoided
[ https://issues.apache.org/jira/browse/HBASE-16372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430036#comment-15430036 ] ramkrishna.s.vasudevan commented on HBASE-16372: bq. we change the curBlock and move the old cur block to prevBlock. If there was an already block pointed by prevBlock move that to oldBlocks. So when the call returnBlocks(boolean returnAll) comes, we will return only oldBlocks if param is false. If true we return from all 3 refs. My initial impl to solve this was to always return the (last - 1) blocks in a shipped() call and then if the shipped() call is with true return all blocks. We won't have 3 references. So the list holding the previous blocks will always have one element in it (the last block that was accessed). Internally which means we still maintain a ref to the actual previous block. In all the above ways - the problem of not giving a chance for the block to be evicted happens - yes though it is LRU stil this is there. So did not want to do that way. But yes we can explore it no problem. bq.One more thing to note is that in read flow, when a seek or next call result in jumping out many blocks in btw, are we assigning curBlock with the in btw blocks? I don't think we do it. Once we are sure we are going to read from this block only that we update as the curBlock ref. I can verify once again. > References to previous cell in read path should be avoided > -- > > Key: HBASE-16372 > URL: https://issues.apache.org/jira/browse/HBASE-16372 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16372_testcase.patch, HBASE-16372_testcase_1.patch > > > Came as part of review discussion in HBASE-15554. If there are references > kept to previous cells in the read path, with the Ref count based eviction > mechanism in trunk, then chances are there to evict a block backing the > previous cell but the read path still does some operations on that garbage > collected previous cell leading to incorrect results. > Areas to target > -> Storescanner > -> Bloom filters (particularly in compaction path) > Thanks to [~anoop.hbase] to point out this in bloomfilter path. But we found > it could be in other areas also. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16372) References to previous cell in read path should be avoided
[ https://issues.apache.org/jira/browse/HBASE-16372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428111#comment-15428111 ] Anoop Sam John commented on HBASE-16372: I am thinking abt a simple way where there is no need to copy. Did not check in details wrt code. Here it is We keep referring to cur block in HFileReaderImpl and once we move out of that we will add it to prevBlocks list and change curBlock to the new one. So there comes the issue. The prevCell we refer some where else in the flow might have been from this just moved block and that might get evicted if it is returned by an in btw shipped call. So how abt it will be if we keep ref to curBlock and prevBlock and the oldBlocks. When the curBlock move to next, we change the curBlock and move the old cur block to prevBlock. If there was an already block pointed by prevBlock move that to oldBlocks.So when the call returnBlocks(boolean returnAll) comes, we will return only oldBlocks if param is false. If true we return from all 3 refs. Ideally when the read flow happens and we move to the second cell of the cur block (means the prevCell would be the 1st one from cur block), the prevBlock is ready for return. It might be hard to impl that. If not done we might delay the return of one block until the cur block is completed. But it might be also ok IMO. Because even if return, chances of this block getting evicted is rare as the eviction follows LRU. This block is very recently used any way. One more thing to note is that in read flow, when a seek or next call result in jumping out many blocks in btw, are we assigning curBlock with the in btw blocks? If so there is a chance that the prevCell is not just in prevBlock but in some old block.. Ya in btw many blocks we read but all skipped. May be because all cells in that are deleted or so. Then there need some refactor in way how we handle the curBlock ref. > References to previous cell in read path should be avoided > -- > > Key: HBASE-16372 > URL: https://issues.apache.org/jira/browse/HBASE-16372 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16372_testcase.patch, HBASE-16372_testcase_1.patch > > > Came as part of review discussion in HBASE-15554. If there are references > kept to previous cells in the read path, with the Ref count based eviction > mechanism in trunk, then chances are there to evict a block backing the > previous cell but the read path still does some operations on that garbage > collected previous cell leading to incorrect results. > Areas to target > -> Storescanner > -> Bloom filters (particularly in compaction path) > Thanks to [~anoop.hbase] to point out this in bloomfilter path. But we found > it could be in other areas also. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16372) References to previous cell in read path should be avoided
[ https://issues.apache.org/jira/browse/HBASE-16372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428056#comment-15428056 ] ramkrishna.s.vasudevan commented on HBASE-16372: As explained in the earlier comment yes for the reader case we need to copy under certain cases. For writer too we need to do but it has to be done seemlessly with the compactionn flow. Am able to address all cases except one. May be after checking that can post an updated comment on that. > References to previous cell in read path should be avoided > -- > > Key: HBASE-16372 > URL: https://issues.apache.org/jira/browse/HBASE-16372 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16372_testcase.patch, HBASE-16372_testcase_1.patch > > > Came as part of review discussion in HBASE-15554. If there are references > kept to previous cells in the read path, with the Ref count based eviction > mechanism in trunk, then chances are there to evict a block backing the > previous cell but the read path still does some operations on that garbage > collected previous cell leading to incorrect results. > Areas to target > -> Storescanner > -> Bloom filters (particularly in compaction path) > Thanks to [~anoop.hbase] to point out this in bloomfilter path. But we found > it could be in other areas also. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16372) References to previous cell in read path should be avoided
[ https://issues.apache.org/jira/browse/HBASE-16372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428070#comment-15428070 ] ramkrishna.s.vasudevan commented on HBASE-16372: But the problem with doing in this StoreScanner layer is that we are not sure if the underlying block is actually of SHARED type or EXCLUSIVE type. > References to previous cell in read path should be avoided > -- > > Key: HBASE-16372 > URL: https://issues.apache.org/jira/browse/HBASE-16372 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16372_testcase.patch, HBASE-16372_testcase_1.patch > > > Came as part of review discussion in HBASE-15554. If there are references > kept to previous cells in the read path, with the Ref count based eviction > mechanism in trunk, then chances are there to evict a block backing the > previous cell but the read path still does some operations on that garbage > collected previous cell leading to incorrect results. > Areas to target > -> Storescanner > -> Bloom filters (particularly in compaction path) > Thanks to [~anoop.hbase] to point out this in bloomfilter path. But we found > it could be in other areas also. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16372) References to previous cell in read path should be avoided
[ https://issues.apache.org/jira/browse/HBASE-16372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428058#comment-15428058 ] ramkrishna.s.vasudevan commented on HBASE-16372: One thing is handling it in the lower levels like the HFileReader level is very tricky but the StoreScanner layer is in a way better. Atleast we know why we take references and in what scenario the references are not updated to the actual last Cell. > References to previous cell in read path should be avoided > -- > > Key: HBASE-16372 > URL: https://issues.apache.org/jira/browse/HBASE-16372 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16372_testcase.patch, HBASE-16372_testcase_1.patch > > > Came as part of review discussion in HBASE-15554. If there are references > kept to previous cells in the read path, with the Ref count based eviction > mechanism in trunk, then chances are there to evict a block backing the > previous cell but the read path still does some operations on that garbage > collected previous cell leading to incorrect results. > Areas to target > -> Storescanner > -> Bloom filters (particularly in compaction path) > Thanks to [~anoop.hbase] to point out this in bloomfilter path. But we found > it could be in other areas also. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16372) References to previous cell in read path should be avoided
[ https://issues.apache.org/jira/browse/HBASE-16372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428022#comment-15428022 ] Anoop Sam John commented on HBASE-16372: Thanks. So what is the solution proposal ? We need copy when keeping ref? > References to previous cell in read path should be avoided > -- > > Key: HBASE-16372 > URL: https://issues.apache.org/jira/browse/HBASE-16372 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16372_testcase.patch, HBASE-16372_testcase_1.patch > > > Came as part of review discussion in HBASE-15554. If there are references > kept to previous cells in the read path, with the Ref count based eviction > mechanism in trunk, then chances are there to evict a block backing the > previous cell but the read path still does some operations on that garbage > collected previous cell leading to incorrect results. > Areas to target > -> Storescanner > -> Bloom filters (particularly in compaction path) > Thanks to [~anoop.hbase] to point out this in bloomfilter path. But we found > it could be in other areas also. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16372) References to previous cell in read path should be avoided
[ https://issues.apache.org/jira/browse/HBASE-16372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418798#comment-15418798 ] ramkrishna.s.vasudevan commented on HBASE-16372: Will comment on the findings till now. Will upload patch later. These are my findings, I will prepare a patch early next week (its a long weekend here) and see if am missing some other area too. StoreScanner The 'prevCell' is getting used for ensuring we have moved on to the next Cell lexographically. I would think though this can be removed this check using prevCell is needed atleast for the Reverse scan cases. So better we will not remove 'prevCell'. The problem of 'prevCell' referring to the last cell in the previous block and those blocks getting evicted happens when a)the size limit is reached b) the batch limit is reached c) the store offset limit is reached. In other words code wise where ever we break the while loop 'LOOP'. Why only in the above 3 cases is a problem because in all other cases whether we INCLUDE the current cell or SEEK_TO_NEXT_ROW or SEEK_TO_NEXT_COL or SKIP we always tend to do a heap.peek() after either heap.next() or heap.seek(). Internally storeFileScanner returns the current 'cur' cached cell for every peek() call. If a heap.seek() had happened the seek call would have either fetched a 'cur' cell in the current block or in a new block. So when StoreScanner does a heap.peek() this 'cur' cell is fetched and that is what we set to the prevCell. So when the next() continues like this to either INCLUDE or SKIP or again SEEK we are sure that the current block is in use and if the scan is completed we can safely return the previous blocks thus accumulated. Same is the case with StoreScanner calling heap.next() because StorefileScanner.next() updates 'cur' ref with the next cell (which can be from a new block). So when StoreScanner.peek() is called this new cell from the new blocks is fetched and the same is updated as the prevCell. So when the StoreScanner.next() completes its iteration we are safe to return all the previous blocks. But when the 3 conditions mentioned above is reached what happens is that we have either done a heap.seek() or heap.next() but seeing the above conditions we abruptly break the loop and return the results. Now for eg take the case of a break that happens after heap.next(). The StorefileScanner.next() has fetched a new cell from the new block and so the previous blocks are added to the return block list and when the scanned results are returned for this current call to next() we return all the previous blocks. At this point 'prevCell' ref is pointing to a cell that was part of these previous blocks. This is where the issue could happen. So in all the above conditions before breaking the loop we could safely do a copy(clone) of the 'prevCell' and update the 'prevCell' ref to the newly created cell. Compaction write flow In compaction write flow we read from the underlying files and keep writing those cells one by one to a new file. In case of SHARED memory we tend to return the blocks when a certain threshold of bytes have been written to new files in order to avoid OOME and to mimic the normal scan behaviour. Because we try to return the blocks in between the compation writes the following areas needs to be addressed a)Writer's state variables b)bloom filter's state variables Writer state variables = Writer state variables are the ones like 'lastCellInPreviousBlock',' firstCellinBlock', 'lastCell' etc which needs to be handled so that the returning of blocks do not corrupt the data behind these state variables. Bloom filter state variables === The bloom filter variable 'lastCell' should not be cached for the above reason. The 'lastCell' is needed in bloom area to compare if a new key has arrived as per the BloomType. Currently the Writer that is created in the Compaction path is done using the CellSink. This Writer creates all type of writes for the default compaction, the stripe compaction, the data tiered compaction and for the Mob compaction. So since the compactor framework is the same for all these and we call intermediate shipped() call for any of the above compaction, we could add an API in CellSink and say updateWriterState() (I am sure we need a better name). This will call similar APIs in all the underlying writers. When the call moves to the StoreFileWriter (this will be there in all the compaction writers because this is where the actual cell is written) - here we have 3 writes per store file writer. a) The HFileWriter b) the General bloom writer c) the delete bloom writer So this new API updateWriterState() will also be in StoreFileWriter because it also implements CellSink. Now this call will actually allow us to do a clone of all such cells which we use as
[jira] [Commented] (HBASE-16372) References to previous cell in read path should be avoided
[ https://issues.apache.org/jira/browse/HBASE-16372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411583#comment-15411583 ] Anoop Sam John commented on HBASE-16372: Converted it to be a sub task of HBASE-11425 so that some one trying to backport that issue wont miss this important fix. > References to previous cell in read path should be avoided > -- > > Key: HBASE-16372 > URL: https://issues.apache.org/jira/browse/HBASE-16372 > Project: HBase > Issue Type: Sub-task > Components: Scanners >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Critical > > Came as part of review discussion in HBASE-15554. If there are references > kept to previous cells in the read path, with the Ref count based eviction > mechanism in trunk, then chances are there to evict a block backing the > previous cell but the read path still does some operations on that garbage > collected previous cell leading to incorrect results. > Areas to target > -> Storescanner > -> Bloom filters (particularly in compaction path) > Thanks to [~anoop.hbase] to point out this in bloomfilter path. But we found > it could be in other areas also. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
