[jira] [Commented] (HBASE-16650) Wrong usage of BlockCache eviction stat for heap memory tuning
[ https://issues.apache.org/jira/browse/HBASE-16650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514232#comment-15514232 ] Hudson commented on HBASE-16650: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1654 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1654/]) HBASE-16650 Wrong usage of BlockCache eviction stat for heap memory (anoopsamjohn: rev 1384c9a08dd356556748ca0cc5853a695a40932c) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestLazyDataBlockDecompression.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestBlockCacheReporting.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheConfig.java * (edit) hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerSource.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerFromBucketCache.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HeapMemoryManager.java > Wrong usage of BlockCache eviction stat for heap memory tuning > -- > > Key: HBASE-16650 > URL: https://issues.apache.org/jira/browse/HBASE-16650 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.0.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16650.patch, HBASE-16650_V2.patch, > HBASE-16650_V3.patch, HBASE-16650_V4.patch, HBASE-16650_V5.patch > > > 1. We use the stat evictedBlocksCount - A block can get evicted because of > eviction thread due to lack of space or because of removal of an HFile itself > (After a compaction). We should not consider the latter in the tune decision > at all. These are actually invalidation of blocks. Should the stat counter > itself not use this count of evicted blocks? I think yes. This will give > wrong message to users that there are lot of real eviction happening. > 2. In case L1+ L2 combined block cache, what we use is the sum of evictions > from both. But we will be tuning L1 size alone. Eviction count from L2 should > not affect the tuning of L1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16650) Wrong usage of BlockCache eviction stat for heap memory tuning
[
https://issues.apache.org/jira/browse/HBASE-16650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513461#comment-15513461
]
Hadoop QA commented on HBASE-16650:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 18s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 36s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 117m 48s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestAdmin2 |
| | org.apache.hadoop.hbase.client.TestHCM |
| |
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829844/HBASE-16650_V5.patch |
| JIRA Issue | HBASE-16650 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux ffa64cb67f0f 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ce49364 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3664/artifact/patchprocess/patch-unit-hbase-server.txt
[jira] [Commented] (HBASE-16650) Wrong usage of BlockCache eviction stat for heap memory tuning
[
https://issues.apache.org/jira/browse/HBASE-16650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513161#comment-15513161
]
Hadoop QA commented on HBASE-16650:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m 48s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 30s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
5s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
39m 54s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 43s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 100m 34s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestStoreFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829811/HBASE-16650_V4.patch |
| JIRA Issue | HBASE-16650 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 49a47c263ba9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ce49364 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3661/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3661/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Commented] (HBASE-16650) Wrong usage of BlockCache eviction stat for heap memory tuning
[
https://issues.apache.org/jira/browse/HBASE-16650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512825#comment-15512825
]
Hadoop QA commented on HBASE-16650:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 11s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 6s
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 17s
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 59s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 25s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Incorrect lazy initialization of static field
org.apache.hadoop.hbase.io.hfile.CacheConfig.GLOBAL_L1_CACHE_INSTANCE in
org.apache.hadoop.hbase.io.hfile.CacheConfig.getL1(Configuration, long) At
CacheConfig.java:field
org.apache.hadoop.hbase.io.hfile.CacheConfig.GLOBAL_L1_CACHE_INSTANCE in
org.apache.hadoop.hbase.io.hfile.CacheConfig.getL1(Configuration, long) At
CacheConfig.java:[lines 570-577] |
| Failed junit tests | hadoop.hbase.regionserver.TestStoreFile |
| | hadoop.hbase.io.hfile.TestBlockCacheReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829798/HBASE-16650_V3.patch |
| JIRA Issue | HBASE-16650 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux caad65e8ad15 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
|
[jira] [Commented] (HBASE-16650) Wrong usage of BlockCache eviction stat for heap memory tuning
[
https://issues.apache.org/jira/browse/HBASE-16650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512414#comment-15512414
]
Hadoop QA commented on HBASE-16650:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 33s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 54s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 53s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 52s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 52s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 36s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 12s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 53s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 40s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 21s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m 1s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 11m 44s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 13m 26s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 15m 2s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 26s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 49s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} |
[jira] [Commented] (HBASE-16650) Wrong usage of BlockCache eviction stat for heap memory tuning
[
https://issues.apache.org/jira/browse/HBASE-16650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512212#comment-15512212
]
Anoop Sam John commented on HBASE-16650:
bq.We are setting some flags in this block that are used later. Do we need
those flags when an eviction because an hfile has been removed?
No there are no flags being set inside the if and used outside..
Code before patch
{code}
stats.evicted(block.getCachedTime(), block.getCacheKey().isPrimary());
if (evictedByEvictionProcess && victimHandler != null) {
if (victimHandler instanceof BucketCache) {
boolean wait = getCurrentSize() < acceptableSize();
boolean inMemory = block.getPriority() == BlockPriority.MEMORY;
((BucketCache)victimHandler).cacheBlockWithWait(block.getCacheKey(),
block.getBuffer(),
inMemory, wait);
} else {
victimHandler.cacheBlock(block.getCacheKey(), block.getBuffer());
}
}
{code}
Code after this patch
{code}
if (evictedByEvictionProcess ) {
stats.evicted(block.getCachedTime(), block.getCacheKey().isPrimary());
if(victimHandler != null){
if (victimHandler instanceof BucketCache) {
boolean wait = getCurrentSize() < acceptableSize();
boolean inMemory = block.getPriority() == BlockPriority.MEMORY;
((BucketCache)victimHandler).cacheBlockWithWait(block.getCacheKey(),
block.getBuffer(),
inMemory, wait);
} else {
victimHandler.cacheBlock(block.getCacheKey(), block.getBuffer());
}
}
}
{code}
Will commit with addressing ur 1st comment
> Wrong usage of BlockCache eviction stat for heap memory tuning
> --
>
> Key: HBASE-16650
> URL: https://issues.apache.org/jira/browse/HBASE-16650
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 1.0.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16650.patch
>
>
> 1. We use the stat evictedBlocksCount - A block can get evicted because of
> eviction thread due to lack of space or because of removal of an HFile itself
> (After a compaction). We should not consider the latter in the tune decision
> at all. These are actually invalidation of blocks. Should the stat counter
> itself not use this count of evicted blocks? I think yes. This will give
> wrong message to users that there are lot of real eviction happening.
> 2. In case L1+ L2 combined block cache, what we use is the sum of evictions
> from both. But we will be tuning L1 size alone. Eviction count from L2 should
> not affect the tuning of L1
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16650) Wrong usage of BlockCache eviction stat for heap memory tuning
[
https://issues.apache.org/jira/browse/HBASE-16650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508478#comment-15508478
]
Anoop Sam John commented on HBASE-16650:
Thanks Stack
bq.Is it right putting code under this check: if (evictedByEvictionProcess) {
The code was already under this check.. What I did is moved the stat update
also inside the if.
bq.We are setting some flags in this block that are used later. Do we need
those flags when an eviction because an hfile has been removed?
Sorry where is this set?
bq.This is ugly but we are already keeping blockcache in a global when we
fix this, we can fix this new addition:
Ya we need this new as u can see we pass the L1 cache instance to
HeapMemManager so that part we init L1 cache alone and later some one init
global cache it has to use this same instance.
> Wrong usage of BlockCache eviction stat for heap memory tuning
> --
>
> Key: HBASE-16650
> URL: https://issues.apache.org/jira/browse/HBASE-16650
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 1.0.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16650.patch
>
>
> 1. We use the stat evictedBlocksCount - A block can get evicted because of
> eviction thread due to lack of space or because of removal of an HFile itself
> (After a compaction). We should not consider the latter in the tune decision
> at all. These are actually invalidation of blocks. Should the stat counter
> itself not use this count of evicted blocks? I think yes. This will give
> wrong message to users that there are lot of real eviction happening.
> 2. In case L1+ L2 combined block cache, what we use is the sum of evictions
> from both. But we will be tuning L1 size alone. Eviction count from L2 should
> not affect the tuning of L1
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16650) Wrong usage of BlockCache eviction stat for heap memory tuning
[
https://issues.apache.org/jira/browse/HBASE-16650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507567#comment-15507567
]
stack commented on HBASE-16650:
---
Looking at the patch, can we update the description for the metric to explain
we DO NOT include evictions because an hfile was removed. Can do on commit.
Is it right putting code under this check: if
(evictedByEvictionProcess) {
We are setting some flags in this block that are used later. Do we need those
flags when an eviction because an hfile has been removed?
This is ugly but we are already keeping blockcache in a global when we fix
this, we can fix this new addition:
private static LruBlockCache GLOBAL_L1_CACHE_INSTANCE;
Otherwise, +1.
> Wrong usage of BlockCache eviction stat for heap memory tuning
> --
>
> Key: HBASE-16650
> URL: https://issues.apache.org/jira/browse/HBASE-16650
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 1.0.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16650.patch
>
>
> 1. We use the stat evictedBlocksCount - A block can get evicted because of
> eviction thread due to lack of space or because of removal of an HFile itself
> (After a compaction). We should not consider the latter in the tune decision
> at all. These are actually invalidation of blocks. Should the stat counter
> itself not use this count of evicted blocks? I think yes. This will give
> wrong message to users that there are lot of real eviction happening.
> 2. In case L1+ L2 combined block cache, what we use is the sum of evictions
> from both. But we will be tuning L1 size alone. Eviction count from L2 should
> not affect the tuning of L1
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16650) Wrong usage of BlockCache eviction stat for heap memory tuning
[
https://issues.apache.org/jira/browse/HBASE-16650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503135#comment-15503135
]
Hadoop QA commented on HBASE-16650:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 195m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s
{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 48s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 53s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 53s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 9s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 46s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 23s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 56s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 29s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.5.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m 5s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 11m 42s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 13m 17s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 14m 51s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 25s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 48s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 234m 42s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.0 Server=1.12.0 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829151/HBASE-16650.patch
[jira] [Commented] (HBASE-16650) Wrong usage of BlockCache eviction stat for heap memory tuning
[ https://issues.apache.org/jira/browse/HBASE-16650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15502488#comment-15502488 ] Anoop Sam John commented on HBASE-16650: Ping for reviews [~stack], [~ram_krish] > Wrong usage of BlockCache eviction stat for heap memory tuning > -- > > Key: HBASE-16650 > URL: https://issues.apache.org/jira/browse/HBASE-16650 > Project: HBase > Issue Type: Sub-task >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16650.patch > > > 1. We use the stat evictedBlocksCount - A block can get evicted because of > eviction thread due to lack of space or because of removal of an HFile itself > (After a compaction). We should not consider the latter in the tune decision > at all. These are actually invalidation of blocks. Should the stat counter > itself not use this count of evicted blocks? I think yes. This will give > wrong message to users that there are lot of real eviction happening. > 2. In case L1+ L2 combined block cache, what we use is the sum of evictions > from both. But we will be tuning L1 size alone. Eviction count from L2 should > not affect the tuning of L1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
