[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165495#comment-17165495 ] Sean Busbey commented on HBASE-22114: - IIRC we need the ability for precommit to only run some modules if the using jdk8. I think I had a WIP branch somewhere that had this working. If the precommit builds are working over on the new CI set up I could try digging it up and getting it to work? > Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1 > --- > > Key: HBASE-22114 > URL: https://issues.apache.org/jira/browse/HBASE-22114 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 1.7.0 > > Attachments: HBASE-22114-branch-1.patch, HBASE-22114-branch-1.patch, > HBASE-22114-branch-1.patch > > > HBASE-15560 introduces the TinyLFU cache policy for the blockcache. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > The implementation of HBASE-15560 uses several Java 8 idioms, depends on JRE > 8+ type Optional, and has dependencies on libraries compiled with Java 8+ > bytecode. It could be backported to branch-1 but must be made optional both > at compile time and runtime, enabled by the 'build-with-jdk8' build profile. > The TinyLFU policy must go into its own build module. > The blockcache must be modified to load L1 implementation/policy dynamically > at startup by reflection if the policy is "TinyLFU" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165391#comment-17165391 ] Duo Zhang commented on HBASE-22114: --- Any updates here? Thanks. > Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1 > --- > > Key: HBASE-22114 > URL: https://issues.apache.org/jira/browse/HBASE-22114 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 1.7.0 > > Attachments: HBASE-22114-branch-1.patch, HBASE-22114-branch-1.patch, > HBASE-22114-branch-1.patch > > > HBASE-15560 introduces the TinyLFU cache policy for the blockcache. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > The implementation of HBASE-15560 uses several Java 8 idioms, depends on JRE > 8+ type Optional, and has dependencies on libraries compiled with Java 8+ > bytecode. It could be backported to branch-1 but must be made optional both > at compile time and runtime, enabled by the 'build-with-jdk8' build profile. > The TinyLFU policy must go into its own build module. > The blockcache must be modified to load L1 implementation/policy dynamically > at startup by reflection if the policy is "TinyLFU" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062084#comment-17062084
]
Hudson commented on HBASE-22114:
Results for branch HBASE-22114-branch-1
[build #107 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22114-branch-1/107/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22114-branch-1/107//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22114-branch-1/107//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22114-branch-1/107//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
> ---
>
> Key: HBASE-22114
> URL: https://issues.apache.org/jira/browse/HBASE-22114
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 1.7.0
>
> Attachments: HBASE-22114-branch-1.patch, HBASE-22114-branch-1.patch,
> HBASE-22114-branch-1.patch
>
>
> HBASE-15560 introduces the TinyLFU cache policy for the blockcache.
> W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the
> frequency in a counting sketch, ages periodically by halving the counters,
> and orders entries by SLRU. An entry is discarded by comparing the frequency
> of the new arrival (candidate) to the SLRU's victim, and keeping the one with
> the highest frequency. This allows the operations to be performed in O(1)
> time and, though the use of a compact sketch, a much larger history is
> retained beyond the current working set. In a variety of real world traces
> the policy had [near optimal hit
> rates|https://github.com/ben-manes/caffeine/wiki/Efficiency].
> The implementation of HBASE-15560 uses several Java 8 idioms, depends on JRE
> 8+ type Optional, and has dependencies on libraries compiled with Java 8+
> bytecode. It could be backported to branch-1 but must be made optional both
> at compile time and runtime, enabled by the 'build-with-jdk8' build profile.
> The TinyLFU policy must go into its own build module.
> The blockcache must be modified to load L1 implementation/policy dynamically
> at startup by reflection if the policy is "TinyLFU"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061213#comment-17061213
]
Hudson commented on HBASE-22114:
Results for branch HBASE-22114-branch-1
[build #106 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22114-branch-1/106/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22114-branch-1/106//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22114-branch-1/106//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22114-branch-1/106//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
> ---
>
> Key: HBASE-22114
> URL: https://issues.apache.org/jira/browse/HBASE-22114
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 1.7.0
>
> Attachments: HBASE-22114-branch-1.patch, HBASE-22114-branch-1.patch,
> HBASE-22114-branch-1.patch
>
>
> HBASE-15560 introduces the TinyLFU cache policy for the blockcache.
> W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the
> frequency in a counting sketch, ages periodically by halving the counters,
> and orders entries by SLRU. An entry is discarded by comparing the frequency
> of the new arrival (candidate) to the SLRU's victim, and keeping the one with
> the highest frequency. This allows the operations to be performed in O(1)
> time and, though the use of a compact sketch, a much larger history is
> retained beyond the current working set. In a variety of real world traces
> the policy had [near optimal hit
> rates|https://github.com/ben-manes/caffeine/wiki/Efficiency].
> The implementation of HBASE-15560 uses several Java 8 idioms, depends on JRE
> 8+ type Optional, and has dependencies on libraries compiled with Java 8+
> bytecode. It could be backported to branch-1 but must be made optional both
> at compile time and runtime, enabled by the 'build-with-jdk8' build profile.
> The TinyLFU policy must go into its own build module.
> The blockcache must be modified to load L1 implementation/policy dynamically
> at startup by reflection if the policy is "TinyLFU"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061081#comment-17061081
]
Michael Stack commented on HBASE-22114:
---
I pushed the below change on this branch:
{code}
commit 24bb5c349d17975e0b598b101b53ee6640f57859 (HEAD,
origin/HBASE-22114-branch-1)
Author: stack
Date: Tue Mar 17 10:48:00 2020 -0700
HBASE-22114 See comment on end; Make the nightly run once a week on this
branch
diff --git a/dev-support/Jenkinsfile b/dev-support/Jenkinsfile
index ee5f9c1ec2..29498933b0 100644
--- a/dev-support/Jenkinsfile
+++ b/dev-support/Jenkinsfile
@@ -21,7 +21,7 @@ pipeline {
}
}
triggers {
-cron('@daily')
+cron('@weekly')
}
options {
buildDiscarder(logRotator(numToKeepStr: '15'))
{code}
> Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
> ---
>
> Key: HBASE-22114
> URL: https://issues.apache.org/jira/browse/HBASE-22114
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 1.7.0
>
> Attachments: HBASE-22114-branch-1.patch, HBASE-22114-branch-1.patch,
> HBASE-22114-branch-1.patch
>
>
> HBASE-15560 introduces the TinyLFU cache policy for the blockcache.
> W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the
> frequency in a counting sketch, ages periodically by halving the counters,
> and orders entries by SLRU. An entry is discarded by comparing the frequency
> of the new arrival (candidate) to the SLRU's victim, and keeping the one with
> the highest frequency. This allows the operations to be performed in O(1)
> time and, though the use of a compact sketch, a much larger history is
> retained beyond the current working set. In a variety of real world traces
> the policy had [near optimal hit
> rates|https://github.com/ben-manes/caffeine/wiki/Efficiency].
> The implementation of HBASE-15560 uses several Java 8 idioms, depends on JRE
> 8+ type Optional, and has dependencies on libraries compiled with Java 8+
> bytecode. It could be backported to branch-1 but must be made optional both
> at compile time and runtime, enabled by the 'build-with-jdk8' build profile.
> The TinyLFU policy must go into its own build module.
> The blockcache must be modified to load L1 implementation/policy dynamically
> at startup by reflection if the policy is "TinyLFU"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061066#comment-17061066 ] Sean Busbey commented on HBASE-22114: - Once a week should be fine for the branch > Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1 > --- > > Key: HBASE-22114 > URL: https://issues.apache.org/jira/browse/HBASE-22114 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 1.7.0 > > Attachments: HBASE-22114-branch-1.patch, HBASE-22114-branch-1.patch, > HBASE-22114-branch-1.patch > > > HBASE-15560 introduces the TinyLFU cache policy for the blockcache. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > The implementation of HBASE-15560 uses several Java 8 idioms, depends on JRE > 8+ type Optional, and has dependencies on libraries compiled with Java 8+ > bytecode. It could be backported to branch-1 but must be made optional both > at compile time and runtime, enabled by the 'build-with-jdk8' build profile. > The TinyLFU policy must go into its own build module. > The blockcache must be modified to load L1 implementation/policy dynamically > at startup by reflection if the policy is "TinyLFU" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061029#comment-17061029 ] Michael Stack commented on HBASE-22114: --- We need this running every night still? [~busbey]/[~apurtell] I'm trying to clean up nightly test runs. Thanks. > Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1 > --- > > Key: HBASE-22114 > URL: https://issues.apache.org/jira/browse/HBASE-22114 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 1.7.0 > > Attachments: HBASE-22114-branch-1.patch, HBASE-22114-branch-1.patch, > HBASE-22114-branch-1.patch > > > HBASE-15560 introduces the TinyLFU cache policy for the blockcache. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > The implementation of HBASE-15560 uses several Java 8 idioms, depends on JRE > 8+ type Optional, and has dependencies on libraries compiled with Java 8+ > bytecode. It could be backported to branch-1 but must be made optional both > at compile time and runtime, enabled by the 'build-with-jdk8' build profile. > The TinyLFU policy must go into its own build module. > The blockcache must be modified to load L1 implementation/policy dynamically > at startup by reflection if the policy is "TinyLFU" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031073#comment-17031073
]
HBase QA commented on HBASE-22114:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 11s{color}
| {color:red} HBASE-22114 does not apply to branch-1. Rebase required? Wrong
Branch? See
https://yetus.apache.org/documentation/in-progress/precommit-patchnames for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-22114 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12964768/HBASE-22114-branch-1.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/1119/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
> Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
> ---
>
> Key: HBASE-22114
> URL: https://issues.apache.org/jira/browse/HBASE-22114
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 1.6.0
>
> Attachments: HBASE-22114-branch-1.patch, HBASE-22114-branch-1.patch,
> HBASE-22114-branch-1.patch
>
>
> HBASE-15560 introduces the TinyLFU cache policy for the blockcache.
> W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the
> frequency in a counting sketch, ages periodically by halving the counters,
> and orders entries by SLRU. An entry is discarded by comparing the frequency
> of the new arrival (candidate) to the SLRU's victim, and keeping the one with
> the highest frequency. This allows the operations to be performed in O(1)
> time and, though the use of a compact sketch, a much larger history is
> retained beyond the current working set. In a variety of real world traces
> the policy had [near optimal hit
> rates|https://github.com/ben-manes/caffeine/wiki/Efficiency].
> The implementation of HBASE-15560 uses several Java 8 idioms, depends on JRE
> 8+ type Optional, and has dependencies on libraries compiled with Java 8+
> bytecode. It could be backported to branch-1 but must be made optional both
> at compile time and runtime, enabled by the 'build-with-jdk8' build profile.
> The TinyLFU policy must go into its own build module.
> The blockcache must be modified to load L1 implementation/policy dynamically
> at startup by reflection if the policy is "TinyLFU"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908373#comment-16908373
]
HBase QA commented on HBASE-22114:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} HBASE-22114 does not apply to branch-1. Rebase required? Wrong
Branch? See
https://yetus.apache.org/documentation/in-progress/precommit-patchnames for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-22114 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12964768/HBASE-22114-branch-1.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/778/console |
| Powered by | Apache Yetus 0.9.0 http://yetus.apache.org |
This message was automatically generated.
> Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
> ---
>
> Key: HBASE-22114
> URL: https://issues.apache.org/jira/browse/HBASE-22114
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Major
> Fix For: 1.6.0
>
> Attachments: HBASE-22114-branch-1.patch, HBASE-22114-branch-1.patch,
> HBASE-22114-branch-1.patch
>
>
> HBASE-15560 introduces the TinyLFU cache policy for the blockcache.
> W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the
> frequency in a counting sketch, ages periodically by halving the counters,
> and orders entries by SLRU. An entry is discarded by comparing the frequency
> of the new arrival (candidate) to the SLRU's victim, and keeping the one with
> the highest frequency. This allows the operations to be performed in O(1)
> time and, though the use of a compact sketch, a much larger history is
> retained beyond the current working set. In a variety of real world traces
> the policy had [near optimal hit
> rates|https://github.com/ben-manes/caffeine/wiki/Efficiency].
> The implementation of HBASE-15560 uses several Java 8 idioms, depends on JRE
> 8+ type Optional, and has dependencies on libraries compiled with Java 8+
> bytecode. It could be backported to branch-1 but must be made optional both
> at compile time and runtime, enabled by the 'build-with-jdk8' build profile.
> The TinyLFU policy must go into its own build module.
> The blockcache must be modified to load L1 implementation/policy dynamically
> at startup by reflection if the policy is "TinyLFU"
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812503#comment-16812503 ] Sean Busbey commented on HBASE-22114: - sounds good. I'm going to try to clean up branch-1 precommit as a part of HBASE-22020 and then run through my proposed precommit changes for this here. some of the structural issues look like they also come up on the current branch due to the hbase-error-prone module. > Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1 > --- > > Key: HBASE-22114 > URL: https://issues.apache.org/jira/browse/HBASE-22114 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.6.0 > > Attachments: HBASE-22114-branch-1.patch, HBASE-22114-branch-1.patch, > HBASE-22114-branch-1.patch > > > HBASE-15560 introduces the TinyLFU cache policy for the blockcache. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > The implementation of HBASE-15560 uses several Java 8 idioms, depends on JRE > 8+ type Optional, and has dependencies on libraries compiled with Java 8+ > bytecode. It could be backported to branch-1 but must be made optional both > at compile time and runtime, enabled by the 'build-with-jdk8' build profile. > The TinyLFU policy must go into its own build module. > The blockcache must be modified to load L1 implementation/policy dynamically > at startup by reflection if the policy is "TinyLFU" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811695#comment-16811695 ] Andrew Purtell commented on HBASE-22114: Thanks for looking into those issues [~busbey] . I’ve rescheduled this for a future 1.6.0 and hopefully by then > Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1 > --- > > Key: HBASE-22114 > URL: https://issues.apache.org/jira/browse/HBASE-22114 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.6.0 > > Attachments: HBASE-22114-branch-1.patch, HBASE-22114-branch-1.patch, > HBASE-22114-branch-1.patch > > > HBASE-15560 introduces the TinyLFU cache policy for the blockcache. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > The implementation of HBASE-15560 uses several Java 8 idioms, depends on JRE > 8+ type Optional, and has dependencies on libraries compiled with Java 8+ > bytecode. It could be backported to branch-1 but must be made optional both > at compile time and runtime, enabled by the 'build-with-jdk8' build profile. > The TinyLFU policy must go into its own build module. > The blockcache must be modified to load L1 implementation/policy dynamically > at startup by reflection if the policy is "TinyLFU" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811441#comment-16811441 ] Sean Busbey commented on HBASE-22114: - okay shoot we're going to be blocked on HBASE-22175. > Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1 > --- > > Key: HBASE-22114 > URL: https://issues.apache.org/jira/browse/HBASE-22114 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.6.0 > > Attachments: HBASE-22114-branch-1.patch, HBASE-22114-branch-1.patch, > HBASE-22114-branch-1.patch > > > HBASE-15560 introduces the TinyLFU cache policy for the blockcache. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > The implementation of HBASE-15560 uses several Java 8 idioms, depends on JRE > 8+ type Optional, and has dependencies on libraries compiled with Java 8+ > bytecode. It could be backported to branch-1 but must be made optional both > at compile time and runtime, enabled by the 'build-with-jdk8' build profile. > The TinyLFU policy must go into its own build module. > The blockcache must be modified to load L1 implementation/policy dynamically > at startup by reflection if the policy is "TinyLFU" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811439#comment-16811439 ] Sean Busbey commented on HBASE-22114: - okay I think have personality changes that will keep the module from being built in jdk7 and will run it through a test on jdk8. let me get that xml error worked out (HBASE-22020) and figure out why the RAT failures are happening to me locally when I use jdk8. > Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1 > --- > > Key: HBASE-22114 > URL: https://issues.apache.org/jira/browse/HBASE-22114 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.6.0 > > Attachments: HBASE-22114-branch-1.patch, HBASE-22114-branch-1.patch, > HBASE-22114-branch-1.patch > > > HBASE-15560 introduces the TinyLFU cache policy for the blockcache. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > The implementation of HBASE-15560 uses several Java 8 idioms, depends on JRE > 8+ type Optional, and has dependencies on libraries compiled with Java 8+ > bytecode. It could be backported to branch-1 but must be made optional both > at compile time and runtime, enabled by the 'build-with-jdk8' build profile. > The TinyLFU policy must go into its own build module. > The blockcache must be modified to load L1 implementation/policy dynamically > at startup by reflection if the policy is "TinyLFU" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16810261#comment-16810261
]
Andrew Purtell commented on HBASE-22114:
{quote}I'm concerned we'll break precommit on branch-1 for good though.
{quote}
Me too
> Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
> ---
>
> Key: HBASE-22114
> URL: https://issues.apache.org/jira/browse/HBASE-22114
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Major
> Fix For: 1.6.0
>
> Attachments: HBASE-22114-branch-1.patch, HBASE-22114-branch-1.patch,
> HBASE-22114-branch-1.patch
>
>
> HBASE-15560 introduces the TinyLFU cache policy for the blockcache.
> W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the
> frequency in a counting sketch, ages periodically by halving the counters,
> and orders entries by SLRU. An entry is discarded by comparing the frequency
> of the new arrival (candidate) to the SLRU's victim, and keeping the one with
> the highest frequency. This allows the operations to be performed in O(1)
> time and, though the use of a compact sketch, a much larger history is
> retained beyond the current working set. In a variety of real world traces
> the policy had [near optimal hit
> rates|https://github.com/ben-manes/caffeine/wiki/Efficiency].
> The implementation of HBASE-15560 uses several Java 8 idioms, depends on JRE
> 8+ type Optional, and has dependencies on libraries compiled with Java 8+
> bytecode. It could be backported to branch-1 but must be made optional both
> at compile time and runtime, enabled by the 'build-with-jdk8' build profile.
> The TinyLFU policy must go into its own build module.
> The blockcache must be modified to load L1 implementation/policy dynamically
> at startup by reflection if the policy is "TinyLFU"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809965#comment-16809965
]
Sean Busbey commented on HBASE-22114:
-
{quote}
Also, it looks like javadoc was invoked on the tinylfu module under java 7 even
though that module is only ever conditionally defined and required from the
build-with-jdk8 profile, which has an activation predicate of jdk=1.8. Doesn't
happen when you build normally from the command line. I'm inclined to think
this is an issue with how Yetus decides what has changed and how Maven is
invoked to test those changes.
{quote}
yes it is. It detects that a module is present and expressly invokes maven to
build it, which we would tell folks not to do.
I think the patch looks good fundamentally. I'm concerned we'll break precommit
on branch-1 for good though.
Let me dedicate some time tomorrow to trying to untangle this.
> Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
> ---
>
> Key: HBASE-22114
> URL: https://issues.apache.org/jira/browse/HBASE-22114
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Major
> Fix For: 1.6.0
>
> Attachments: HBASE-22114-branch-1.patch, HBASE-22114-branch-1.patch,
> HBASE-22114-branch-1.patch
>
>
> HBASE-15560 introduces the TinyLFU cache policy for the blockcache.
> W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the
> frequency in a counting sketch, ages periodically by halving the counters,
> and orders entries by SLRU. An entry is discarded by comparing the frequency
> of the new arrival (candidate) to the SLRU's victim, and keeping the one with
> the highest frequency. This allows the operations to be performed in O(1)
> time and, though the use of a compact sketch, a much larger history is
> retained beyond the current working set. In a variety of real world traces
> the policy had [near optimal hit
> rates|https://github.com/ben-manes/caffeine/wiki/Efficiency].
> The implementation of HBASE-15560 uses several Java 8 idioms, depends on JRE
> 8+ type Optional, and has dependencies on libraries compiled with Java 8+
> bytecode. It could be backported to branch-1 but must be made optional both
> at compile time and runtime, enabled by the 'build-with-jdk8' build profile.
> The TinyLFU policy must go into its own build module.
> The blockcache must be modified to load L1 implementation/policy dynamically
> at startup by reflection if the policy is "TinyLFU"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809363#comment-16809363
]
Hadoop QA commented on HBASE-22114:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 20m
47s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:red}-1{color} | {color:red} hbaseanti {color} | {color:red} 0m
0s{color} | {color:red} The patch appears use Hadoop classification instead of
HBase. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
51s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} branch-1 passed with JDK v1.8.0_202 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} branch-1 passed with JDK v1.7.0_211 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
22s{color} | {color:green} branch-1 passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 3m
16s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
37s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
44s{color} | {color:green} branch-1 passed with JDK v1.8.0_202 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
34s{color} | {color:green} branch-1 passed with JDK v1.7.0_211 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} the patch passed with JDK v1.8.0_202 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} the patch passed with JDK v1.7.0_211 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch passed checkstyle in hbase-resource-bundle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} The patch passed checkstyle in hbase-common {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s{color} | {color:green} hbase-server: The patch generated 0 new + 84
unchanged - 12 fixed = 84 total (was 96) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} The patch passed checkstyle in hbase-it {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
12s{color} | {color:green} root: The patch generated 0 new + 93 unchanged - 12
fixed = 93 total (was 105) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} hbase-tinylfu-blockcache: The patch generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} xml {color} | {color:red} 0m 1s{color} |
{color:red} The patch has 5 ill-formed XML file(s). {color} |
| {color:blue}0{color} | {color:blue}
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809135#comment-16809135 ] Andrew Purtell commented on HBASE-22114: Good news is I've been testing TinyLFU under load in a small cluster, with and without chaos, and it's stable. > Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1 > --- > > Key: HBASE-22114 > URL: https://issues.apache.org/jira/browse/HBASE-22114 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.6.0 > > Attachments: HBASE-22114-branch-1.patch, HBASE-22114-branch-1.patch, > HBASE-22114-branch-1.patch > > > HBASE-15560 introduces the TinyLFU cache policy for the blockcache. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > The implementation of HBASE-15560 uses several Java 8 idioms, depends on JRE > 8+ type Optional, and has dependencies on libraries compiled with Java 8+ > bytecode. It could be backported to branch-1 but must be made optional both > at compile time and runtime, enabled by the 'build-with-jdk8' build profile. > The TinyLFU policy must go into its own build module. > The blockcache must be modified to load L1 implementation/policy dynamically > at startup by reflection if the policy is "TinyLFU" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809117#comment-16809117 ] Andrew Purtell commented on HBASE-22114: Need to downgrade to caffeine 2.6.2 per parent. Let me do that and post an update shortly. > Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1 > --- > > Key: HBASE-22114 > URL: https://issues.apache.org/jira/browse/HBASE-22114 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.6.0 > > Attachments: HBASE-22114-branch-1.patch, HBASE-22114-branch-1.patch > > > HBASE-15560 introduces the TinyLFU cache policy for the blockcache. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > The implementation of HBASE-15560 uses several Java 8 idioms, depends on JRE > 8+ type Optional, and has dependencies on libraries compiled with Java 8+ > bytecode. It could be backported to branch-1 but must be made optional both > at compile time and runtime, enabled by the 'build-with-jdk8' build profile. > The TinyLFU policy must go into its own build module. > The blockcache must be modified to load L1 implementation/policy dynamically > at startup by reflection if the policy is "TinyLFU" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809096#comment-16809096 ] Andrew Purtell commented on HBASE-22114: I was hoping to get this in to the next 1.5.0 RC but that isn't going to happen looks like. Stymied by precommit limitations, unless we want to plow ahead and disregard them. > Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1 > --- > > Key: HBASE-22114 > URL: https://issues.apache.org/jira/browse/HBASE-22114 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0 > > Attachments: HBASE-22114-branch-1.patch, HBASE-22114-branch-1.patch > > > HBASE-15560 introduces the TinyLFU cache policy for the blockcache. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > The implementation of HBASE-15560 uses several Java 8 idioms, depends on JRE > 8+ type Optional, and has dependencies on libraries compiled with Java 8+ > bytecode. It could be backported to branch-1 but must be made optional both > at compile time and runtime, enabled by the 'build-with-jdk8' build profile. > The TinyLFU policy must go into its own build module. > The blockcache must be modified to load L1 implementation/policy dynamically > at startup by reflection if the policy is "TinyLFU" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807247#comment-16807247
]
Hadoop QA commented on HBASE-22114:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:red}-1{color} | {color:red} hbaseanti {color} | {color:red} 0m
0s{color} | {color:red} The patch appears use Hadoop classification instead of
HBase. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} branch-1 passed with JDK v1.8.0_202 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} branch-1 passed with JDK v1.7.0_211 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
29s{color} | {color:green} branch-1 passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 3m
2s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
26s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
47s{color} | {color:green} branch-1 passed with JDK v1.8.0_202 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
38s{color} | {color:green} branch-1 passed with JDK v1.7.0_211 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} the patch passed with JDK v1.8.0_202 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed with JDK v1.7.0_211 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch passed checkstyle in hbase-resource-bundle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} The patch passed checkstyle in hbase-common {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} hbase-server: The patch generated 0 new + 84
unchanged - 12 fixed = 84 total (was 96) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} The patch passed checkstyle in hbase-it {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
14s{color} | {color:green} root: The patch generated 0 new + 93 unchanged - 12
fixed = 93 total (was 105) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} hbase-tinylfu-blockcache: The patch generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} xml {color} | {color:red} 0m 1s{color} |
{color:red} The patch has 5 ill-formed XML file(s). {color} |
| {color:blue}0{color} | {color:blue}
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807110#comment-16807110 ] Andrew Purtell commented on HBASE-22114: Attached new patch addressing checkstyle and whitespace nits and RB feedback. > Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1 > --- > > Key: HBASE-22114 > URL: https://issues.apache.org/jira/browse/HBASE-22114 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0 > > Attachments: HBASE-22114-branch-1.patch, HBASE-22114-branch-1.patch > > > HBASE-15560 introduces the TinyLFU cache policy for the blockcache. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > The implementation of HBASE-15560 uses several Java 8 idioms, depends on JRE > 8+ type Optional, and has dependencies on libraries compiled with Java 8+ > bytecode. It could be backported to branch-1 but must be made optional both > at compile time and runtime, enabled by the 'build-with-jdk8' build profile. > The TinyLFU policy must go into its own build module. > The blockcache must be modified to load L1 implementation/policy dynamically > at startup by reflection if the policy is "TinyLFU" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807081#comment-16807081 ] Andrew Purtell commented on HBASE-22114: Ack on your reviewboard comments [~busbey]. Will address them when fixing checkstyle and whitespace nits > Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1 > --- > > Key: HBASE-22114 > URL: https://issues.apache.org/jira/browse/HBASE-22114 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0 > > Attachments: HBASE-22114-branch-1.patch > > > HBASE-15560 introduces the TinyLFU cache policy for the blockcache. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > The implementation of HBASE-15560 uses several Java 8 idioms, depends on JRE > 8+ type Optional, and has dependencies on libraries compiled with Java 8+ > bytecode. It could be backported to branch-1 but must be made optional both > at compile time and runtime, enabled by the 'build-with-jdk8' build profile. > The TinyLFU policy must go into its own build module. > The blockcache must be modified to load L1 implementation/policy dynamically > at startup by reflection if the policy is "TinyLFU" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16807074#comment-16807074
]
Andrew Purtell commented on HBASE-22114:
The checkstyle and whitespace warnings are legit, will fix those.
When using the make_rc.sh script the RAT check problem does not manifest. This
will not be a problem for RMs. Looks like precommit is invoking Maven build
targets in some different way that is problematic. Could we add a RAT exclusion
like "***/target/***" ? Anyway seems like a precommit only problem.
I also don't understand this:
{noformat}
[INFO] Apache HBase - TinyLFU Block Cache SUCCESS [2.479s]
[INFO] Apache HBase - Integration Tests .. FAILURE [0.202s]
...
[ERROR] Failed to execute goal on project hbase-it: Could not resolve
dependencies for project org.apache.hbase:hbase-it:jar:1.5.0: Failure to find
org.apache.hbase:hbase-tinylfu-blockcache:jar:1.5.0 in
https://repository.apache.org/content/repositories/releases/ was cached in the
local repository, resolution will not be reattempted until the update interval
of apache release has elapsed or updates are forced -> [Help 1]{noformat}
My guess is Yetus is not invoking the 'install' target here so the hbase-it
module has nothing in the local cache to resolve the dependency with. Doesn't
show up when you do the local build yourself. I always do 'mvn clean install
...'. I tested assembly with 'mvn clean install package assembly:single
-DskipTests' .
Also, it looks like javadoc was invoked on the tinylfu module under java 7 even
though that module is only ever conditionally defined and required from the
build-with-jdk8 profile, which has an activation predicate of jdk=1.8. Doesn't
happen when you build normally from the command line. I'm inclined to think
this is an issue with how Yetus decides what has changed and how Maven is
invoked to test those changes.
> Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
> ---
>
> Key: HBASE-22114
> URL: https://issues.apache.org/jira/browse/HBASE-22114
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: HBASE-22114-branch-1.patch
>
>
> HBASE-15560 introduces the TinyLFU cache policy for the blockcache.
> W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the
> frequency in a counting sketch, ages periodically by halving the counters,
> and orders entries by SLRU. An entry is discarded by comparing the frequency
> of the new arrival (candidate) to the SLRU's victim, and keeping the one with
> the highest frequency. This allows the operations to be performed in O(1)
> time and, though the use of a compact sketch, a much larger history is
> retained beyond the current working set. In a variety of real world traces
> the policy had [near optimal hit
> rates|https://github.com/ben-manes/caffeine/wiki/Efficiency].
> The implementation of HBASE-15560 uses several Java 8 idioms, depends on JRE
> 8+ type Optional, and has dependencies on libraries compiled with Java 8+
> bytecode. It could be backported to branch-1 but must be made optional both
> at compile time and runtime, enabled by the 'build-with-jdk8' build profile.
> The TinyLFU policy must go into its own build module.
> The blockcache must be modified to load L1 implementation/policy dynamically
> at startup by reflection if the policy is "TinyLFU"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805493#comment-16805493
]
Sean Busbey commented on HBASE-22114:
-
the rat failures (which I think are causing most of the above failures) are
because the rat module only ignores {{target/}} directories for modules.
Because we do a jdk8 check we see the module and build it. So when we're using
jdk7 the new tinylfu module is not defined, so the {{target/}} output is
checked and fails.
> Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
> ---
>
> Key: HBASE-22114
> URL: https://issues.apache.org/jira/browse/HBASE-22114
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: HBASE-22114-branch-1.patch
>
>
> HBASE-15560 introduces the TinyLFU cache policy for the blockcache.
> W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the
> frequency in a counting sketch, ages periodically by halving the counters,
> and orders entries by SLRU. An entry is discarded by comparing the frequency
> of the new arrival (candidate) to the SLRU's victim, and keeping the one with
> the highest frequency. This allows the operations to be performed in O(1)
> time and, though the use of a compact sketch, a much larger history is
> retained beyond the current working set. In a variety of real world traces
> the policy had [near optimal hit
> rates|https://github.com/ben-manes/caffeine/wiki/Efficiency].
> The implementation of HBASE-15560 uses several Java 8 idioms, depends on JRE
> 8+ type Optional, and has dependencies on libraries compiled with Java 8+
> bytecode. It could be backported to branch-1 but must be made optional both
> at compile time and runtime, enabled by the 'build-with-jdk8' build profile.
> The TinyLFU policy must go into its own build module.
> The blockcache must be modified to load L1 implementation/policy dynamically
> at startup by reflection if the policy is "TinyLFU"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804583#comment-16804583
]
Hadoop QA commented on HBASE-22114:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 20m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:red}-1{color} | {color:red} hbaseanti {color} | {color:red} 0m
0s{color} | {color:red} The patch appears use Hadoop classification instead of
HBase. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
9s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} branch-1 passed with JDK v1.8.0_202 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
40s{color} | {color:green} branch-1 passed with JDK v1.7.0_211 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
27s{color} | {color:green} branch-1 passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 3m
38s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
0s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
32s{color} | {color:green} branch-1 passed with JDK v1.8.0_202 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
43s{color} | {color:green} branch-1 passed with JDK v1.7.0_211 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} the patch passed with JDK v1.8.0_202 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed with JDK v1.7.0_211 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
32s{color} | {color:red} hbase-server: The patch generated 2 new + 84 unchanged
- 12 fixed = 86 total (was 96) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m
48s{color} | {color:red} root: The patch generated 2 new + 93 unchanged - 12
fixed = 95 total (was 105) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} hbase-tinylfu-blockcache: The patch generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 12 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} xml {color} | {color:red} 0m 1s{color} |
{color:red} The patch has 5 ill-formed XML file(s). {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 3m
5s{color} | {color:blue} patch has no errors when building the reference guide.
See footer for rendered docs, which you should manually inspect. {color} |
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 0m
15s{color} |
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804573#comment-16804573
]
Hadoop QA commented on HBASE-22114:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 20m
57s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:red}-1{color} | {color:red} hbaseanti {color} | {color:red} 0m
0s{color} | {color:red} The patch appears use Hadoop classification instead of
HBase. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
33s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} branch-1 passed with JDK v1.8.0_202 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} branch-1 passed with JDK v1.7.0_211 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
19s{color} | {color:green} branch-1 passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 3m
3s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
30s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
50s{color} | {color:green} branch-1 passed with JDK v1.8.0_202 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
35s{color} | {color:green} branch-1 passed with JDK v1.7.0_211 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} the patch passed with JDK v1.8.0_202 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} the patch passed with JDK v1.7.0_211 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
12s{color} | {color:red} hbase-server: The patch generated 2 new + 84 unchanged
- 12 fixed = 86 total (was 96) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m
5s{color} | {color:red} root: The patch generated 2 new + 93 unchanged - 12
fixed = 95 total (was 105) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} hbase-tinylfu-blockcache: The patch generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 12 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} xml {color} | {color:red} 0m 0s{color} |
{color:red} The patch has 5 ill-formed XML file(s). {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 2m
43s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 0m
11s{color} |
[jira] [Commented] (HBASE-22114) Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-22114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16804484#comment-16804484 ] Andrew Purtell commented on HBASE-22114: WDYT [~busbey] ? Confirmed build is ok when Java 7. Confirmed build is ok when Java 8. Confirmed assembly includes hbase-tinylfu-blockcache and caffiene jars in lib/ if built with Java 8. New unit tests pass in new hbase-tinylfu-blockcache module. Other unit tests pass. Confirmed by hand that test clusters will start with hfile.block.cache.policy=TinyLFU when built with Java 8. > Port HBASE-15560 (TinyLFU-based BlockCache) to branch-1 > --- > > Key: HBASE-22114 > URL: https://issues.apache.org/jira/browse/HBASE-22114 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0 > > Attachments: HBASE-22114-branch-1.patch > > > HBASE-15560 introduces the TinyLFU cache policy for the blockcache. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > The implementation of HBASE-15560 uses several Java 8 idioms, depends on JRE > 8+ type Optional, and has dependencies on libraries compiled with Java 8+ > bytecode. It could be backported to branch-1 but must be made optional both > at compile time and runtime, enabled by the 'build-with-jdk8' build profile. > The TinyLFU policy must go into its own build module. > The blockcache must be modified to load L1 (and perhaps L2) > implementation/policy dynamically at startup by reflection, with > implementation classname specified in site configuration. This modification > must be forward ported to maintain configuration sanity among the branches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
