[jira] [Commented] (HBASE-22206) dist.apache.org must not be used for public downloads
[
https://issues.apache.org/jira/browse/HBASE-22206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815982#comment-16815982
]
HBase QA commented on HBASE-22206:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
12s{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 9m
8s{color} | {color:red} root in master failed. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 8m
42s{color} | {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce base:
https://builds.apache.org/job/PreCommit-HBASE-Build/71/artifact/patchprocess/Dockerfile
|
| JIRA Issue | HBASE-22206 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12965670/HBASE-22206.master.001.patch
|
| Optional Tests | dupname asflicense mvnsite xml |
| uname | Linux 5925e1fe0eb8 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb
13 15:00:41 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 2bae04fb2b |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| mvnsite |
https://builds.apache.org/job/PreCommit-HBASE-Build/71/artifact/patchprocess/branch-mvnsite-root.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-HBASE-Build/71/artifact/patchprocess/patch-mvnsite-root.txt
|
| Max. process+thread count | 85 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/71/console |
| Powered by | Apache Yetus 0.9.0 http://yetus.apache.org |
This message was automatically generated.
> dist.apache.org must not be used for public downloads
> -
>
> Key: HBASE-22206
> URL: https://issues.apache.org/jira/browse/HBASE-22206
> Project: HBase
> Issue Type: Bug
> Components: website
>Reporter: Sebb
>Assignee: Dima Spivak
>Priority: Major
> Attachments: HBASE-22206.master.001.patch,
> HBASE-22206.master.001.patch
>
>
> The dist.apache.org server is only intended for use by developers in staging
> releases.
> It must not be used on public download pages.
> Please use www.apache.org/dist (for KEYS, hashes and sigs) and the mirror
> system instead.
> The current download page has lots of references to dist.a.o; please replace
> thes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-22219) Backport HBASE-19049 to branch-1 to prevent DIRKRB-613
[
https://issues.apache.org/jira/browse/HBASE-22219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-22219:
--
Attachment: HBASE-22219.branch-1.patch
> Backport HBASE-19049 to branch-1 to prevent DIRKRB-613
> ---
>
> Key: HBASE-22219
> URL: https://issues.apache.org/jira/browse/HBASE-22219
> Project: HBase
> Issue Type: Improvement
> Components: test
>Affects Versions: 1.4.8, 1.4.7, 1.4.9
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Fix For: 1.5.0
>
> Attachments: HBASE-22219.branch-1.patch
>
>
> Observed several UT failures when verifying 1.5.0 release, one of which is as
> follows:
> {noformat}
> [ERROR] org.apache.hadoop.hbase.http.TestSpnegoHttpServer Time elapsed:
> 0.005 s <<< ERROR!
> java.lang.RuntimeException: Unable to parse:includedir /etc/krb5.conf.d/
> at
> org.apache.hadoop.hbase.http.TestSpnegoHttpServer.buildMiniKdc(TestSpnegoHttpServer.java:143)
> at
> org.apache.hadoop.hbase.http.TestSpnegoHttpServer.setupServer(TestSpnegoHttpServer.java:91)
> {noformat}
> And this is a known issue of kerby 1.0.0-RC2 as indicated by DIRKRB-613 and
> fixed in 1.0.0 release (by [this
> commit|https://github.com/apache/directory-kerby/commit/e34b1ef8fec64e89780aec37aac903d4608e215f])
> Thus we should backport HBASE-19049 which upgrade kerby dependency from
> 1.0.0-RC2 to 1.0.1
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22219) Backport HBASE-19049 to branch-1 to prevent DIRKRB-613
[
https://issues.apache.org/jira/browse/HBASE-22219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815975#comment-16815975
]
Yu Li commented on HBASE-22219:
---
Update: could not cherry-pick but simply a version change. Let me attach a
patch here...
> Backport HBASE-19049 to branch-1 to prevent DIRKRB-613
> ---
>
> Key: HBASE-22219
> URL: https://issues.apache.org/jira/browse/HBASE-22219
> Project: HBase
> Issue Type: Improvement
> Components: test
>Affects Versions: 1.4.8, 1.4.7, 1.4.9
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Fix For: 1.5.0
>
>
> Observed several UT failures when verifying 1.5.0 release, one of which is as
> follows:
> {noformat}
> [ERROR] org.apache.hadoop.hbase.http.TestSpnegoHttpServer Time elapsed:
> 0.005 s <<< ERROR!
> java.lang.RuntimeException: Unable to parse:includedir /etc/krb5.conf.d/
> at
> org.apache.hadoop.hbase.http.TestSpnegoHttpServer.buildMiniKdc(TestSpnegoHttpServer.java:143)
> at
> org.apache.hadoop.hbase.http.TestSpnegoHttpServer.setupServer(TestSpnegoHttpServer.java:91)
> {noformat}
> And this is a known issue of kerby 1.0.0-RC2 as indicated by DIRKRB-613 and
> fixed in 1.0.0 release (by [this
> commit|https://github.com/apache/directory-kerby/commit/e34b1ef8fec64e89780aec37aac903d4608e215f])
> Thus we should backport HBASE-19049 which upgrade kerby dependency from
> 1.0.0-RC2 to 1.0.1
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22219) Backport HBASE-19049 to branch-1 to prevent DIRKRB-613
[
https://issues.apache.org/jira/browse/HBASE-22219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815973#comment-16815973
]
Yu Li commented on HBASE-22219:
---
[~andrew.purt...@gmail.com] this could be done with a simple cherry-pick and if
you agree I will do the job. Thanks.
> Backport HBASE-19049 to branch-1 to prevent DIRKRB-613
> ---
>
> Key: HBASE-22219
> URL: https://issues.apache.org/jira/browse/HBASE-22219
> Project: HBase
> Issue Type: Improvement
> Components: test
>Affects Versions: 1.4.8, 1.4.7, 1.4.9
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Fix For: 1.5.0
>
>
> Observed several UT failures when verifying 1.5.0 release, one of which is as
> follows:
> {noformat}
> [ERROR] org.apache.hadoop.hbase.http.TestSpnegoHttpServer Time elapsed:
> 0.005 s <<< ERROR!
> java.lang.RuntimeException: Unable to parse:includedir /etc/krb5.conf.d/
> at
> org.apache.hadoop.hbase.http.TestSpnegoHttpServer.buildMiniKdc(TestSpnegoHttpServer.java:143)
> at
> org.apache.hadoop.hbase.http.TestSpnegoHttpServer.setupServer(TestSpnegoHttpServer.java:91)
> {noformat}
> And this is a known issue of kerby 1.0.0-RC2 as indicated by DIRKRB-613 and
> fixed in 1.0.0 release (by [this
> commit|https://github.com/apache/directory-kerby/commit/e34b1ef8fec64e89780aec37aac903d4608e215f])
> Thus we should backport HBASE-19049 which upgrade kerby dependency from
> 1.0.0-RC2 to 1.0.1
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21129) Clean up duplicate codes in #equals and #hashCode methods of Filter
[
https://issues.apache.org/jira/browse/HBASE-21129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815971#comment-16815971
]
Hudson commented on HBASE-21129:
Results for branch branch-2.0
[build #1512 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1512/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1512//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1512//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1512//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Clean up duplicate codes in #equals and #hashCode methods of Filter
> ---
>
> Key: HBASE-21129
> URL: https://issues.apache.org/jira/browse/HBASE-21129
> Project: HBase
> Issue Type: Improvement
> Components: Filters
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Minor
> Fix For: 3.0.0, 2.2.0
>
> Attachments: 21129.addendum, HBASE-21129.master.001.patch,
> HBASE-21129.master.002.patch, HBASE-21129.master.003.patch,
> HBASE-21129.master.004.patch, HBASE-21129.master.005.patch,
> HBASE-21129.master.006.patch, HBASE-21129.master.007.patch,
> HBASE-21129.master.008.patch
>
>
> It is a follow-up of HBASE-19008, aiming to clean up duplicate codes in
> #equals and #hashCode methods.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22144) MultiRowRangeFilter does not work with reversed scans
[
https://issues.apache.org/jira/browse/HBASE-22144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815972#comment-16815972
]
Hudson commented on HBASE-22144:
Results for branch branch-2.0
[build #1512 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1512/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1512//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1512//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1512//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> MultiRowRangeFilter does not work with reversed scans
> -
>
> Key: HBASE-22144
> URL: https://issues.apache.org/jira/browse/HBASE-22144
> Project: HBase
> Issue Type: Bug
> Components: Filters, scan
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.3.0, 2.0.6, 2.1.5
>
> Attachments: HBASE-22144.001.patch, HBASE-22144.002.patch,
> HBASE-22144.002.patch
>
>
> It appears that MultiRowRangeFilter was never written to function with
> reverse scans. There is too much logic that operates with the assumption that
> we are always moving "forward" through increasing ranges. It needs to be
> rewritten to "traverse" forward or backward, given how the context of the
> scan being used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19008) Add missing equals or hashCode method(s) to stock Filter implementations
[
https://issues.apache.org/jira/browse/HBASE-19008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815970#comment-16815970
]
Hudson commented on HBASE-19008:
Results for branch branch-2.0
[build #1512 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1512/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1512//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1512//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1512//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Add missing equals or hashCode method(s) to stock Filter implementations
>
>
> Key: HBASE-19008
> URL: https://issues.apache.org/jira/browse/HBASE-19008
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Labels: filter
> Fix For: 3.0.0, 2.2.0
>
> Attachments: Filters.png, HBASE-19008-1.patch, HBASE-19008-10.patch,
> HBASE-19008-11.patch, HBASE-19008-12.patch, HBASE-19008-13.patch,
> HBASE-19008-14.patch, HBASE-19008-15.patch, HBASE-19008-16.patch,
> HBASE-19008-17.patch, HBASE-19008-18.patch, HBASE-19008-2.patch,
> HBASE-19008-20.patch, HBASE-19008-21.patch, HBASE-19008-3.patch,
> HBASE-19008-4.patch, HBASE-19008-5.patch, HBASE-19008-6.patch,
> HBASE-19008-7.patch, HBASE-19008-8.patch, HBASE-19008-9.patch,
> HBASE-19008.patch
>
>
> In HBASE-15410, [~mdrob] reminded me that Filter implementations may not
> write {{equals}} or {{hashCode}} method(s).
> This issue is to add missing {{equals}} or {{hashCode}} method(s) to stock
> Filter implementations such as KeyOnlyFilter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21666) Break up the TestExportSnapshot UTs; they can timeout
[ https://issues.apache.org/jira/browse/HBASE-21666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815969#comment-16815969 ] Tak Lon (Stephen) Wu commented on HBASE-21666: -- ping again ? or should we resolve it with `won't fix` ? > Break up the TestExportSnapshot UTs; they can timeout > - > > Key: HBASE-21666 > URL: https://issues.apache.org/jira/browse/HBASE-21666 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 3.0.0 >Reporter: stack >Assignee: Tak Lon (Stephen) Wu >Priority: Major > Labels: beginner > Attachments: HBASE-21666.master.001.patch, > HBASE-21666.master.002.patch, HBASE-21666.master.003.patch > > > These timed out for [~Apache9] when he ran with the -PrunAllTests. Suggests > breaking them up into smaller tests so less likely they'll timeout. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [hbase] saintstack commented on issue #137: HBASE-22203 Reformatted DemoClient.java
saintstack commented on issue #137: HBASE-22203 Reformatted DemoClient.java URL: https://github.com/apache/hbase/pull/137#issuecomment-482438597 +1 from me. Thanks for the cleanup. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (HBASE-22219) Backport HBASE-19049 to branch-1 to prevent DIRKRB-613
Yu Li created HBASE-22219:
-
Summary: Backport HBASE-19049 to branch-1 to prevent DIRKRB-613
Key: HBASE-22219
URL: https://issues.apache.org/jira/browse/HBASE-22219
Project: HBase
Issue Type: Improvement
Components: test
Affects Versions: 1.4.9, 1.4.7, 1.4.8
Reporter: Yu Li
Assignee: Yu Li
Fix For: 1.5.0
Observed several UT failures when verifying 1.5.0 release, one of which is as
follows:
{noformat}
[ERROR] org.apache.hadoop.hbase.http.TestSpnegoHttpServer Time elapsed: 0.005
s <<< ERROR!
java.lang.RuntimeException: Unable to parse:includedir /etc/krb5.conf.d/
at
org.apache.hadoop.hbase.http.TestSpnegoHttpServer.buildMiniKdc(TestSpnegoHttpServer.java:143)
at
org.apache.hadoop.hbase.http.TestSpnegoHttpServer.setupServer(TestSpnegoHttpServer.java:91)
{noformat}
And this is a known issue of kerby 1.0.0-RC2 as indicated by DIRKRB-613 and
fixed in 1.0.0 release (by [this
commit|https://github.com/apache/directory-kerby/commit/e34b1ef8fec64e89780aec37aac903d4608e215f])
Thus we should backport HBASE-19049 which upgrade kerby dependency from
1.0.0-RC2 to 1.0.1
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22206) dist.apache.org must not be used for public downloads
[ https://issues.apache.org/jira/browse/HBASE-22206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815965#comment-16815965 ] stack commented on HBASE-22206: --- +1 on the patch. The build failure can't be related. Do you remember how to commit [~dimaspivak] ? > dist.apache.org must not be used for public downloads > - > > Key: HBASE-22206 > URL: https://issues.apache.org/jira/browse/HBASE-22206 > Project: HBase > Issue Type: Bug > Components: website >Reporter: Sebb >Assignee: Dima Spivak >Priority: Major > Attachments: HBASE-22206.master.001.patch, > HBASE-22206.master.001.patch > > > The dist.apache.org server is only intended for use by developers in staging > releases. > It must not be used on public download pages. > Please use www.apache.org/dist (for KEYS, hashes and sigs) and the mirror > system instead. > The current download page has lots of references to dist.a.o; please replace > thes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22206) dist.apache.org must not be used for public downloads
[ https://issues.apache.org/jira/browse/HBASE-22206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-22206: -- Attachment: HBASE-22206.master.001.patch > dist.apache.org must not be used for public downloads > - > > Key: HBASE-22206 > URL: https://issues.apache.org/jira/browse/HBASE-22206 > Project: HBase > Issue Type: Bug > Components: website >Reporter: Sebb >Assignee: Dima Spivak >Priority: Major > Attachments: HBASE-22206.master.001.patch, > HBASE-22206.master.001.patch > > > The dist.apache.org server is only intended for use by developers in staging > releases. > It must not be used on public download pages. > Please use www.apache.org/dist (for KEYS, hashes and sigs) and the mirror > system instead. > The current download page has lots of references to dist.a.o; please replace > thes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22213) Create a Java based BulkLoadPartitioner
[ https://issues.apache.org/jira/browse/HBASE-22213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815961#comment-16815961 ] stack commented on HBASE-22213: --- Interested in why you can't do #1 might [~jmspaggi]. > Create a Java based BulkLoadPartitioner > --- > > Key: HBASE-22213 > URL: https://issues.apache.org/jira/browse/HBASE-22213 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.1.4 >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari >Priority: Minor > > We have a scala based partitionner, but not all projects are build in Scala. > We should provide a Java based version of it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22217) HBase shell command proposal : "rit assign all"
[ https://issues.apache.org/jira/browse/HBASE-22217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815960#comment-16815960 ] stack commented on HBASE-22217: --- 'rit assign all' sounds like hbck2 territory. If 'rit', that is of interest but hopefully it a temporary state of affairs. Should it persist, then time to roll up the sleeves and crack the hbck2 tooling? > HBase shell command proposal : "rit assign all" > > > Key: HBASE-22217 > URL: https://issues.apache.org/jira/browse/HBASE-22217 > Project: HBase > Issue Type: New Feature > Components: Operability, Region Assignment, shell >Reporter: Xu Cang >Assignee: Sakthi >Priority: Minor > > HBase shell command proposal : "rit assign all" > > Currently we have shell command "rit" to list all RITs. > It would be handy having a command "rit assign all" to assign all RITs. > This equals to getting the list of RITs from 'rit' command and running > "assign " one by one. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21129) Clean up duplicate codes in #equals and #hashCode methods of Filter
[
https://issues.apache.org/jira/browse/HBASE-21129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815956#comment-16815956
]
Hudson commented on HBASE-21129:
Results for branch branch-2.1
[build #1046 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1046/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1046//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1046//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1046//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Clean up duplicate codes in #equals and #hashCode methods of Filter
> ---
>
> Key: HBASE-21129
> URL: https://issues.apache.org/jira/browse/HBASE-21129
> Project: HBase
> Issue Type: Improvement
> Components: Filters
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Minor
> Fix For: 3.0.0, 2.2.0
>
> Attachments: 21129.addendum, HBASE-21129.master.001.patch,
> HBASE-21129.master.002.patch, HBASE-21129.master.003.patch,
> HBASE-21129.master.004.patch, HBASE-21129.master.005.patch,
> HBASE-21129.master.006.patch, HBASE-21129.master.007.patch,
> HBASE-21129.master.008.patch
>
>
> It is a follow-up of HBASE-19008, aiming to clean up duplicate codes in
> #equals and #hashCode methods.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19008) Add missing equals or hashCode method(s) to stock Filter implementations
[
https://issues.apache.org/jira/browse/HBASE-19008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815955#comment-16815955
]
Hudson commented on HBASE-19008:
Results for branch branch-2.1
[build #1046 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1046/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1046//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1046//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1046//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add missing equals or hashCode method(s) to stock Filter implementations
>
>
> Key: HBASE-19008
> URL: https://issues.apache.org/jira/browse/HBASE-19008
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Labels: filter
> Fix For: 3.0.0, 2.2.0
>
> Attachments: Filters.png, HBASE-19008-1.patch, HBASE-19008-10.patch,
> HBASE-19008-11.patch, HBASE-19008-12.patch, HBASE-19008-13.patch,
> HBASE-19008-14.patch, HBASE-19008-15.patch, HBASE-19008-16.patch,
> HBASE-19008-17.patch, HBASE-19008-18.patch, HBASE-19008-2.patch,
> HBASE-19008-20.patch, HBASE-19008-21.patch, HBASE-19008-3.patch,
> HBASE-19008-4.patch, HBASE-19008-5.patch, HBASE-19008-6.patch,
> HBASE-19008-7.patch, HBASE-19008-8.patch, HBASE-19008-9.patch,
> HBASE-19008.patch
>
>
> In HBASE-15410, [~mdrob] reminded me that Filter implementations may not
> write {{equals}} or {{hashCode}} method(s).
> This issue is to add missing {{equals}} or {{hashCode}} method(s) to stock
> Filter implementations such as KeyOnlyFilter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22144) MultiRowRangeFilter does not work with reversed scans
[
https://issues.apache.org/jira/browse/HBASE-22144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815957#comment-16815957
]
Hudson commented on HBASE-22144:
Results for branch branch-2.1
[build #1046 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1046/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1046//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1046//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1046//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> MultiRowRangeFilter does not work with reversed scans
> -
>
> Key: HBASE-22144
> URL: https://issues.apache.org/jira/browse/HBASE-22144
> Project: HBase
> Issue Type: Bug
> Components: Filters, scan
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.3.0, 2.0.6, 2.1.5
>
> Attachments: HBASE-22144.001.patch, HBASE-22144.002.patch,
> HBASE-22144.002.patch
>
>
> It appears that MultiRowRangeFilter was never written to function with
> reverse scans. There is too much logic that operates with the assumption that
> we are always moving "forward" through increasing ranges. It needs to be
> rewritten to "traverse" forward or backward, given how the context of the
> scan being used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22212) [1.x] Backport missing filter improvements
[
https://issues.apache.org/jira/browse/HBASE-22212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815954#comment-16815954
]
HBase QA commented on HBASE-22212:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
42s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} branch-1 passed with JDK v1.8.0_202 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} branch-1 passed with JDK v1.7.0_211 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
54s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
36s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} branch-1 passed with JDK v1.8.0_202 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} branch-1 passed with JDK v1.7.0_211 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed with JDK v1.8.0_202 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed with JDK v1.7.0_211 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} hbase-client: The patch generated 1 new + 364
unchanged - 11 fixed = 365 total (was 375) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
35s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 37s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed with JDK v1.8.0_202 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed with JDK v1.7.0_211 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
29s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}113m
35s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}154m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |
[jira] [Commented] (HBASE-15560) TinyLFU-based BlockCache
[ https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815950#comment-16815950 ] Sean Busbey commented on HBASE-15560: - okay in that case I'm +1 > TinyLFU-based BlockCache > > > Key: HBASE-15560 > URL: https://issues.apache.org/jira/browse/HBASE-15560 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Affects Versions: 2.0.0 >Reporter: Ben Manes >Assignee: Ben Manes >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, bc.hit.count, bc.miss.count, branch-1.tinylfu.txt, gets, > run_ycsb_c.sh, run_ycsb_loading.sh, tinylfu.patch > > > LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and > recency of the working set. It achieves concurrency by using an O( n ) > background thread to prioritize the entries and evict. Accessing an entry is > O(1) by a hash table lookup, recording its logical access time, and setting a > frequency flag. A write is performed in O(1) time by updating the hash table > and triggering an async eviction thread. This provides ideal concurrency and > minimizes the latencies by penalizing the thread instead of the caller. > However the policy does not age the frequencies and may not be resilient to > various workload patterns. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > Concurrency is achieved by buffering and replaying the operations, similar to > a write-ahead log. A read is recorded into a striped ring buffer and writes > to a queue. The operations are applied in batches under a try-lock by an > asynchronous thread, thereby track the usage pattern without incurring high > latencies > ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]). > In YCSB benchmarks the results were inconclusive. For a large cache (99% hit > rates) the two caches have near identical throughput and latencies with > LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a > 1-4% hit rate improvement and therefore lower latencies. The lack luster > result is because a synthetic Zipfian distribution is used, which SLRU > performs optimally. In a more varied, real-world workload we'd expect to see > improvements by being able to make smarter predictions. > The provided patch implements BlockCache using the > [Caffeine|https://github.com/ben-manes/caffeine] caching library (see > HighScalability > [article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]). > Edward Bortnikov and Eshcar Hillel have graciously provided guidance for > evaluating this patch ([github > branch|https://github.com/ben-manes/hbase/tree/tinylfu]). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22144) MultiRowRangeFilter does not work with reversed scans
[
https://issues.apache.org/jira/browse/HBASE-22144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815946#comment-16815946
]
Hudson commented on HBASE-22144:
Results for branch branch-2
[build #1815 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1815/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1815//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1815//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1815//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> MultiRowRangeFilter does not work with reversed scans
> -
>
> Key: HBASE-22144
> URL: https://issues.apache.org/jira/browse/HBASE-22144
> Project: HBase
> Issue Type: Bug
> Components: Filters, scan
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.3.0, 2.0.6, 2.1.5
>
> Attachments: HBASE-22144.001.patch, HBASE-22144.002.patch,
> HBASE-22144.002.patch
>
>
> It appears that MultiRowRangeFilter was never written to function with
> reverse scans. There is too much logic that operates with the assumption that
> we are always moving "forward" through increasing ranges. It needs to be
> rewritten to "traverse" forward or backward, given how the context of the
> scan being used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21301) Heatmap for key access patterns
[ https://issues.apache.org/jira/browse/HBASE-21301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815944#comment-16815944 ] Archana Katiyar commented on HBASE-21301: - Hi Andrew, I move out of HBase team around 4 months back. I am sorry, I should have informed you earlier. On Thu, 11 Apr 2019 at 10:28 PM, Andrew Purtell (JIRA) > Heatmap for key access patterns > --- > > Key: HBASE-21301 > URL: https://issues.apache.org/jira/browse/HBASE-21301 > Project: HBase > Issue Type: Improvement >Reporter: Archana Katiyar >Assignee: Archana Katiyar >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.3.0 > > Attachments: HBASE-21301.v0.master.patch > > > Google recently released a beta feature for Cloud Bigtable which presents a > heat map of the keyspace. *Given how hotspotting comes up now and again here, > this is a good idea for giving HBase ops a tool to be proactive about it.* > >>> > Additionally, we are announcing the beta version of Key Visualizer, a > visualization tool for Cloud Bigtable key access patterns. Key Visualizer > helps debug performance issues due to unbalanced access patterns across the > key space, or single rows that are too large or receiving too much read or > write activity. With Key Visualizer, you get a heat map visualization of > access patterns over time, along with the ability to zoom into specific key > or time ranges, or select a specific row to find the full row key ID that's > responsible for a hotspot. Key Visualizer is automatically enabled for Cloud > Bigtable clusters with sufficient data or activity, and does not affect Cloud > Bigtable cluster performance. > <<< > From > [https://cloudplatform.googleblog.com/2018/07/on-gcp-your-database-your-way.html] > (Copied this description from the write-up by [~apurtell], thanks Andrew.) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-12829) Request count in RegionLoad may not accurate to compute the load cost for region
[
https://issues.apache.org/jira/browse/HBASE-12829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815935#comment-16815935
]
Biju Nair edited comment on HBASE-12829 at 4/12/19 3:39 AM:
In the current version of SLB,
[Read-writeRequestCostFunction|https://github.com/apache/hbase/blob/baf3ae80f5588ee848176adefc9f56818458a387/hbaseserver/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java#L1465]
extends
[CostFromRegionLoadAsRateFunction|https://github.com/apache/hbase/blob/baf3ae80f5588ee848176adefc9f56818458a387/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java#L1436]
which in turn uses the [average of the region requests stored for a
period|https://github.com/apache/hbase/blob/baf3ae80f5588ee848176adefc9f56818458a387/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java#L1443]
to calculate cost which seems to address this issue. Can this be closed?
was (Author: gsbiju):
In the current version of SLB,
[Read-writeRequestCostFunction|https://github.com/apache/hbase/blob/baf3ae80f5588ee848176adefc9f56818458a387/hbaseserver/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java#L1465]
extends
[CostFromRegionLoadAsRateFunction|https://github.com/apache/hbase/blob/baf3ae80f5588ee848176adefc9f56818458a387/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java#L1436]
which in turn uses the [average of the region requests stored for a
period|https://github.com/apache/hbase/blob/baf3ae80f5588ee848176adefc9f56818458a387/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java#L1443]
which seems to address this issue. Can this be closed?
> Request count in RegionLoad may not accurate to compute the load cost for
> region
>
>
> Key: HBASE-12829
> URL: https://issues.apache.org/jira/browse/HBASE-12829
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Affects Versions: 0.99.2
>Reporter: Jianwei Cui
>Priority: Minor
>
> StochasticLoadBalancer#RequestCostFunction(ReadRequestCostFunction and
> WriteRequestCostFunction) will compute load cost for a region based on a
> number of remembered region loads. Each region load records the total count
> for read/write request at reported time since it opened. However, the request
> count will be reset if region moved, making the new reported count could not
> represent the total request. For example, if a region has high write
> throughput, the WrtieRequest in region load will be very big after onlined
> for a long time, then if the region moved, the new WriteRequest will be much
> smaller, making the region contributes much smaller to the cost of its
> belonging rs. We may need to consider the region open time to get more
> accurate region load.
> As another way, how about using read/write request count at each time slots
> instead of total request count? The total count will make older read/write
> request throughput contribute more to the cost by
> CostFromRegionLoadFunction#getRegionLoadCost:
> {code}
> protected double getRegionLoadCost(Collection regionLoadList)
> {
> double cost = 0;
> for (RegionLoad rl : regionLoadList) {
> double toAdd = getCostFromRl(rl);
> if (cost == 0) {
> cost = toAdd;
> } else {
> cost = (.5 * cost) + (.5 * toAdd);
> }
> }
> return cost;
> }
> {code}
> For example, assume the balancer now remembers three loads for a region at
> time t1, t2, t3(t1 < t2 < t3), the write request is w1, w2, w3 respectively
> for time slots [0, t1), [t1, t2), [t2, t3), so the WriteRequest in the region
> load at t1, t2, t3 will be w1, w1 + w2, w1 + w2 + w3 and the WriteRequest
> cost will be:
> {code}
> 0.5 * (w1 + w2 + w3) + 0.25 * (w1 + w2) + 0.25 * w1 = w1 + 0.75 * w2 +
> 0.5 * w3
> {code}
> The w1 contributes more to the cost than w2 and w3. However, intuitively, I
> think the recent read/write throughput should represent the current load of
> the region better than the older ones. Therefore, how about using w1, w2 and
> w3 directly when computing? Then, the cost will become:
> {code}
> 0.25 * w1 + 0.25 * w2 + 0.5 * w3
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-12829) Request count in RegionLoad may not accurate to compute the load cost for region
[
https://issues.apache.org/jira/browse/HBASE-12829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815935#comment-16815935
]
Biju Nair commented on HBASE-12829:
---
In the current version of SLB,
[Read-writeRequestCostFunction|https://github.com/apache/hbase/blob/baf3ae80f5588ee848176adefc9f56818458a387/hbaseserver/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java#L1465]
extends
[CostFromRegionLoadAsRateFunction|https://github.com/apache/hbase/blob/baf3ae80f5588ee848176adefc9f56818458a387/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java#L1436]
which in turn uses the [average of the region requests stored for a
period|https://github.com/apache/hbase/blob/baf3ae80f5588ee848176adefc9f56818458a387/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java#L1443]
which seems to address this issue. Can this be closed?
> Request count in RegionLoad may not accurate to compute the load cost for
> region
>
>
> Key: HBASE-12829
> URL: https://issues.apache.org/jira/browse/HBASE-12829
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Affects Versions: 0.99.2
>Reporter: Jianwei Cui
>Priority: Minor
>
> StochasticLoadBalancer#RequestCostFunction(ReadRequestCostFunction and
> WriteRequestCostFunction) will compute load cost for a region based on a
> number of remembered region loads. Each region load records the total count
> for read/write request at reported time since it opened. However, the request
> count will be reset if region moved, making the new reported count could not
> represent the total request. For example, if a region has high write
> throughput, the WrtieRequest in region load will be very big after onlined
> for a long time, then if the region moved, the new WriteRequest will be much
> smaller, making the region contributes much smaller to the cost of its
> belonging rs. We may need to consider the region open time to get more
> accurate region load.
> As another way, how about using read/write request count at each time slots
> instead of total request count? The total count will make older read/write
> request throughput contribute more to the cost by
> CostFromRegionLoadFunction#getRegionLoadCost:
> {code}
> protected double getRegionLoadCost(Collection regionLoadList)
> {
> double cost = 0;
> for (RegionLoad rl : regionLoadList) {
> double toAdd = getCostFromRl(rl);
> if (cost == 0) {
> cost = toAdd;
> } else {
> cost = (.5 * cost) + (.5 * toAdd);
> }
> }
> return cost;
> }
> {code}
> For example, assume the balancer now remembers three loads for a region at
> time t1, t2, t3(t1 < t2 < t3), the write request is w1, w2, w3 respectively
> for time slots [0, t1), [t1, t2), [t2, t3), so the WriteRequest in the region
> load at t1, t2, t3 will be w1, w1 + w2, w1 + w2 + w3 and the WriteRequest
> cost will be:
> {code}
> 0.5 * (w1 + w2 + w3) + 0.25 * (w1 + w2) + 0.25 * w1 = w1 + 0.75 * w2 +
> 0.5 * w3
> {code}
> The w1 contributes more to the cost than w2 and w3. However, intuitively, I
> think the recent read/write throughput should represent the current load of
> the region better than the older ones. Therefore, how about using w1, w2 and
> w3 directly when computing? Then, the cost will become:
> {code}
> 0.25 * w1 + 0.25 * w2 + 0.5 * w3
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-10761) StochasticLoadBalancer still uses SimpleLoadBalancer's needBalance logic
[ https://issues.apache.org/jira/browse/HBASE-10761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815932#comment-16815932 ] stack edited comment on HBASE-10761 at 4/12/19 3:33 AM: Makes sense [~gsbiju] Thanks. Resolving as implemented. Can open new issue if need more (this one is way old anyways). was (Author: stack): Makes sense [~gsbiju] Thanks. Resolving as implemented. > StochasticLoadBalancer still uses SimpleLoadBalancer's needBalance logic > > > Key: HBASE-10761 > URL: https://issues.apache.org/jira/browse/HBASE-10761 > Project: HBase > Issue Type: Bug > Components: Balancer >Affects Versions: 0.98.0 >Reporter: Victor Xu >Priority: Major > Attachments: HBASE_10761.patch, HBASE_10761_v2.patch > > > StochasticLoadBalancer has become the default balancer since 0.98.0. But its > balanceCluster method still uses the BaseLoadBalancer.needBalance() which is > originally designed for SimpleLoadBalancer. It's all based on the number of > regions on the regionservers. > This can cause such a problem: when the cluster has some Hot Spot Region, the > balance process may not be triggered because the numbers of regions on the > RegionServers are averaged. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-10761) StochasticLoadBalancer still uses SimpleLoadBalancer's needBalance logic
[ https://issues.apache.org/jira/browse/HBASE-10761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-10761: -- Resolution: Implemented Status: Resolved (was: Patch Available) Makes sense [~gsbiju] Thanks. Resolving as implemented. > StochasticLoadBalancer still uses SimpleLoadBalancer's needBalance logic > > > Key: HBASE-10761 > URL: https://issues.apache.org/jira/browse/HBASE-10761 > Project: HBase > Issue Type: Bug > Components: Balancer >Affects Versions: 0.98.0 >Reporter: Victor Xu >Priority: Major > Attachments: HBASE_10761.patch, HBASE_10761_v2.patch > > > StochasticLoadBalancer has become the default balancer since 0.98.0. But its > balanceCluster method still uses the BaseLoadBalancer.needBalance() which is > originally designed for SimpleLoadBalancer. It's all based on the number of > regions on the regionservers. > This can cause such a problem: when the cluster has some Hot Spot Region, the > balance process may not be triggered because the numbers of regions on the > RegionServers are averaged. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-10761) StochasticLoadBalancer still uses SimpleLoadBalancer's needBalance logic
[
https://issues.apache.org/jira/browse/HBASE-10761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815930#comment-16815930
]
HBase QA commented on HBASE-10761:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HBASE-10761 does not apply to master. Rebase required? Wrong
Branch? See
https://yetus.apache.org/documentation/in-progress/precommit-patchnames for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-10761 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12634893/HBASE_10761_v2.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/70/console |
| Powered by | Apache Yetus 0.9.0 http://yetus.apache.org |
This message was automatically generated.
> StochasticLoadBalancer still uses SimpleLoadBalancer's needBalance logic
>
>
> Key: HBASE-10761
> URL: https://issues.apache.org/jira/browse/HBASE-10761
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Affects Versions: 0.98.0
>Reporter: Victor Xu
>Priority: Major
> Attachments: HBASE_10761.patch, HBASE_10761_v2.patch
>
>
> StochasticLoadBalancer has become the default balancer since 0.98.0. But its
> balanceCluster method still uses the BaseLoadBalancer.needBalance() which is
> originally designed for SimpleLoadBalancer. It's all based on the number of
> regions on the regionservers.
> This can cause such a problem: when the cluster has some Hot Spot Region, the
> balance process may not be triggered because the numbers of regions on the
> RegionServers are averaged.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-10761) StochasticLoadBalancer still uses SimpleLoadBalancer's needBalance logic
[
https://issues.apache.org/jira/browse/HBASE-10761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815927#comment-16815927
]
Biju Nair edited comment on HBASE-10761 at 4/12/19 3:21 AM:
Currently
[SLB|https://github.com/apache/hbase/blob/baf3ae80f5588ee848176adefc9f56818458a387/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java#L275]
has logic to check {{needBalance}} and not use the one from SimpleLoadBalancer
which seems to satisfy this requirement. Can this be closed?
was (Author: gsbiju):
Currently
[SLB|https://github.com/apache/hbase/blob/baf3ae80f5588ee848176adefc9f56818458a387/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java#L275]
has logic to check {{needBalance}} which seems to satisfy this requirement.
Can this be closed?
> StochasticLoadBalancer still uses SimpleLoadBalancer's needBalance logic
>
>
> Key: HBASE-10761
> URL: https://issues.apache.org/jira/browse/HBASE-10761
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Affects Versions: 0.98.0
>Reporter: Victor Xu
>Priority: Major
> Attachments: HBASE_10761.patch, HBASE_10761_v2.patch
>
>
> StochasticLoadBalancer has become the default balancer since 0.98.0. But its
> balanceCluster method still uses the BaseLoadBalancer.needBalance() which is
> originally designed for SimpleLoadBalancer. It's all based on the number of
> regions on the regionservers.
> This can cause such a problem: when the cluster has some Hot Spot Region, the
> balance process may not be triggered because the numbers of regions on the
> RegionServers are averaged.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-10761) StochasticLoadBalancer still uses SimpleLoadBalancer's needBalance logic
[
https://issues.apache.org/jira/browse/HBASE-10761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815927#comment-16815927
]
Biju Nair commented on HBASE-10761:
---
Currently
[SLB|https://github.com/apache/hbase/blob/baf3ae80f5588ee848176adefc9f56818458a387/hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/StochasticLoadBalancer.java#L275]
has logic to check {{needBalance}} which seems to satisfy this requirement.
Can this be closed?
> StochasticLoadBalancer still uses SimpleLoadBalancer's needBalance logic
>
>
> Key: HBASE-10761
> URL: https://issues.apache.org/jira/browse/HBASE-10761
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Affects Versions: 0.98.0
>Reporter: Victor Xu
>Priority: Major
> Attachments: HBASE_10761.patch, HBASE_10761_v2.patch
>
>
> StochasticLoadBalancer has become the default balancer since 0.98.0. But its
> balanceCluster method still uses the BaseLoadBalancer.needBalance() which is
> originally designed for SimpleLoadBalancer. It's all based on the number of
> regions on the regionservers.
> This can cause such a problem: when the cluster has some Hot Spot Region, the
> balance process may not be triggered because the numbers of regions on the
> RegionServers are averaged.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22144) MultiRowRangeFilter does not work with reversed scans
[
https://issues.apache.org/jira/browse/HBASE-22144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815926#comment-16815926
]
Hudson commented on HBASE-22144:
Results for branch master
[build #926 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/926/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/926//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/926//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/926//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> MultiRowRangeFilter does not work with reversed scans
> -
>
> Key: HBASE-22144
> URL: https://issues.apache.org/jira/browse/HBASE-22144
> Project: HBase
> Issue Type: Bug
> Components: Filters, scan
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.3.0, 2.0.6, 2.1.5
>
> Attachments: HBASE-22144.001.patch, HBASE-22144.002.patch,
> HBASE-22144.002.patch
>
>
> It appears that MultiRowRangeFilter was never written to function with
> reverse scans. There is too much logic that operates with the assumption that
> we are always moving "forward" through increasing ranges. It needs to be
> rewritten to "traverse" forward or backward, given how the context of the
> scan being used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22194) Snapshot unittests fail on Windows due to invalid file path uri
[
https://issues.apache.org/jira/browse/HBASE-22194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815925#comment-16815925
]
Hudson commented on HBASE-22194:
Results for branch master
[build #926 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/926/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/926//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/926//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/926//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Snapshot unittests fail on Windows due to invalid file path uri
> ---
>
> Key: HBASE-22194
> URL: https://issues.apache.org/jira/browse/HBASE-22194
> Project: HBase
> Issue Type: Bug
> Components: regionserver, test
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Bahram Chehrazy
>Assignee: Bahram Chehrazy
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
> Attachments: unittest-fix-for-windows.patch
>
>
> These unittests are failing on Windows because the temporary snapshot file
> path is not valid.
> hadoop.hbase.client.TestSnapshotTemporaryDirectory
> hadoop.hbase.client.TestSnapshotDFSTemporaryDirectory
> hadoop.hbase.client.TestSnapshotTemporaryDirectoryWithRegionReplica
>
> The error is:
>
> 2019-04-08 23:42:02,080 ERROR [master/MININT-2D9TFVB:0:becomeActiveMaster]
> helpers.MarkerIgnoringBase(159): * ABORTING master
> minint-2d9tfvb.northamerica.corp.microsoft.com,57169,1554792118500: Unhandled
> exception. Starting shutdown. *
> java.lang.IllegalArgumentException: *Wrong FS:
> file://C:\src\hbase\hbase-server/2f6562be-fe12-49a4-b370-2b6928e5aa72,
> expected: file:///*
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:647)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:82)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:606)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.delete(ChecksumFileSystem.java:638)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.resetTempDir(SnapshotManager.java:298)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.initialize(SnapshotManager.java:1186)
> at
> org.apache.hadoop.hbase.procedure.MasterProcedureManagerHost.initialize(MasterProcedureManagerHost.java:50)
> at
> org.apache.hadoop.hbase.master.HMaster.initializeZKBasedSystemTrackers(HMaster.java:828)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1004)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2373)
> at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:603)
> at java.lang.Thread.run(Thread.java:748)
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19762) Fix Checkstyle errors in hbase-http
[
https://issues.apache.org/jira/browse/HBASE-19762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815924#comment-16815924
]
Hudson commented on HBASE-19762:
Results for branch master
[build #926 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/926/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/926//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/926//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/926//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Fix Checkstyle errors in hbase-http
> ---
>
> Key: HBASE-19762
> URL: https://issues.apache.org/jira/browse/HBASE-19762
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jan Hentschel
>Assignee: Jan Hentschel
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HBASE-19762.master.001.patch,
> HBASE-19762.master.002.patch, HBASE-19762.master.003.patch,
> HBASE-19762.master.004.patch
>
>
> Fix the remaining Checkstyle errors in the *hbase-http* module and enable
> Checkstyle to fail on violations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21152) Add parameter validations for commands in HBase shell
[
https://issues.apache.org/jira/browse/HBASE-21152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815921#comment-16815921
]
Biju Nair commented on HBASE-21152:
---
If {{status}} is used as a parameter for a shell command for e.g. {{describe
status}} then {{status}} gets executed as a command followed by the user
intended command i.e. {{describe}}. Looks like a broader issues and not
confined to {{balance_switch}} command alone.
> Add parameter validations for commands in HBase shell
> -
>
> Key: HBASE-21152
> URL: https://issues.apache.org/jira/browse/HBASE-21152
> Project: HBase
> Issue Type: Improvement
> Components: Balancer, shell
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
>
> One of our customers got confused with "balance_switch" command in HBase
> shell.
> They mistakenly ran "balance_swich status" command instead of
> "balancer_enabled" command to see if the balancer is enabled. However, like
> the following, it didn't cause any errors and it looks like the command was
> successful.
> {code}
> hbase> balance_switch status
> 1 active master, 0 backup masters, 1 servers, 0 dead, 2. average load
> Took 0.0055 seconds
> Previous balancer state : true
> Took 0.0041 seconds
> => "true"
> {code}
> To make matters worse, the "balance_swich status" command will make the
> balancer disabled.
> Of course, the command was wrong but I think that's a little bit confusing.
> So I think we need to add parameter validations for commands in HBase shell.
> I'll check if we need to add validations for any other commands in this Jira.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21152) Add parameter validations for commands in HBase shell
[
https://issues.apache.org/jira/browse/HBASE-21152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815921#comment-16815921
]
Biju Nair edited comment on HBASE-21152 at 4/12/19 3:10 AM:
If {{status}} is used as a parameter for a shell command for e.g. {{describe
status}} then {{status}} gets executed as a command followed by the user
intended command i.e. {{describe}}. Looks like a broader issue and not confined
to {{balance_switch}} command alone.
was (Author: gsbiju):
If {{status}} is used as a parameter for a shell command for e.g. {{describe
status}} then {{status}} gets executed as a command followed by the user
intended command i.e. {{describe}}. Looks like a broader issues and not
confined to {{balance_switch}} command alone.
> Add parameter validations for commands in HBase shell
> -
>
> Key: HBASE-21152
> URL: https://issues.apache.org/jira/browse/HBASE-21152
> Project: HBase
> Issue Type: Improvement
> Components: Balancer, shell
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
>
> One of our customers got confused with "balance_switch" command in HBase
> shell.
> They mistakenly ran "balance_swich status" command instead of
> "balancer_enabled" command to see if the balancer is enabled. However, like
> the following, it didn't cause any errors and it looks like the command was
> successful.
> {code}
> hbase> balance_switch status
> 1 active master, 0 backup masters, 1 servers, 0 dead, 2. average load
> Took 0.0055 seconds
> Previous balancer state : true
> Took 0.0041 seconds
> => "true"
> {code}
> To make matters worse, the "balance_swich status" command will make the
> balancer disabled.
> Of course, the command was wrong but I think that's a little bit confusing.
> So I think we need to add parameter validations for commands in HBase shell.
> I'll check if we need to add validations for any other commands in this Jira.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-9741) Remove hbase.regions.slop from hbase-default.xml
[
https://issues.apache.org/jira/browse/HBASE-9741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815920#comment-16815920
]
Biju Nair commented on HBASE-9741:
--
Current
[hbase-default.xml|https://github.com/apache/hbase/blob/f22b8ade631367d584b7065c7c76b3e0b6eca97b/hbase-common/src/main/resources/hbase-default.xml#L624-L630]
has the smaller {{slop}} for SLB which is enabled by default and HBase book
documents about the higher {{slop}} value used by SimpleLoadBalancer. Can we
close this ticket?
> Remove hbase.regions.slop from hbase-default.xml
>
>
> Key: HBASE-9741
> URL: https://issues.apache.org/jira/browse/HBASE-9741
> Project: HBase
> Issue Type: Bug
> Components: Balancer
>Reporter: Elliott Clark
>Priority: Major
> Labels: beginner
> Attachments: HBASE-9741-v0.patch, HBASE-9741-v1.patch,
> HBASE-9741-v3.patch, HBASE-9741-v4.patch
>
>
> Different balancers have different slop default values. We should remove
> hbase.regions.slop from hbase-default.xml
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HBASE-8549) Integrate Favored Nodes into StochasticLoadBalancer
[ https://issues.apache.org/jira/browse/HBASE-8549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-8549. -- Resolution: Implemented I'll take your word for it [~gsbiju]. Implemented by HBASE-16942 > Integrate Favored Nodes into StochasticLoadBalancer > --- > > Key: HBASE-8549 > URL: https://issues.apache.org/jira/browse/HBASE-8549 > Project: HBase > Issue Type: Bug > Components: Balancer >Reporter: Elliott Clark >Priority: Major > Attachments: HBASE-8549-0.patch > > > Right now we have a FavoredNodeLoadBalancer. It would be pretty easy to > integrate the favored node list into the strochastic balancer. Then we would > have the best of both worlds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-16942) Add FavoredStochasticLoadBalancer and FN Candidate generators
[ https://issues.apache.org/jira/browse/HBASE-16942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815916#comment-16815916 ] stack commented on HBASE-16942: --- Do I have to do anything to turn this on [~thiruvel]? If so, maybe release note it please sir? Otherwise, folks are unlikely to find this nice addition. Thanks sir. > Add FavoredStochasticLoadBalancer and FN Candidate generators > - > > Key: HBASE-16942 > URL: https://issues.apache.org/jira/browse/HBASE-16942 > Project: HBase > Issue Type: Sub-task > Components: FavoredNodes >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-16942.master.001.patch, > HBASE-16942.master.002.patch, HBASE-16942.master.003.patch, > HBASE-16942.master.004.patch, HBASE-16942.master.005.patch, > HBASE-16942.master.006.patch, HBASE-16942.master.007.patch, > HBASE-16942.master.008.patch, HBASE-16942.master.009.patch, > HBASE-16942.master.010.patch, HBASE-16942.master.011.patch, > HBASE-16942.master.012.patch, HBASE_16942_rough_draft.patch > > > This deals with the balancer based enhancements to favored nodes patch as > discussed in HBASE-15532. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22020) upgrade to yetus 0.9.0
[
https://issues.apache.org/jira/browse/HBASE-22020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815913#comment-16815913
]
Hudson commented on HBASE-22020:
Results for branch HBASE-22020
[build #9 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22020/9/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22020/9//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22020/9//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-22020/9//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> upgrade to yetus 0.9.0
> --
>
> Key: HBASE-22020
> URL: https://issues.apache.org/jira/browse/HBASE-22020
> Project: HBase
> Issue Type: Task
> Components: build, community
>Reporter: stack
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-22020-branch-1.v1.patch, HBASE-22020.0.patch,
> HBASE-22020.1.patch
>
>
> branch-1/jdk7 checkstyle dtd xml parse complaint; "script engine for language
> js can not be found"
> See parent for some context. Checkstyle references dtds that were hosted on
> puppycrawl, then on sourceforge up until ten days ago. Nightlies are failing
> for among other reasons, complaint that there is bad xml in the build...
> notably, the unresolvable DTDs.
> I'd just update the DTDs but there is a need for a js engine some where and
> openjdk7 doesn't ship with one (openjdk8 does). That needs addressing and
> then we can backport the parent issue...
> See
> https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-1/710/artifact/output-general/xml.txt
> ... which in case its rolled away, is filled with this message:
> "script engine for language js can not be found"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-10075) add a locality-aware balancer
[ https://issues.apache.org/jira/browse/HBASE-10075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815910#comment-16815910 ] Biju Nair commented on HBASE-10075: --- With SLB taking into account data locality to calculate the cost and come-up with target balanced cluster, looks like the idea behind this request is taken care. no? Can this be closed? > add a locality-aware balancer > - > > Key: HBASE-10075 > URL: https://issues.apache.org/jira/browse/HBASE-10075 > Project: HBase > Issue Type: New Feature > Components: Balancer >Affects Versions: 0.94.12 >Reporter: Chengxiang Li >Priority: Major > > basic idea: > during rebalance. For each region server, iterate regions, give each region a > balance score, remove the lowest one until the region number of region server > reach avg floor. > during assignment. match to-be-assigned regions with each active region > server as pairs, give each pair a balance score, the highest win this region. > here is the mark formula: > (1 – tableRegionNumberOnServer/allTableRegionNumber) * tableBalancerWeight > + (1 – regionNumberOnServer/allRegionNumber) * serverBalancerWeight + > regionBlockSizeOnServer/regionBlockSize * localityWeight > + (previousServer?1:0) * stickinessWeight > there are 4 factors which would influence the final balance score: > 1. region balance > 2. table region balance > 3. region locality > 4. region stickiness > through adjust the weight of these 4 factors, we can balance the cluster in > different strategy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-20151) Bug with SingleColumnValueFilter and FamilyFilter
[
https://issues.apache.org/jira/browse/HBASE-20151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu reassigned HBASE-20151:
Assignee: Zheng Hu (was: Reid Chan)
> Bug with SingleColumnValueFilter and FamilyFilter
> -
>
> Key: HBASE-20151
> URL: https://issues.apache.org/jira/browse/HBASE-20151
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.0.1, 1.4.5
> Environment: MacOS 10.13.3
> HBase 1.3.1
>Reporter: Steven Sadowski
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
> Attachments: HBASE-20151.master.001.patch,
> HBASE-20151.master.002.patch, HBASE-20151.master.003.patch,
> HBASE-20151.master.004.patch, HBASE-20151.master.004.patch,
> HBASE-20151.master.005.patch, HBASE-20151.master.006.patch,
> filter-list-type.v1.txt
>
>
> When running the following queries, the result is sometimes return correctly
> and other times incorrectly based on the qualifier queried.
> Setup:
> {code:java}
> create 'test', 'a', 'b'
> test = get_table 'test'
> test.put '1', 'a:1', nil
> test.put '1', 'a:10', nil
> test.put '1', 'b:2', nil
> {code}
>
> This query works fine when the SCVF's qualifier has length 1 (i.e. '1') :
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','1',=,'binary:',true,true) AND
> FamilyFilter(=,'binary:b') )"})
> ROW COLUMN+CELL
> 1column=b:2,
> timestamp=1520455888059, value=
> 1 row(s) in 0.0060 seconds
> {code}
>
> The query should return the same result when passed a qualifier of length 2
> (i.e. '10') :
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','10',=,'binary:',true,true) AND
> FamilyFilter(=,'binary:b') )"})
> ROW COLUMN+CELL
> 0 row(s) in 0.0110 seconds
> {code}
> However, in this case, it does not return any row (expected result would be
> to return the same result as the first query).
>
> Removing the family filter while the qualifier is '10' yields expected
> results:
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','10',=,'binary:',true,true) )"})
> ROW COLUMN+CELL
> 1column=a:1,
> timestamp=1520455887954, value=
> 1column=a:10,
> timestamp=1520455888024, value=
> 1column=b:2,
> timestamp=1520455888059, value=
> 1 row(s) in 0.0140 seconds
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20151) Bug with SingleColumnValueFilter and FamilyFilter
[
https://issues.apache.org/jira/browse/HBASE-20151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815908#comment-16815908
]
Zheng Hu commented on HBASE-20151:
--
Assigned to me firstly, I'll think about this issue, if no good solution,
I'll close this. Thanks.
> Bug with SingleColumnValueFilter and FamilyFilter
> -
>
> Key: HBASE-20151
> URL: https://issues.apache.org/jira/browse/HBASE-20151
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.0.1, 1.4.5
> Environment: MacOS 10.13.3
> HBase 1.3.1
>Reporter: Steven Sadowski
>Assignee: Reid Chan
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
> Attachments: HBASE-20151.master.001.patch,
> HBASE-20151.master.002.patch, HBASE-20151.master.003.patch,
> HBASE-20151.master.004.patch, HBASE-20151.master.004.patch,
> HBASE-20151.master.005.patch, HBASE-20151.master.006.patch,
> filter-list-type.v1.txt
>
>
> When running the following queries, the result is sometimes return correctly
> and other times incorrectly based on the qualifier queried.
> Setup:
> {code:java}
> create 'test', 'a', 'b'
> test = get_table 'test'
> test.put '1', 'a:1', nil
> test.put '1', 'a:10', nil
> test.put '1', 'b:2', nil
> {code}
>
> This query works fine when the SCVF's qualifier has length 1 (i.e. '1') :
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','1',=,'binary:',true,true) AND
> FamilyFilter(=,'binary:b') )"})
> ROW COLUMN+CELL
> 1column=b:2,
> timestamp=1520455888059, value=
> 1 row(s) in 0.0060 seconds
> {code}
>
> The query should return the same result when passed a qualifier of length 2
> (i.e. '10') :
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','10',=,'binary:',true,true) AND
> FamilyFilter(=,'binary:b') )"})
> ROW COLUMN+CELL
> 0 row(s) in 0.0110 seconds
> {code}
> However, in this case, it does not return any row (expected result would be
> to return the same result as the first query).
>
> Removing the family filter while the qualifier is '10' yields expected
> results:
> {code:java}
> test.scan({ FILTER => "(
> SingleColumnValueFilter('a','10',=,'binary:',true,true) )"})
> ROW COLUMN+CELL
> 1column=a:1,
> timestamp=1520455887954, value=
> 1column=a:10,
> timestamp=1520455888024, value=
> 1column=b:2,
> timestamp=1520455888059, value=
> 1 row(s) in 0.0140 seconds
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20993) [Auth] IPC client fallback to simple auth allowed doesn't work
[
https://issues.apache.org/jira/browse/HBASE-20993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815909#comment-16815909
]
Reid Chan commented on HBASE-20993:
---
Got stuck in async(netty) ipc, but i'm still working.
> [Auth] IPC client fallback to simple auth allowed doesn't work
> --
>
> Key: HBASE-20993
> URL: https://issues.apache.org/jira/browse/HBASE-20993
> Project: HBase
> Issue Type: Bug
> Components: Client, IPC/RPC, security
>Affects Versions: 1.2.6, 1.3.2, 1.2.7, 1.4.7
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Critical
> Fix For: 1.5.0, 1.4.10, 1.3.4
>
> Attachments: HBASE-20993.001.patch,
> HBASE-20993.003.branch-1.flowchart.png, HBASE-20993.branch-1.002.patch,
> HBASE-20993.branch-1.003.patch, HBASE-20993.branch-1.004.patch,
> HBASE-20993.branch-1.005.patch, HBASE-20993.branch-1.006.patch,
> HBASE-20993.branch-1.007.patch, HBASE-20993.branch-1.008.patch,
> HBASE-20993.branch-1.009.patch, HBASE-20993.branch-1.009.patch,
> HBASE-20993.branch-1.010.patch, HBASE-20993.branch-1.011.patch,
> HBASE-20993.branch-1.012.patch, HBASE-20993.branch-1.2.001.patch,
> HBASE-20993.branch-1.wip.002.patch, HBASE-20993.branch-1.wip.patch,
> yetus-local-testpatch-output-009.txt
>
>
> It is easily reproducible.
> client's hbase-site.xml: hadoop.security.authentication:kerberos,
> hbase.security.authentication:kerberos,
> hbase.ipc.client.fallback-to-simple-auth-allowed:true, keytab and principal
> are right set
> A simple auth hbase cluster, a kerberized hbase client application.
> application trying to r/w/c/d table will have following exception:
> {code}
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> at
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:58383)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1592)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1530)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1552)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1581)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1738)
> at
> org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4297)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4289)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTableAsyncV2(HBaseAdmin.java:753)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:674)
> at
>
[jira] [Commented] (HBASE-8443) Queue a balancer run when regionservers report in for the first time
[ https://issues.apache.org/jira/browse/HBASE-8443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815907#comment-16815907 ] Biju Nair commented on HBASE-8443: -- Similar to HBASE-3268. Can we close this? [~apurtell] . > Queue a balancer run when regionservers report in for the first time > > > Key: HBASE-8443 > URL: https://issues.apache.org/jira/browse/HBASE-8443 > Project: HBase > Issue Type: Improvement > Components: Balancer >Reporter: Elliott Clark >Priority: Major > > When running integration tests it's apparent that lots of region servers sit > for quite a while in between balancer runs. > I propose > * Queuing one balancer run that will run 30 seconds after a new region server > checks in. > * Reset the balancer period if we queue a balancer run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-8549) Integrate Favored Nodes into StochasticLoadBalancer
[ https://issues.apache.org/jira/browse/HBASE-8549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815905#comment-16815905 ] Biju Nair commented on HBASE-8549: -- HBASE-16942 seems to take care of this requirement? Can be closed? > Integrate Favored Nodes into StochasticLoadBalancer > --- > > Key: HBASE-8549 > URL: https://issues.apache.org/jira/browse/HBASE-8549 > Project: HBase > Issue Type: Bug > Components: Balancer >Reporter: Elliott Clark >Priority: Major > Attachments: HBASE-8549-0.patch > > > Right now we have a FavoredNodeLoadBalancer. It would be pretty easy to > integrate the favored node list into the strochastic balancer. Then we would > have the best of both worlds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22144) MultiRowRangeFilter does not work with reversed scans
[
https://issues.apache.org/jira/browse/HBASE-22144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815898#comment-16815898
]
Hudson commented on HBASE-22144:
Results for branch branch-2.2
[build #176 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/176/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/176//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/176//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/176//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> MultiRowRangeFilter does not work with reversed scans
> -
>
> Key: HBASE-22144
> URL: https://issues.apache.org/jira/browse/HBASE-22144
> Project: HBase
> Issue Type: Bug
> Components: Filters, scan
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.3.0, 2.0.6, 2.1.5
>
> Attachments: HBASE-22144.001.patch, HBASE-22144.002.patch,
> HBASE-22144.002.patch
>
>
> It appears that MultiRowRangeFilter was never written to function with
> reverse scans. There is too much logic that operates with the assumption that
> we are always moving "forward" through increasing ranges. It needs to be
> rewritten to "traverse" forward or backward, given how the context of the
> scan being used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22217) HBase shell command proposal : "rit assign all"
[ https://issues.apache.org/jira/browse/HBASE-22217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815897#comment-16815897 ] Sakthi commented on HBASE-22217: Would like to work on this one. > HBase shell command proposal : "rit assign all" > > > Key: HBASE-22217 > URL: https://issues.apache.org/jira/browse/HBASE-22217 > Project: HBase > Issue Type: New Feature > Components: Operability, Region Assignment, shell >Reporter: Xu Cang >Assignee: Sakthi >Priority: Minor > > HBase shell command proposal : "rit assign all" > > Currently we have shell command "rit" to list all RITs. > It would be handy having a command "rit assign all" to assign all RITs. > This equals to getting the list of RITs from 'rit' command and running > "assign " one by one. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-22217) HBase shell command proposal : "rit assign all"
[ https://issues.apache.org/jira/browse/HBASE-22217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sakthi reassigned HBASE-22217: -- Assignee: Sakthi > HBase shell command proposal : "rit assign all" > > > Key: HBASE-22217 > URL: https://issues.apache.org/jira/browse/HBASE-22217 > Project: HBase > Issue Type: New Feature > Components: Operability, Region Assignment, shell >Reporter: Xu Cang >Assignee: Sakthi >Priority: Minor > > HBase shell command proposal : "rit assign all" > > Currently we have shell command "rit" to list all RITs. > It would be handy having a command "rit assign all" to assign all RITs. > This equals to getting the list of RITs from 'rit' command and running > "assign " one by one. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22212) [1.x] Backport missing filter improvements
[ https://issues.apache.org/jira/browse/HBASE-22212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-22212: --- Attachment: HBASE-22212.002.branch-1.patch > [1.x] Backport missing filter improvements > -- > > Key: HBASE-22212 > URL: https://issues.apache.org/jira/browse/HBASE-22212 > Project: HBase > Issue Type: Bug > Components: Filters >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 1.5.0, 1.4.10 > > Attachments: HBASE-22212.001.branch-1.patch, > HBASE-22212.002.branch-1.patch > > > HBASE-19008 and HBASE-21129 were never backported beyond branch-2. I can't > find any reason that this was not done. Despite these being public-tagged > classes, no incompatible changes were added. > The lack of these changes prevents HBASE-22144 from being backported cleanly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-16513) Documentation for new RowIndex DataBlockEncoding
[ https://issues.apache.org/jira/browse/HBASE-16513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin resolved HBASE-16513. -- Resolution: Resolved Fix Version/s: (was: 1.5.0) (was: 3.0.0) > Documentation for new RowIndex DataBlockEncoding > > > Key: HBASE-16513 > URL: https://issues.apache.org/jira/browse/HBASE-16513 > Project: HBase > Issue Type: Improvement > Components: documentation, Performance >Reporter: binlijin >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-16513) Documentation for new RowIndex DataBlockEncoding
[ https://issues.apache.org/jira/browse/HBASE-16513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815893#comment-16815893 ] binlijin commented on HBASE-16513: -- it resolved in parent Jira HBASE-16213, so close it > Documentation for new RowIndex DataBlockEncoding > > > Key: HBASE-16513 > URL: https://issues.apache.org/jira/browse/HBASE-16513 > Project: HBase > Issue Type: Improvement > Components: documentation, Performance >Reporter: binlijin >Priority: Major > Fix For: 3.0.0, 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22212) [1.x] Backport missing filter improvements
[
https://issues.apache.org/jira/browse/HBASE-22212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815891#comment-16815891
]
HBase QA commented on HBASE-22212:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} branch-1 passed with JDK v1.8.0_202 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} branch-1 passed with JDK v1.7.0_211 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
57s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
49s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} branch-1 passed with JDK v1.8.0_202 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} branch-1 passed with JDK v1.7.0_211 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed with JDK v1.8.0_202 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} the patch passed with JDK v1.7.0_211 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
34s{color} | {color:red} hbase-client: The patch generated 6 new + 364
unchanged - 11 fixed = 370 total (was 375) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
46s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 43s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed with JDK v1.8.0_202 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed with JDK v1.7.0_211 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
27s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}112m 46s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}139m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
[jira] [Commented] (HBASE-22194) Snapshot unittests fail on Windows due to invalid file path uri
[
https://issues.apache.org/jira/browse/HBASE-22194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815884#comment-16815884
]
Hudson commented on HBASE-22194:
Results for branch branch-1
[build #768 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/768/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/768//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/768//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/768//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Snapshot unittests fail on Windows due to invalid file path uri
> ---
>
> Key: HBASE-22194
> URL: https://issues.apache.org/jira/browse/HBASE-22194
> Project: HBase
> Issue Type: Bug
> Components: regionserver, test
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Bahram Chehrazy
>Assignee: Bahram Chehrazy
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
> Attachments: unittest-fix-for-windows.patch
>
>
> These unittests are failing on Windows because the temporary snapshot file
> path is not valid.
> hadoop.hbase.client.TestSnapshotTemporaryDirectory
> hadoop.hbase.client.TestSnapshotDFSTemporaryDirectory
> hadoop.hbase.client.TestSnapshotTemporaryDirectoryWithRegionReplica
>
> The error is:
>
> 2019-04-08 23:42:02,080 ERROR [master/MININT-2D9TFVB:0:becomeActiveMaster]
> helpers.MarkerIgnoringBase(159): * ABORTING master
> minint-2d9tfvb.northamerica.corp.microsoft.com,57169,1554792118500: Unhandled
> exception. Starting shutdown. *
> java.lang.IllegalArgumentException: *Wrong FS:
> file://C:\src\hbase\hbase-server/2f6562be-fe12-49a4-b370-2b6928e5aa72,
> expected: file:///*
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:647)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:82)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:606)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.delete(ChecksumFileSystem.java:638)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.resetTempDir(SnapshotManager.java:298)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.initialize(SnapshotManager.java:1186)
> at
> org.apache.hadoop.hbase.procedure.MasterProcedureManagerHost.initialize(MasterProcedureManagerHost.java:50)
> at
> org.apache.hadoop.hbase.master.HMaster.initializeZKBasedSystemTrackers(HMaster.java:828)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1004)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2373)
> at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:603)
> at java.lang.Thread.run(Thread.java:748)
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22086) space quota issue: deleting snapshot doesn't update the usage of table
[ https://issues.apache.org/jira/browse/HBASE-22086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815878#comment-16815878 ] Sakthi commented on HBASE-22086: [~elserj], Can we push this in? :) > space quota issue: deleting snapshot doesn't update the usage of table > -- > > Key: HBASE-22086 > URL: https://issues.apache.org/jira/browse/HBASE-22086 > Project: HBase > Issue Type: Bug >Reporter: Ajeet Rai >Assignee: Sakthi >Priority: Minor > Attachments: hbase-22086.master.001.patch, > hbase-22086.master.002.patch, hbase-22086.master.003.patch, > hbase-22086.master.004.patch > > > space quota issue: deleting snapshot doesn't update the usage of table > Steps: 1: > set_quota TYPE => SPACE, TABLE => 'bugatti', LIMIT => '7M', POLICY => > NO_WRITES_COMPACTIONS > 2: ./hbase pe --table="bugatti" --nomapred --rows=200 sequentialWrite 10 > 3: ./hbase pe --table="bugatti" --nomapred --rows=200 sequentialWrite 10 > 4: snapshot 'bugatti','bugatti_snapshot' > 5: ./hbase pe --table="bugatti" --nomapred --rows=200 sequentialWrite 10 > 6: major_compact 'bugatti' > 7: delete_snapshot 'bugatti_snapshot' > now check the usage and observe that it is not getting updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-22218) Shell throws "Unsupported Java version" when tried with Java 11 (run-time)
Sakthi created HBASE-22218:
--
Summary: Shell throws "Unsupported Java version" when tried with
Java 11 (run-time)
Key: HBASE-22218
URL: https://issues.apache.org/jira/browse/HBASE-22218
Project: HBase
Issue Type: Bug
Reporter: Sakthi
Assignee: Sakthi
Following warning is thrown in the shell.
{noformat}
unsupported Java version "11", defaulting to 1.7{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22086) space quota issue: deleting snapshot doesn't update the usage of table
[
https://issues.apache.org/jira/browse/HBASE-22086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815865#comment-16815865
]
HBase QA commented on HBASE-22086:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
22s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
50s{color} | {color:blue} hbase-server in master has 11 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
20s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 14s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
24s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}132m
37s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
55s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}178m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce base:
https://builds.apache.org/job/PreCommit-HBASE-Build/67/artifact/patchprocess/Dockerfile
|
| JIRA Issue | HBASE-22086 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12965648/hbase-22086.master.004.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 3558ad7260db 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 2c8b813810 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
|
[jira] [Updated] (HBASE-22216) "Waiting on master failover to complete" shows 30 to 40 time per millisecond
[ https://issues.apache.org/jira/browse/HBASE-22216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang updated HBASE-22216: Description: "Waiting on master failover to complete" appears 30 to 40 times per *millisecond* from one host when master is initializing. This message is too noisy, epecially when there are issues when master init. Should fix this by either rate-limiting or even revisit is that normal #executeFromState gets called 30-40 times per ms. was: "Waiting on master failover to complete" appears 30 to 40 times per *millisecond* from one host when master is initializing. This message is too noisy, epecially when there are issues when master init. > "Waiting on master failover to complete" shows 30 to 40 time per millisecond > - > > Key: HBASE-22216 > URL: https://issues.apache.org/jira/browse/HBASE-22216 > Project: HBase > Issue Type: Bug > Components: logging, Operability, proc-v2 >Affects Versions: 1.3.0 >Reporter: Xu Cang >Priority: Minor > > "Waiting on master failover to complete" appears 30 to 40 times per > *millisecond* from one host when master is initializing. > This message is too noisy, epecially when there are issues when master init. > Should fix this by either rate-limiting or even revisit is that normal > #executeFromState gets called 30-40 times per ms. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22216) "Waiting on master failover to complete" shows 30 to 40 time per millisecond
[ https://issues.apache.org/jira/browse/HBASE-22216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang updated HBASE-22216: Description: "Waiting on master failover to complete" appears 30 to 40 times per *millisecond* from one host when master is initializing. This message is too noisy, epecially when there are issues when master init. was: "Waiting on master failover to complete" appears 30 to 40 times per *millisecond* from one host when master is initializing. This message is too noisy. Need to fix this. > "Waiting on master failover to complete" shows 30 to 40 time per millisecond > - > > Key: HBASE-22216 > URL: https://issues.apache.org/jira/browse/HBASE-22216 > Project: HBase > Issue Type: Bug > Components: logging, Operability, proc-v2 >Affects Versions: 1.3.0 >Reporter: Xu Cang >Priority: Minor > > "Waiting on master failover to complete" appears 30 to 40 times per > *millisecond* from one host when master is initializing. > This message is too noisy, epecially when there are issues when master init. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22216) "Waiting on master failover to complete" shows 30 to 40 time per millisecond
[ https://issues.apache.org/jira/browse/HBASE-22216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang updated HBASE-22216: Description: "Waiting on master failover to complete" appears 30 to 40 times per *millisecond* from one host when master is initializing. This message is too noisy. Need to fix this. was: "Waiting on master failover to complete" shows 30 to 40 time per millisecond from one host when master is initializing. This message is too noisy. Need to fix this. > "Waiting on master failover to complete" shows 30 to 40 time per millisecond > - > > Key: HBASE-22216 > URL: https://issues.apache.org/jira/browse/HBASE-22216 > Project: HBase > Issue Type: Bug > Components: logging, Operability, proc-v2 >Affects Versions: 1.3.0 >Reporter: Xu Cang >Priority: Minor > > "Waiting on master failover to complete" appears 30 to 40 times per > *millisecond* from one host when master is initializing. > This message is too noisy. Need to fix this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22216) "Waiting on master failover to complete" shows 30 to 40 time per millisecond
[ https://issues.apache.org/jira/browse/HBASE-22216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-22216: Component/s: Operability logging > "Waiting on master failover to complete" shows 30 to 40 time per millisecond > - > > Key: HBASE-22216 > URL: https://issues.apache.org/jira/browse/HBASE-22216 > Project: HBase > Issue Type: Bug > Components: logging, Operability, proc-v2 >Affects Versions: 1.3.0 >Reporter: Xu Cang >Priority: Minor > > "Waiting on master failover to complete" shows 30 to 40 time per millisecond > from one host when master is initializing. > This message is too noisy. Need to fix this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-16218) Eliminate use of UGI.doAs() in AccessController testing
[ https://issues.apache.org/jira/browse/HBASE-16218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling updated HBASE-16218: -- Fix Version/s: (was: 2.3.0) (was: 1.5.0) (was: 3.0.0) No progress from me, and, unfortunately, no bandwidth to handle this in the near future. I've unscheduled from the release branches. Feel free to close it out instead. > Eliminate use of UGI.doAs() in AccessController testing > --- > > Key: HBASE-16218 > URL: https://issues.apache.org/jira/browse/HBASE-16218 > Project: HBase > Issue Type: Sub-task > Components: security >Reporter: Gary Helmling >Assignee: Gary Helmling >Priority: Major > > Many tests for AccessController observer coprocessor hooks make use of > UGI.doAs() when the test user could simply be passed through. Eliminate the > unnecessary use of doAs(). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22144) MultiRowRangeFilter does not work with reversed scans
[ https://issues.apache.org/jira/browse/HBASE-22144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-22144: Component/s: scan > MultiRowRangeFilter does not work with reversed scans > - > > Key: HBASE-22144 > URL: https://issues.apache.org/jira/browse/HBASE-22144 > Project: HBase > Issue Type: Bug > Components: Filters, scan >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.3.0, 2.0.6, 2.1.5 > > Attachments: HBASE-22144.001.patch, HBASE-22144.002.patch, > HBASE-22144.002.patch > > > It appears that MultiRowRangeFilter was never written to function with > reverse scans. There is too much logic that operates with the assumption that > we are always moving "forward" through increasing ranges. It needs to be > rewritten to "traverse" forward or backward, given how the context of the > scan being used. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22144) MultiRowRangeFilter does not work with reversed scans
[ https://issues.apache.org/jira/browse/HBASE-22144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-22144: Priority: Critical (was: Major) > MultiRowRangeFilter does not work with reversed scans > - > > Key: HBASE-22144 > URL: https://issues.apache.org/jira/browse/HBASE-22144 > Project: HBase > Issue Type: Bug > Components: Filters >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.3.0, 2.0.6, 2.1.5 > > Attachments: HBASE-22144.001.patch, HBASE-22144.002.patch, > HBASE-22144.002.patch > > > It appears that MultiRowRangeFilter was never written to function with > reverse scans. There is too much logic that operates with the assumption that > we are always moving "forward" through increasing ranges. It needs to be > rewritten to "traverse" forward or backward, given how the context of the > scan being used. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22217) HBase shell command proposal : "rit assign all"
[ https://issues.apache.org/jira/browse/HBASE-22217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-22217: Component/s: shell Region Assignment Operability > HBase shell command proposal : "rit assign all" > > > Key: HBASE-22217 > URL: https://issues.apache.org/jira/browse/HBASE-22217 > Project: HBase > Issue Type: New Feature > Components: Operability, Region Assignment, shell >Reporter: Xu Cang >Priority: Minor > > HBase shell command proposal : "rit assign all" > > Currently we have shell command "rit" to list all RITs. > It would be handy having a command "rit assign all" to assign all RITs. > This equals to getting the list of RITs from 'rit' command and running > "assign " one by one. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22217) HBase shell command proposal : "rit assign all"
[ https://issues.apache.org/jira/browse/HBASE-22217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815858#comment-16815858 ] Sean Busbey commented on HBASE-22217: - we'll need to be clear in the docs here and on hbck2 what the difference is in the two cases of using the "assign" command on a RIT. > HBase shell command proposal : "rit assign all" > > > Key: HBASE-22217 > URL: https://issues.apache.org/jira/browse/HBASE-22217 > Project: HBase > Issue Type: New Feature >Reporter: Xu Cang >Priority: Minor > > HBase shell command proposal : "rit assign all" > > Currently we have shell command "rit" to list all RITs. > It would be handy having a command "rit assign all" to assign all RITs. > This equals to getting the list of RITs from 'rit' command and running > "assign " one by one. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-22217) HBase shell command proposal : "rit assign all"
Xu Cang created HBASE-22217: --- Summary: HBase shell command proposal : "rit assign all" Key: HBASE-22217 URL: https://issues.apache.org/jira/browse/HBASE-22217 Project: HBase Issue Type: New Feature Reporter: Xu Cang HBase shell command proposal : "rit assign all" Currently we have shell command "rit" to list all RITs. It would be handy having a command "rit assign all" to assign all RITs. This equals to getting the list of RITs from 'rit' command and running "assign " one by one. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-22216) "Waiting on master failover to complete" shows 30 to 40 time per millisecond
Xu Cang created HBASE-22216: --- Summary: "Waiting on master failover to complete" shows 30 to 40 time per millisecond Key: HBASE-22216 URL: https://issues.apache.org/jira/browse/HBASE-22216 Project: HBase Issue Type: Bug Components: proc-v2 Affects Versions: 1.3.0 Reporter: Xu Cang "Waiting on master failover to complete" shows 30 to 40 time per millisecond from one host when master is initializing. This message is too noisy. Need to fix this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-7297) Allow load balancer to accommodate different region server configurations
[
https://issues.apache.org/jira/browse/HBASE-7297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jean-Marc Spaggiari resolved HBASE-7297.
Resolution: Duplicate
Duplicate of HBASE-11780.
> Allow load balancer to accommodate different region server configurations
> -
>
> Key: HBASE-7297
> URL: https://issues.apache.org/jira/browse/HBASE-7297
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Reporter: Ted Yu
>Priority: Major
>
> Robert Dyer raised the following scenario under the thread of 'Multiple
> regionservers on a single node':
> {quote}
> I have a very small cluster where all nodes are identical. However, I was
> just given a very powerful node to add into this cluster which effectively
> doubles the total CPUs, RAM, and HDDs in the cluster.
> As such, when I run a MR job half the jobs go to this single, new node yet
> most of the data is not local due to HBase balancing the regions.
> {quote}
> Load balancer should take region server config (total heap in the above case)
> into account when allocating regions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22194) Snapshot unittests fail on Windows due to invalid file path uri
[
https://issues.apache.org/jira/browse/HBASE-22194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815851#comment-16815851
]
Hudson commented on HBASE-22194:
Results for branch branch-2
[build #1814 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1814/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1814//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1814//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1814//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Snapshot unittests fail on Windows due to invalid file path uri
> ---
>
> Key: HBASE-22194
> URL: https://issues.apache.org/jira/browse/HBASE-22194
> Project: HBase
> Issue Type: Bug
> Components: regionserver, test
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Bahram Chehrazy
>Assignee: Bahram Chehrazy
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.3.0
>
> Attachments: unittest-fix-for-windows.patch
>
>
> These unittests are failing on Windows because the temporary snapshot file
> path is not valid.
> hadoop.hbase.client.TestSnapshotTemporaryDirectory
> hadoop.hbase.client.TestSnapshotDFSTemporaryDirectory
> hadoop.hbase.client.TestSnapshotTemporaryDirectoryWithRegionReplica
>
> The error is:
>
> 2019-04-08 23:42:02,080 ERROR [master/MININT-2D9TFVB:0:becomeActiveMaster]
> helpers.MarkerIgnoringBase(159): * ABORTING master
> minint-2d9tfvb.northamerica.corp.microsoft.com,57169,1554792118500: Unhandled
> exception. Starting shutdown. *
> java.lang.IllegalArgumentException: *Wrong FS:
> file://C:\src\hbase\hbase-server/2f6562be-fe12-49a4-b370-2b6928e5aa72,
> expected: file:///*
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:647)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:82)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:606)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:824)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:601)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.delete(ChecksumFileSystem.java:638)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.resetTempDir(SnapshotManager.java:298)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.initialize(SnapshotManager.java:1186)
> at
> org.apache.hadoop.hbase.procedure.MasterProcedureManagerHost.initialize(MasterProcedureManagerHost.java:50)
> at
> org.apache.hadoop.hbase.master.HMaster.initializeZKBasedSystemTrackers(HMaster.java:828)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1004)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2373)
> at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:603)
> at java.lang.Thread.run(Thread.java:748)
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-15560) TinyLFU-based BlockCache
[ https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815849#comment-16815849 ] Andrew Purtell commented on HBASE-15560: [~zyork] Sorry, that was my misunderstanding. I agree, if TinyLFU were to be made the default each change in these tests will need to be individually evaluated for missing coverage. Even if TinyLFU is not made default we can still do the parameterized testing as a follow up. > TinyLFU-based BlockCache > > > Key: HBASE-15560 > URL: https://issues.apache.org/jira/browse/HBASE-15560 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Affects Versions: 2.0.0 >Reporter: Ben Manes >Assignee: Ben Manes >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, bc.hit.count, bc.miss.count, branch-1.tinylfu.txt, gets, > run_ycsb_c.sh, run_ycsb_loading.sh, tinylfu.patch > > > LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and > recency of the working set. It achieves concurrency by using an O( n ) > background thread to prioritize the entries and evict. Accessing an entry is > O(1) by a hash table lookup, recording its logical access time, and setting a > frequency flag. A write is performed in O(1) time by updating the hash table > and triggering an async eviction thread. This provides ideal concurrency and > minimizes the latencies by penalizing the thread instead of the caller. > However the policy does not age the frequencies and may not be resilient to > various workload patterns. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > Concurrency is achieved by buffering and replaying the operations, similar to > a write-ahead log. A read is recorded into a striped ring buffer and writes > to a queue. The operations are applied in batches under a try-lock by an > asynchronous thread, thereby track the usage pattern without incurring high > latencies > ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]). > In YCSB benchmarks the results were inconclusive. For a large cache (99% hit > rates) the two caches have near identical throughput and latencies with > LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a > 1-4% hit rate improvement and therefore lower latencies. The lack luster > result is because a synthetic Zipfian distribution is used, which SLRU > performs optimally. In a more varied, real-world workload we'd expect to see > improvements by being able to make smarter predictions. > The provided patch implements BlockCache using the > [Caffeine|https://github.com/ben-manes/caffeine] caching library (see > HighScalability > [article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]). > Edward Bortnikov and Eshcar Hillel have graciously provided guidance for > evaluating this patch ([github > branch|https://github.com/ben-manes/hbase/tree/tinylfu]). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-3268) Run balancer when a new node is added
[ https://issues.apache.org/jira/browse/HBASE-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-3268. --- Resolution: Incomplete > Run balancer when a new node is added > - > > Key: HBASE-3268 > URL: https://issues.apache.org/jira/browse/HBASE-3268 > Project: HBase > Issue Type: Improvement > Components: Balancer, master >Reporter: Todd Lipcon >Priority: Major > > Right now we only balance the cluster once every 5 minutes by default. This > is likely to confuse new users. When you start a new region server, you > expect it to pick up some load very quickly, but right now you have to wait 5 > minutes for it to start doing anything in the worst case. > We could/should also add a button/shell command to "trigger balance now" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22212) [1.x] Backport missing filter improvements
[ https://issues.apache.org/jira/browse/HBASE-22212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815847#comment-16815847 ] Andrew Purtell commented on HBASE-22212: Skimmed the patch, lgtm, provided unit tests all pass. See my note about 1.4.10 RC next week? Please consider getting it in on time. I'll hold things up a day or two if you need it. > [1.x] Backport missing filter improvements > -- > > Key: HBASE-22212 > URL: https://issues.apache.org/jira/browse/HBASE-22212 > Project: HBase > Issue Type: Bug > Components: Filters >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 1.5.0, 1.4.10 > > Attachments: HBASE-22212.001.branch-1.patch > > > HBASE-19008 and HBASE-21129 were never backported beyond branch-2. I can't > find any reason that this was not done. Despite these being public-tagged > classes, no incompatible changes were added. > The lack of these changes prevents HBASE-22144 from being backported cleanly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22212) [1.x] Backport missing filter improvements
[ https://issues.apache.org/jira/browse/HBASE-22212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815842#comment-16815842 ] Josh Elser commented on HBASE-22212: .001 is a squash of 19008 and 21129 for QA to chug on. > [1.x] Backport missing filter improvements > -- > > Key: HBASE-22212 > URL: https://issues.apache.org/jira/browse/HBASE-22212 > Project: HBase > Issue Type: Bug > Components: Filters >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 1.5.0, 1.4.10 > > Attachments: HBASE-22212.001.branch-1.patch > > > HBASE-19008 and HBASE-21129 were never backported beyond branch-2. I can't > find any reason that this was not done. Despite these being public-tagged > classes, no incompatible changes were added. > The lack of these changes prevents HBASE-22144 from being backported cleanly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22212) [1.x] Backport missing filter improvements
[ https://issues.apache.org/jira/browse/HBASE-22212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-22212: --- Status: Patch Available (was: Open) > [1.x] Backport missing filter improvements > -- > > Key: HBASE-22212 > URL: https://issues.apache.org/jira/browse/HBASE-22212 > Project: HBase > Issue Type: Bug > Components: Filters >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 1.5.0, 1.4.10 > > Attachments: HBASE-22212.001.branch-1.patch > > > HBASE-19008 and HBASE-21129 were never backported beyond branch-2. I can't > find any reason that this was not done. Despite these being public-tagged > classes, no incompatible changes were added. > The lack of these changes prevents HBASE-22144 from being backported cleanly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22212) [1.x] Backport missing filter improvements
[ https://issues.apache.org/jira/browse/HBASE-22212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-22212: --- Attachment: HBASE-22212.001.branch-1.patch > [1.x] Backport missing filter improvements > -- > > Key: HBASE-22212 > URL: https://issues.apache.org/jira/browse/HBASE-22212 > Project: HBase > Issue Type: Bug > Components: Filters >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 1.5.0, 1.4.10 > > Attachments: HBASE-22212.001.branch-1.patch > > > HBASE-19008 and HBASE-21129 were never backported beyond branch-2. I can't > find any reason that this was not done. Despite these being public-tagged > classes, no incompatible changes were added. > The lack of these changes prevents HBASE-22144 from being backported cleanly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-22215) Backport MultiRowRangeFilter does not work with reverse scans
Josh Elser created HBASE-22215: -- Summary: Backport MultiRowRangeFilter does not work with reverse scans Key: HBASE-22215 URL: https://issues.apache.org/jira/browse/HBASE-22215 Project: HBase Issue Type: Sub-task Components: Filters Reporter: Josh Elser Assignee: Josh Elser Fix For: 1.5.0, 1.4.10 See parent. Modify and apply to 1.x lines. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22144) MultiRowRangeFilter does not work with reversed scans
[ https://issues.apache.org/jira/browse/HBASE-22144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-22144: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > MultiRowRangeFilter does not work with reversed scans > - > > Key: HBASE-22144 > URL: https://issues.apache.org/jira/browse/HBASE-22144 > Project: HBase > Issue Type: Bug > Components: Filters >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.3.0, 2.0.6, 2.1.5 > > Attachments: HBASE-22144.001.patch, HBASE-22144.002.patch, > HBASE-22144.002.patch > > > It appears that MultiRowRangeFilter was never written to function with > reverse scans. There is too much logic that operates with the assumption that > we are always moving "forward" through increasing ranges. It needs to be > rewritten to "traverse" forward or backward, given how the context of the > scan being used. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-22214) [2.x] Backport missing filter improvements
[ https://issues.apache.org/jira/browse/HBASE-22214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser resolved HBASE-22214. Resolution: Fixed > [2.x] Backport missing filter improvements > -- > > Key: HBASE-22214 > URL: https://issues.apache.org/jira/browse/HBASE-22214 > Project: HBase > Issue Type: Bug > Components: Filters >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 2.0.6, 2.1.5 > > > HBASE-19008 and HBASE-21129 were never backported beyond branch-2. I can't > find any reason that this was not done. Despite these being public-tagged > classes, no incompatible changes were added. > The lack of these changes prevents HBASE-22144 from being backported cleanly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22214) [2.x] Backport missing filter improvements
[ https://issues.apache.org/jira/browse/HBASE-22214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815839#comment-16815839 ] Josh Elser commented on HBASE-22214: These came back cleanly so I'm not re-running through QA. > [2.x] Backport missing filter improvements > -- > > Key: HBASE-22214 > URL: https://issues.apache.org/jira/browse/HBASE-22214 > Project: HBase > Issue Type: Bug > Components: Filters >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 2.0.6, 2.1.5 > > > HBASE-19008 and HBASE-21129 were never backported beyond branch-2. I can't > find any reason that this was not done. Despite these being public-tagged > classes, no incompatible changes were added. > The lack of these changes prevents HBASE-22144 from being backported cleanly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-3268) Run balancer when a new node is added
[ https://issues.apache.org/jira/browse/HBASE-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815822#comment-16815822 ] Biju Nair commented on HBASE-3268: -- Even though the delay in regions getting assigned to a new RS can cause confusion to a new user, it takes care of outliers like the one mentioned in earlier comment like adding multiple new RSes . Do we still need this feature or can it be closed? > Run balancer when a new node is added > - > > Key: HBASE-3268 > URL: https://issues.apache.org/jira/browse/HBASE-3268 > Project: HBase > Issue Type: Improvement > Components: Balancer, master >Reporter: Todd Lipcon >Priority: Major > > Right now we only balance the cluster once every 5 minutes by default. This > is likely to confuse new users. When you start a new region server, you > expect it to pick up some load very quickly, but right now you have to wait 5 > minutes for it to start doing anything in the worst case. > We could/should also add a button/shell command to "trigger balance now" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22057) Impose upper-bound on size of ZK ops sent in a single multi()
[
https://issues.apache.org/jira/browse/HBASE-22057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815821#comment-16815821
]
Lars Hofhansl commented on HBASE-22057:
---
As long as "real" multi calls are still guaranteed to be processed as such (for
example moving replication queues). That seems to be case.
> Impose upper-bound on size of ZK ops sent in a single multi()
> -
>
> Key: HBASE-22057
> URL: https://issues.apache.org/jira/browse/HBASE-22057
> Project: HBase
> Issue Type: Bug
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-22057-branch-1.patch, HBASE-22057.001.patch,
> HBASE-22057.002.patch, HBASE-22057.003.patch, HBASE-22057.004.patch
>
>
> In {{ZKUtil#multiOrSequential}}, we accept a list of {{ZKUtilOp}}'s to pass
> down to the {{ZooKeeper#multi(Iterable)}} method.
> One problem with this approach is that we may generate a large list of ZNodes
> to mutate in one batch which exceeds the allowable client package length,
> specified by {{jute.maxbuffer}}.
> This problem can manifest when we have a large number of WALs to replicate,
> queued in ZooKeeper, from a disabled peer. When that peer is dropped, the RS
> would submit deletes of those queued WALs. The RS will see ConnectionLoss for
> the resulting {{multi()}} calls it tries to make, because we are sending too
> large of a client message (because we're trying to delete too many WALs at
> once). The result (at least in branch-1 ish versions) is that the RS aborts
> after exceeding the ZK retries (as this operation will never succeed).
> A simple fix would be to impose a maximum number of Ops to run in a single
> batch inside ZKUtil, and split apart the caller-submitted batch into smaller
> chunks. Before we make such a change, I do need to make sure that we don't
> have any expectations on atomicity of the operations. I'm not sure what ZK
> provides here -- for the above example, splitting up batches of deletes is
> not an issue, but there could be issues with batches of creates where we only
> apply some.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-7297) Allow load balancer to accommodate different region server configurations
[
https://issues.apache.org/jira/browse/HBASE-7297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815820#comment-16815820
]
Biju Nair commented on HBASE-7297:
--
Can we close this as duplicate in favor of HBASE-11780?
> Allow load balancer to accommodate different region server configurations
> -
>
> Key: HBASE-7297
> URL: https://issues.apache.org/jira/browse/HBASE-7297
> Project: HBase
> Issue Type: New Feature
> Components: Balancer
>Reporter: Ted Yu
>Priority: Major
>
> Robert Dyer raised the following scenario under the thread of 'Multiple
> regionservers on a single node':
> {quote}
> I have a very small cluster where all nodes are identical. However, I was
> just given a very powerful node to add into this cluster which effectively
> doubles the total CPUs, RAM, and HDDs in the cluster.
> As such, when I run a MR job half the jobs go to this single, new node yet
> most of the data is not local due to HBase balancing the regions.
> {quote}
> Load balancer should take region server config (total heap in the above case)
> into account when allocating regions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-22149) HBOSS: A FileSystem implementation to provide HBase's required semantics
[
https://issues.apache.org/jira/browse/HBASE-22149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815804#comment-16815804
]
Vladimir Rodionov commented on HBASE-22149:
---
{quote}
I'd love to explore it but I think it's worth seeing if the ~minute renames are
really a problem first.
{quote}
This will harm r/w latency distribution tail for sure. Copying (moving) GBs of
data in S3 with 3x replication - that is 10s of seconds.
> HBOSS: A FileSystem implementation to provide HBase's required semantics
>
>
> Key: HBASE-22149
> URL: https://issues.apache.org/jira/browse/HBASE-22149
> Project: HBase
> Issue Type: New Feature
> Components: Filesystem Integration
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
>Priority: Critical
> Attachments: HBASE-22149-hadoop.patch, HBASE-22149-hbase-2.patch,
> HBASE-22149-hbase-3.patch, HBASE-22149-hbase.patch
>
>
> (Have been using the name HBOSS for HBase / Object Store Semantics)
> I've had some thoughts about how to solve the problem of running HBase on
> object stores. There has been some thought in the past about adding the
> required semantics to S3Guard, but I have some concerns about that. First,
> it's mixing complicated solutions to different problems (bridging the gap
> between a flat namespace and a hierarchical namespace vs. solving
> inconsistency). Second, it's S3-specific, whereas other objects stores could
> use virtually identical solutions. And third, we can't do things like atomic
> renames in a true sense. There would have to be some trade-offs specific to
> HBase's needs and it's better if we can solve that in an HBase-specific
> module without mixing all that logic in with the rest of S3A.
> Ideas to solve this above the FileSystem layer have been proposed and
> considered (HBASE-20431, for one), and maybe that's the right way forward
> long-term, but it certainly seems to be a hard problem and hasn't been done
> yet. But I don't know enough of all the internal considerations to make much
> of a judgment on that myself.
> I propose a FileSystem implementation that wraps another FileSystem instance
> and provides locking of FileSystem operations to ensure correct semantics.
> Locking could quite possibly be done on the same ZooKeeper ensemble as an
> HBase cluster already uses (I'm sure there are some performance
> considerations here that deserve more attention). I've put together a
> proof-of-concept on which I've tested some aspects of atomic renames and
> atomic file creates. Both of these tests fail reliably on a naked s3a
> instance. I've also done a small YCSB run against a small cluster to sanity
> check other functionality and was successful. I will post the patch, and my
> laundry list of things that still need work. The WAL is still placed on HDFS,
> but the HBase root directory is otherwise on S3.
> Note that my prototype is built on Hadoop's source tree right now. That's
> purely for my convenience in putting it together quickly, as that's where I
> mostly work. I actually think long-term, if this is accepted as a good
> solution, it makes sense to live in HBase (or it's own repository). It only
> depends on stable, public APIs in Hadoop and is targeted entirely at HBase's
> needs, so it should be able to iterate on the HBase community's terms alone.
> Another idea [~ste...@apache.org] proposed to me is that of an inode-based
> FileSystem that keeps hierarchical metadata in a more appropriate store that
> would allow the required transactions (maybe a special table in HBase could
> provide that store itself for other tables), and stores the underlying files
> with unique identifiers on S3. This allows renames to actually become fast
> instead of just large atomic operations. It does however place a strong
> dependency on the metadata store. I have not explored this idea much. My
> current proof-of-concept has been pleasantly simple, so I think it's the
> right solution unless it proves unable to provide the required performance
> characteristics.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-22144) MultiRowRangeFilter does not work with reversed scans
[ https://issues.apache.org/jira/browse/HBASE-22144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-22144: --- Fix Version/s: 2.1.5 2.0.6 2.3.0 2.2.0 3.0.0 > MultiRowRangeFilter does not work with reversed scans > - > > Key: HBASE-22144 > URL: https://issues.apache.org/jira/browse/HBASE-22144 > Project: HBase > Issue Type: Bug > Components: Filters >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.3.0, 2.0.6, 2.1.5 > > Attachments: HBASE-22144.001.patch, HBASE-22144.002.patch, > HBASE-22144.002.patch > > > It appears that MultiRowRangeFilter was never written to function with > reverse scans. There is too much logic that operates with the assumption that > we are always moving "forward" through increasing ranges. It needs to be > rewritten to "traverse" forward or backward, given how the context of the > scan being used. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22201) Multiple test case failures
[
https://issues.apache.org/jira/browse/HBASE-22201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian Buss updated HBASE-22201:
-
Description:
result-test.cc fails on EstimatedSize assertion:
The test assumes {{sizeof(Result)}} will be the same as the estimated size of
an empty Result which is not the case because estimated size also accounts for
current capacity of its {{row_}} string member.
When using docker::
* Classpath is not propagated properly within the docker container:
** if you follow the documentation and build the java project before starting
docker the classpath file contains the wrong paths. also the code to read from
a CLASSPATH env var doesn't quite work as expected
* MiniCluster is not writing an hbase-site.xml to disk as expected by some of
the tests—as a result the default zk port of 2181 is being used by clients
was:The test assumes {{sizeof(Result)}} will be the same as the estimated
size of an empty Result which is not the case because estimated size also
accounts for current capacity of its {{row_}} string member.
> Multiple test case failures
> ---
>
> Key: HBASE-22201
> URL: https://issues.apache.org/jira/browse/HBASE-22201
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: HBASE-14850
>Reporter: Ian Buss
>Assignee: Ian Buss
>Priority: Major
> Attachments: HBASE-22201-HBASE-14850.v0.patch
>
>
> result-test.cc fails on EstimatedSize assertion:
> The test assumes {{sizeof(Result)}} will be the same as the estimated size of
> an empty Result which is not the case because estimated size also accounts
> for current capacity of its {{row_}} string member.
> When using docker::
> * Classpath is not propagated properly within the docker container:
> ** if you follow the documentation and build the java project before
> starting docker the classpath file contains the wrong paths. also the code to
> read from a CLASSPATH env var doesn't quite work as expected
> * MiniCluster is not writing an hbase-site.xml to disk as expected by some
> of the tests—as a result the default zk port of 2181 is being used by clients
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-22201) result-test.cc fails on EstimatedSize assertion
[
https://issues.apache.org/jira/browse/HBASE-22201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian Buss updated HBASE-22201:
-
Status: Open (was: Patch Available)
Have discovered a number of problems with the test cases so will generalize
this a little more.
> result-test.cc fails on EstimatedSize assertion
> ---
>
> Key: HBASE-22201
> URL: https://issues.apache.org/jira/browse/HBASE-22201
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: HBASE-14850
>Reporter: Ian Buss
>Assignee: Ian Buss
>Priority: Major
> Attachments: HBASE-22201-HBASE-14850.v0.patch
>
>
> The test assumes {{sizeof(Result)}} will be the same as the estimated size of
> an empty Result which is not the case because estimated size also accounts
> for current capacity of its {{row_}} string member.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-22201) Multiple test case failures
[
https://issues.apache.org/jira/browse/HBASE-22201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ian Buss updated HBASE-22201:
-
Summary: Multiple test case failures (was: result-test.cc fails on
EstimatedSize assertion)
> Multiple test case failures
> ---
>
> Key: HBASE-22201
> URL: https://issues.apache.org/jira/browse/HBASE-22201
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: HBASE-14850
>Reporter: Ian Buss
>Assignee: Ian Buss
>Priority: Major
> Attachments: HBASE-22201-HBASE-14850.v0.patch
>
>
> The test assumes {{sizeof(Result)}} will be the same as the estimated size of
> an empty Result which is not the case because estimated size also accounts
> for current capacity of its {{row_}} string member.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-15560) TinyLFU-based BlockCache
[ https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815799#comment-16815799 ] Zach York commented on HBASE-15560: --- I don't think the changes required to make it work with branch-1 should be forward ported unless it captures how we would like users to be able to use the TinyLFU implementation (which I don't think it does). Andrew, this was my take on the testing as well. It is sufficient for now, but will likely need to be enhanced when we would consider making TinyLFU the default. Sorry if my wording made it sound like there were current functionality breaks, more like we can't be sure there wouldn't be regressions with the current testing in place. Definitely something that can happen after this gets in though as I've already stated. > TinyLFU-based BlockCache > > > Key: HBASE-15560 > URL: https://issues.apache.org/jira/browse/HBASE-15560 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Affects Versions: 2.0.0 >Reporter: Ben Manes >Assignee: Ben Manes >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, bc.hit.count, bc.miss.count, branch-1.tinylfu.txt, gets, > run_ycsb_c.sh, run_ycsb_loading.sh, tinylfu.patch > > > LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and > recency of the working set. It achieves concurrency by using an O( n ) > background thread to prioritize the entries and evict. Accessing an entry is > O(1) by a hash table lookup, recording its logical access time, and setting a > frequency flag. A write is performed in O(1) time by updating the hash table > and triggering an async eviction thread. This provides ideal concurrency and > minimizes the latencies by penalizing the thread instead of the caller. > However the policy does not age the frequencies and may not be resilient to > various workload patterns. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > Concurrency is achieved by buffering and replaying the operations, similar to > a write-ahead log. A read is recorded into a striped ring buffer and writes > to a queue. The operations are applied in batches under a try-lock by an > asynchronous thread, thereby track the usage pattern without incurring high > latencies > ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]). > In YCSB benchmarks the results were inconclusive. For a large cache (99% hit > rates) the two caches have near identical throughput and latencies with > LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a > 1-4% hit rate improvement and therefore lower latencies. The lack luster > result is because a synthetic Zipfian distribution is used, which SLRU > performs optimally. In a more varied, real-world workload we'd expect to see > improvements by being able to make smarter predictions. > The provided patch implements BlockCache using the > [Caffeine|https://github.com/ben-manes/caffeine] caching library (see > HighScalability > [article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]). > Edward Bortnikov and Eshcar Hillel have graciously provided guidance for > evaluating this patch ([github > branch|https://github.com/ben-manes/hbase/tree/tinylfu]). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22212) [1.x] Backport missing filter improvements
[ https://issues.apache.org/jira/browse/HBASE-22212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-22212: --- Summary: [1.x] Backport missing filter improvements (was: Backport missing filter improvements) > [1.x] Backport missing filter improvements > -- > > Key: HBASE-22212 > URL: https://issues.apache.org/jira/browse/HBASE-22212 > Project: HBase > Issue Type: Bug > Components: Filters >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 1.5.0, 1.4.10, 2.0.6, 2.1.5 > > > HBASE-19008 and HBASE-21129 were never backported beyond branch-2. I can't > find any reason that this was not done. Despite these being public-tagged > classes, no incompatible changes were added. > The lack of these changes prevents HBASE-22144 from being backported cleanly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22212) [1.x] Backport missing filter improvements
[ https://issues.apache.org/jira/browse/HBASE-22212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-22212: --- Fix Version/s: (was: 2.1.5) (was: 2.0.6) > [1.x] Backport missing filter improvements > -- > > Key: HBASE-22212 > URL: https://issues.apache.org/jira/browse/HBASE-22212 > Project: HBase > Issue Type: Bug > Components: Filters >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 1.5.0, 1.4.10 > > > HBASE-19008 and HBASE-21129 were never backported beyond branch-2. I can't > find any reason that this was not done. Despite these being public-tagged > classes, no incompatible changes were added. > The lack of these changes prevents HBASE-22144 from being backported cleanly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-22214) [2.x] Backport missing filter improvements
Josh Elser created HBASE-22214: -- Summary: [2.x] Backport missing filter improvements Key: HBASE-22214 URL: https://issues.apache.org/jira/browse/HBASE-22214 Project: HBase Issue Type: Bug Components: Filters Reporter: Josh Elser Assignee: Josh Elser Fix For: 1.5.0, 1.4.10, 2.0.6, 2.1.5 HBASE-19008 and HBASE-21129 were never backported beyond branch-2. I can't find any reason that this was not done. Despite these being public-tagged classes, no incompatible changes were added. The lack of these changes prevents HBASE-22144 from being backported cleanly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-22214) [2.x] Backport missing filter improvements
[ https://issues.apache.org/jira/browse/HBASE-22214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-22214: --- Fix Version/s: (was: 1.4.10) (was: 1.5.0) > [2.x] Backport missing filter improvements > -- > > Key: HBASE-22214 > URL: https://issues.apache.org/jira/browse/HBASE-22214 > Project: HBase > Issue Type: Bug > Components: Filters >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 2.0.6, 2.1.5 > > > HBASE-19008 and HBASE-21129 were never backported beyond branch-2. I can't > find any reason that this was not done. Despite these being public-tagged > classes, no incompatible changes were added. > The lack of these changes prevents HBASE-22144 from being backported cleanly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-22212) Backport missing filter improvements
[ https://issues.apache.org/jira/browse/HBASE-22212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815796#comment-16815796 ] Josh Elser commented on HBASE-22212: Yes sir. Thanks for the info. I need to break this out for 1.x and 2.x stuff. 2.x is clean, but 1.x I had to do some fixing on and I want to get at QA run for it. > Backport missing filter improvements > > > Key: HBASE-22212 > URL: https://issues.apache.org/jira/browse/HBASE-22212 > Project: HBase > Issue Type: Bug > Components: Filters >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 1.5.0, 1.4.10, 2.0.6, 2.1.5 > > > HBASE-19008 and HBASE-21129 were never backported beyond branch-2. I can't > find any reason that this was not done. Despite these being public-tagged > classes, no incompatible changes were added. > The lack of these changes prevents HBASE-22144 from being backported cleanly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-15560) TinyLFU-based BlockCache
[
https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815785#comment-16815785
]
Andrew Purtell edited comment on HBASE-15560 at 4/11/19 9:31 PM:
-
{quote}but the fact that tests were changed to only test LRUCache (or only
assert data is correct if LRUCache) was concerning to me. That indicates to me
that there is some missing functionality in the TinyLFU
{quote}
You can't make that assumption based on these changes.
What I did is change the default from "LRU" to "TinyLFU" and then execute the
unit tests. The default remains "LRU". An invocation of the unit test suite
will test "LRU". During the modified execution I found some places where the
tests expect LruBlockCache. That assumption is invalid when L1 is TinyLFU. So,
where that assumption is made I made it conditional on the actual class.
Nothing else has changed. This was done to exercise TinyLFU as much as possible
given the complete unit test suite we have available. This might imply TinyLFU
needs more test coverage, or it might not, it depends on the unit test. You
will need to make a closer examination.
As to the rest of your points we are in agreement.
Edit: The TinyLFU contribution comes with its own unit test which is executed
unconditionally no matter the L1 setting. This is not a comprehensive suite of
all blockcache functionality, though. It could be a useful follow up item to
refactor all blockcache tests to make them parameterized, and select "LRU" L1
for one invocation; then "TinyLFU" L1 for the second invocation. This follow up
can be readily done on branch-2 and up. On branch-1, because the compilation of
TinyLFU is conditional and a separate module it seems more complicated and
maybe difficult to do (some kind of conditional test if tinylfu module and jar
is in the test classpath maybe?).
was (Author: apurtell):
{quote}but the fact that tests were changed to only test LRUCache (or only
assert data is correct if LRUCache) was concerning to me. That indicates to me
that there is some missing functionality in the TinyLFU
{quote}
You can't make that assumption based on these changes.
What I did is change the default from "LRU" to "TinyLFU" and then execute the
unit tests. The default remains "LRU". An invocation of the unit test suite
will test "LRU". During the modified execution I found some places where the
tests expect LruBlockCache. That assumption is invalid when L1 is TinyLFU. So,
where that assumption is made I made it conditional on the actual class.
Nothing else has changed. This was done to exercise TinyLFU as much as possible
given the complete unit test suite we have available. This might imply TinyLFU
needs more test coverage, or it might not, it depends on the unit test. You
will need to make a closer examination.
As to the rest of your points we are in agreement.
> TinyLFU-based BlockCache
>
>
> Key: HBASE-15560
> URL: https://issues.apache.org/jira/browse/HBASE-15560
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache
>Affects Versions: 2.0.0
>Reporter: Ben Manes
>Assignee: Ben Manes
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch,
> HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch,
> HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch,
> HBASE-15560.patch, bc.hit.count, bc.miss.count, branch-1.tinylfu.txt, gets,
> run_ycsb_c.sh, run_ycsb_loading.sh, tinylfu.patch
>
>
> LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and
> recency of the working set. It achieves concurrency by using an O( n )
> background thread to prioritize the entries and evict. Accessing an entry is
> O(1) by a hash table lookup, recording its logical access time, and setting a
> frequency flag. A write is performed in O(1) time by updating the hash table
> and triggering an async eviction thread. This provides ideal concurrency and
> minimizes the latencies by penalizing the thread instead of the caller.
> However the policy does not age the frequencies and may not be resilient to
> various workload patterns.
> W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the
> frequency in a counting sketch, ages periodically by halving the counters,
> and orders entries by SLRU. An entry is discarded by comparing the frequency
> of the new arrival (candidate) to the SLRU's victim, and keeping the one with
> the highest frequency. This allows the operations to be performed in O(1)
> time and, though the use of a compact sketch, a much larger history is
> retained beyond the current working set. In a variety of real world traces
> the policy had
[jira] [Commented] (HBASE-22172) Suppress Java 11 reflective access warnings
[
https://issues.apache.org/jira/browse/HBASE-22172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815791#comment-16815791
]
Sakthi commented on HBASE-22172:
Thanks Josh for trying it out.
{quote}I feel like trying to get all of these warnings suppressed is chasing
your tail for little good :(
{quote}
I share this opinion with you. I feel the same. Let's stall this then?
> Suppress Java 11 reflective access warnings
> ---
>
> Key: HBASE-22172
> URL: https://issues.apache.org/jira/browse/HBASE-22172
> Project: HBase
> Issue Type: Task
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Major
> Labels: jdk11
> Attachments: hbase-22172.master.001.patch
>
>
> While running a Java 8 compiled hbase on Java 11 system, I found the
> following warnings being thrown. I think we can add the "--add-opens" flag to
> HBASE_OPTS (if the jdk version is 11) to suppress this warning.
> {code:java}
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> org.apache.hadoop.hbase.util.UnsafeAvailChecker
> (file:/Users/jatsakthi/test/HBASE_TEST_AREA/hbase-3.0.0-SNAPSHOT/lib/hbase-common-3.0.0-SNAPSHOT.jar)
> to method java.nio.Bits.unaligned()
> WARNING: Please consider reporting this to the maintainers of
> org.apache.hadoop.hbase.util.UnsafeAvailChecker
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-15560) TinyLFU-based BlockCache
[ https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815788#comment-16815788 ] Andrew Purtell edited comment on HBASE-15560 at 4/11/19 9:23 PM: - There is no need to break out TinyLFU to a separate module, right? Because branch-2 and up specifies a minimum JRE/JDK of Java 8. I.e. it was never the case that I understood the branch-1 specific changes should be forward ported as it were was (Author: apurtell): There is no need to break out TinyLFU to a separate module, right? Because branch-2 and up specifies a minimum JRE/JDK of Java 8. > TinyLFU-based BlockCache > > > Key: HBASE-15560 > URL: https://issues.apache.org/jira/browse/HBASE-15560 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Affects Versions: 2.0.0 >Reporter: Ben Manes >Assignee: Ben Manes >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, bc.hit.count, bc.miss.count, branch-1.tinylfu.txt, gets, > run_ycsb_c.sh, run_ycsb_loading.sh, tinylfu.patch > > > LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and > recency of the working set. It achieves concurrency by using an O( n ) > background thread to prioritize the entries and evict. Accessing an entry is > O(1) by a hash table lookup, recording its logical access time, and setting a > frequency flag. A write is performed in O(1) time by updating the hash table > and triggering an async eviction thread. This provides ideal concurrency and > minimizes the latencies by penalizing the thread instead of the caller. > However the policy does not age the frequencies and may not be resilient to > various workload patterns. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > Concurrency is achieved by buffering and replaying the operations, similar to > a write-ahead log. A read is recorded into a striped ring buffer and writes > to a queue. The operations are applied in batches under a try-lock by an > asynchronous thread, thereby track the usage pattern without incurring high > latencies > ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]). > In YCSB benchmarks the results were inconclusive. For a large cache (99% hit > rates) the two caches have near identical throughput and latencies with > LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a > 1-4% hit rate improvement and therefore lower latencies. The lack luster > result is because a synthetic Zipfian distribution is used, which SLRU > performs optimally. In a more varied, real-world workload we'd expect to see > improvements by being able to make smarter predictions. > The provided patch implements BlockCache using the > [Caffeine|https://github.com/ben-manes/caffeine] caching library (see > HighScalability > [article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]). > Edward Bortnikov and Eshcar Hillel have graciously provided guidance for > evaluating this patch ([github > branch|https://github.com/ben-manes/hbase/tree/tinylfu]). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-15560) TinyLFU-based BlockCache
[ https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815788#comment-16815788 ] Andrew Purtell commented on HBASE-15560: There is no need to break out TinyLFU to a separate module, right? Because branch-2 and up specifies a minimum JRE/JDK of Java 8. > TinyLFU-based BlockCache > > > Key: HBASE-15560 > URL: https://issues.apache.org/jira/browse/HBASE-15560 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Affects Versions: 2.0.0 >Reporter: Ben Manes >Assignee: Ben Manes >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, bc.hit.count, bc.miss.count, branch-1.tinylfu.txt, gets, > run_ycsb_c.sh, run_ycsb_loading.sh, tinylfu.patch > > > LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and > recency of the working set. It achieves concurrency by using an O( n ) > background thread to prioritize the entries and evict. Accessing an entry is > O(1) by a hash table lookup, recording its logical access time, and setting a > frequency flag. A write is performed in O(1) time by updating the hash table > and triggering an async eviction thread. This provides ideal concurrency and > minimizes the latencies by penalizing the thread instead of the caller. > However the policy does not age the frequencies and may not be resilient to > various workload patterns. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > Concurrency is achieved by buffering and replaying the operations, similar to > a write-ahead log. A read is recorded into a striped ring buffer and writes > to a queue. The operations are applied in batches under a try-lock by an > asynchronous thread, thereby track the usage pattern without incurring high > latencies > ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]). > In YCSB benchmarks the results were inconclusive. For a large cache (99% hit > rates) the two caches have near identical throughput and latencies with > LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a > 1-4% hit rate improvement and therefore lower latencies. The lack luster > result is because a synthetic Zipfian distribution is used, which SLRU > performs optimally. In a more varied, real-world workload we'd expect to see > improvements by being able to make smarter predictions. > The provided patch implements BlockCache using the > [Caffeine|https://github.com/ben-manes/caffeine] caching library (see > HighScalability > [article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]). > Edward Bortnikov and Eshcar Hillel have graciously provided guidance for > evaluating this patch ([github > branch|https://github.com/ben-manes/hbase/tree/tinylfu]). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-15560) TinyLFU-based BlockCache
[
https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815785#comment-16815785
]
Andrew Purtell edited comment on HBASE-15560 at 4/11/19 9:22 PM:
-
{quote}but the fact that tests were changed to only test LRUCache (or only
assert data is correct if LRUCache) was concerning to me. That indicates to me
that there is some missing functionality in the TinyLFU
{quote}
You can't make that assumption based on these changes.
What I did is change the default from "LRU" to "TinyLFU" and then execute the
unit tests. The default remains "LRU". An invocation of the unit test suite
will test "LRU". During the modified execution I found some places where the
tests expect LruBlockCache. That assumption is invalid when L1 is TinyLFU. So,
where that assumption is made I made it conditional on the actual class.
Nothing else has changed. This was done to exercise TinyLFU as much as possible
given the complete unit test suite we have available. This might imply TinyLFU
needs more test coverage, or it might not, it depends on the unit test. You
will need to make a closer examination.
As to the rest of your points we are in agreement.
was (Author: apurtell):
{quote}but the fact that tests were changed to only test LRUCache (or only
assert data is correct if LRUCache) was concerning to me. That indicates to me
that there is some missing functionality in the TinyLFU
{quote}
You can't make that assumption based on these changes.
What I did is change the default from "LRU" to "TinyLFU" and then executed the
unit tests. During that execution I found some places where the tests expect
LruBlockCache. Well, that assumption is now invalid. So, where that assumption
is made I made it conditional on the actual class. Nothing else has changed.
This might imply TinyLFU needs more test coverage, or it might not, it depends
on the unit test. You will need to make a closer examination.
As to the rest of your points we are in agreement.
> TinyLFU-based BlockCache
>
>
> Key: HBASE-15560
> URL: https://issues.apache.org/jira/browse/HBASE-15560
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache
>Affects Versions: 2.0.0
>Reporter: Ben Manes
>Assignee: Ben Manes
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch,
> HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch,
> HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch,
> HBASE-15560.patch, bc.hit.count, bc.miss.count, branch-1.tinylfu.txt, gets,
> run_ycsb_c.sh, run_ycsb_loading.sh, tinylfu.patch
>
>
> LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and
> recency of the working set. It achieves concurrency by using an O( n )
> background thread to prioritize the entries and evict. Accessing an entry is
> O(1) by a hash table lookup, recording its logical access time, and setting a
> frequency flag. A write is performed in O(1) time by updating the hash table
> and triggering an async eviction thread. This provides ideal concurrency and
> minimizes the latencies by penalizing the thread instead of the caller.
> However the policy does not age the frequencies and may not be resilient to
> various workload patterns.
> W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the
> frequency in a counting sketch, ages periodically by halving the counters,
> and orders entries by SLRU. An entry is discarded by comparing the frequency
> of the new arrival (candidate) to the SLRU's victim, and keeping the one with
> the highest frequency. This allows the operations to be performed in O(1)
> time and, though the use of a compact sketch, a much larger history is
> retained beyond the current working set. In a variety of real world traces
> the policy had [near optimal hit
> rates|https://github.com/ben-manes/caffeine/wiki/Efficiency].
> Concurrency is achieved by buffering and replaying the operations, similar to
> a write-ahead log. A read is recorded into a striped ring buffer and writes
> to a queue. The operations are applied in batches under a try-lock by an
> asynchronous thread, thereby track the usage pattern without incurring high
> latencies
> ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]).
> In YCSB benchmarks the results were inconclusive. For a large cache (99% hit
> rates) the two caches have near identical throughput and latencies with
> LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a
> 1-4% hit rate improvement and therefore lower latencies. The lack luster
> result is because a synthetic Zipfian distribution is used, which SLRU
[jira] [Commented] (HBASE-15560) TinyLFU-based BlockCache
[ https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815786#comment-16815786 ] Sean Busbey commented on HBASE-15560: - I had been waiting to review this because before you said any changes in structure needed for branch-1 would also be reflected in this patch. is that no longer the case? > TinyLFU-based BlockCache > > > Key: HBASE-15560 > URL: https://issues.apache.org/jira/browse/HBASE-15560 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Affects Versions: 2.0.0 >Reporter: Ben Manes >Assignee: Ben Manes >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, bc.hit.count, bc.miss.count, branch-1.tinylfu.txt, gets, > run_ycsb_c.sh, run_ycsb_loading.sh, tinylfu.patch > > > LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and > recency of the working set. It achieves concurrency by using an O( n ) > background thread to prioritize the entries and evict. Accessing an entry is > O(1) by a hash table lookup, recording its logical access time, and setting a > frequency flag. A write is performed in O(1) time by updating the hash table > and triggering an async eviction thread. This provides ideal concurrency and > minimizes the latencies by penalizing the thread instead of the caller. > However the policy does not age the frequencies and may not be resilient to > various workload patterns. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > Concurrency is achieved by buffering and replaying the operations, similar to > a write-ahead log. A read is recorded into a striped ring buffer and writes > to a queue. The operations are applied in batches under a try-lock by an > asynchronous thread, thereby track the usage pattern without incurring high > latencies > ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]). > In YCSB benchmarks the results were inconclusive. For a large cache (99% hit > rates) the two caches have near identical throughput and latencies with > LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a > 1-4% hit rate improvement and therefore lower latencies. The lack luster > result is because a synthetic Zipfian distribution is used, which SLRU > performs optimally. In a more varied, real-world workload we'd expect to see > improvements by being able to make smarter predictions. > The provided patch implements BlockCache using the > [Caffeine|https://github.com/ben-manes/caffeine] caching library (see > HighScalability > [article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]). > Edward Bortnikov and Eshcar Hillel have graciously provided guidance for > evaluating this patch ([github > branch|https://github.com/ben-manes/hbase/tree/tinylfu]). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-15560) TinyLFU-based BlockCache
[
https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815785#comment-16815785
]
Andrew Purtell commented on HBASE-15560:
{quote}but the fact that tests were changed to only test LRUCache (or only
assert data is correct if LRUCache) was concerning to me. That indicates to me
that there is some missing functionality in the TinyLFU
{quote}
You can't make that assumption based on these changes.
What I did is change the default from "LRU" to "TinyLFU" and then executed the
unit tests. During that execution I found some places where the tests expect
LruBlockCache. Well, that assumption is now invalid. So, where that assumption
is made I made it conditional on the actual class. Nothing else has changed.
This might imply TinyLFU needs more test coverage, or it might not, it depends
on the unit test. You will need to make a closer examination.
As to the rest of your points we are in agreement.
> TinyLFU-based BlockCache
>
>
> Key: HBASE-15560
> URL: https://issues.apache.org/jira/browse/HBASE-15560
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache
>Affects Versions: 2.0.0
>Reporter: Ben Manes
>Assignee: Ben Manes
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch,
> HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch,
> HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch,
> HBASE-15560.patch, bc.hit.count, bc.miss.count, branch-1.tinylfu.txt, gets,
> run_ycsb_c.sh, run_ycsb_loading.sh, tinylfu.patch
>
>
> LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and
> recency of the working set. It achieves concurrency by using an O( n )
> background thread to prioritize the entries and evict. Accessing an entry is
> O(1) by a hash table lookup, recording its logical access time, and setting a
> frequency flag. A write is performed in O(1) time by updating the hash table
> and triggering an async eviction thread. This provides ideal concurrency and
> minimizes the latencies by penalizing the thread instead of the caller.
> However the policy does not age the frequencies and may not be resilient to
> various workload patterns.
> W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the
> frequency in a counting sketch, ages periodically by halving the counters,
> and orders entries by SLRU. An entry is discarded by comparing the frequency
> of the new arrival (candidate) to the SLRU's victim, and keeping the one with
> the highest frequency. This allows the operations to be performed in O(1)
> time and, though the use of a compact sketch, a much larger history is
> retained beyond the current working set. In a variety of real world traces
> the policy had [near optimal hit
> rates|https://github.com/ben-manes/caffeine/wiki/Efficiency].
> Concurrency is achieved by buffering and replaying the operations, similar to
> a write-ahead log. A read is recorded into a striped ring buffer and writes
> to a queue. The operations are applied in batches under a try-lock by an
> asynchronous thread, thereby track the usage pattern without incurring high
> latencies
> ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]).
> In YCSB benchmarks the results were inconclusive. For a large cache (99% hit
> rates) the two caches have near identical throughput and latencies with
> LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a
> 1-4% hit rate improvement and therefore lower latencies. The lack luster
> result is because a synthetic Zipfian distribution is used, which SLRU
> performs optimally. In a more varied, real-world workload we'd expect to see
> improvements by being able to make smarter predictions.
> The provided patch implements BlockCache using the
> [Caffeine|https://github.com/ben-manes/caffeine] caching library (see
> HighScalability
> [article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]).
> Edward Bortnikov and Eshcar Hillel have graciously provided guidance for
> evaluating this patch ([github
> branch|https://github.com/ben-manes/hbase/tree/tinylfu]).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-22086) space quota issue: deleting snapshot doesn't update the usage of table
[ https://issues.apache.org/jira/browse/HBASE-22086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sakthi updated HBASE-22086: --- Attachment: hbase-22086.master.004.patch > space quota issue: deleting snapshot doesn't update the usage of table > -- > > Key: HBASE-22086 > URL: https://issues.apache.org/jira/browse/HBASE-22086 > Project: HBase > Issue Type: Bug >Reporter: Ajeet Rai >Assignee: Sakthi >Priority: Minor > Attachments: hbase-22086.master.001.patch, > hbase-22086.master.002.patch, hbase-22086.master.003.patch, > hbase-22086.master.004.patch > > > space quota issue: deleting snapshot doesn't update the usage of table > Steps: 1: > set_quota TYPE => SPACE, TABLE => 'bugatti', LIMIT => '7M', POLICY => > NO_WRITES_COMPACTIONS > 2: ./hbase pe --table="bugatti" --nomapred --rows=200 sequentialWrite 10 > 3: ./hbase pe --table="bugatti" --nomapred --rows=200 sequentialWrite 10 > 4: snapshot 'bugatti','bugatti_snapshot' > 5: ./hbase pe --table="bugatti" --nomapred --rows=200 sequentialWrite 10 > 6: major_compact 'bugatti' > 7: delete_snapshot 'bugatti_snapshot' > now check the usage and observe that it is not getting updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-15560) TinyLFU-based BlockCache
[ https://issues.apache.org/jira/browse/HBASE-15560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815779#comment-16815779 ] Zach York commented on HBASE-15560: --- [~apurtell] I am making claim that based on the state of the patch I looked at, not any functional/load testing. Perhaps it will be less than 'significant', but the fact that tests were changed to only test LRUCache (or only assert data is correct if LRUCache) was concerning to me. That indicates to me that there is some missing functionality in the TinyLFU (or a miss in the first level cache implementation) however... Looking again at the test changes that were done... it does indeed look like the tests themselves are at fault (depending on functionality that is questionable to expose), but I do think that the First level cache should include a clearCache() mechanism to ensure we are operating on an empty cache. Regardless, it seems like their is work around decided what is the functionality we expect from the First Level Cache and rewrite tests to only depend on that contract. Given your testing and another look at the actual reasons the tests were changed to only test LRU behavior, I am less hesitant. > TinyLFU-based BlockCache > > > Key: HBASE-15560 > URL: https://issues.apache.org/jira/browse/HBASE-15560 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Affects Versions: 2.0.0 >Reporter: Ben Manes >Assignee: Ben Manes >Priority: Major > Fix For: 3.0.0, 2.3.0 > > Attachments: HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, HBASE-15560.patch, > HBASE-15560.patch, bc.hit.count, bc.miss.count, branch-1.tinylfu.txt, gets, > run_ycsb_c.sh, run_ycsb_loading.sh, tinylfu.patch > > > LruBlockCache uses the Segmented LRU (SLRU) policy to capture frequency and > recency of the working set. It achieves concurrency by using an O( n ) > background thread to prioritize the entries and evict. Accessing an entry is > O(1) by a hash table lookup, recording its logical access time, and setting a > frequency flag. A write is performed in O(1) time by updating the hash table > and triggering an async eviction thread. This provides ideal concurrency and > minimizes the latencies by penalizing the thread instead of the caller. > However the policy does not age the frequencies and may not be resilient to > various workload patterns. > W-TinyLFU ([research paper|http://arxiv.org/pdf/1512.00727.pdf]) records the > frequency in a counting sketch, ages periodically by halving the counters, > and orders entries by SLRU. An entry is discarded by comparing the frequency > of the new arrival (candidate) to the SLRU's victim, and keeping the one with > the highest frequency. This allows the operations to be performed in O(1) > time and, though the use of a compact sketch, a much larger history is > retained beyond the current working set. In a variety of real world traces > the policy had [near optimal hit > rates|https://github.com/ben-manes/caffeine/wiki/Efficiency]. > Concurrency is achieved by buffering and replaying the operations, similar to > a write-ahead log. A read is recorded into a striped ring buffer and writes > to a queue. The operations are applied in batches under a try-lock by an > asynchronous thread, thereby track the usage pattern without incurring high > latencies > ([benchmarks|https://github.com/ben-manes/caffeine/wiki/Benchmarks#server-class]). > In YCSB benchmarks the results were inconclusive. For a large cache (99% hit > rates) the two caches have near identical throughput and latencies with > LruBlockCache narrowly winning. At medium and small caches, TinyLFU had a > 1-4% hit rate improvement and therefore lower latencies. The lack luster > result is because a synthetic Zipfian distribution is used, which SLRU > performs optimally. In a more varied, real-world workload we'd expect to see > improvements by being able to make smarter predictions. > The provided patch implements BlockCache using the > [Caffeine|https://github.com/ben-manes/caffeine] caching library (see > HighScalability > [article|http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html]). > Edward Bortnikov and Eshcar Hillel have graciously provided guidance for > evaluating this patch ([github > branch|https://github.com/ben-manes/hbase/tree/tinylfu]). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
