[jira] [Commented] (HBASE-20628) SegmentScanner does over-comparing when one flushing
[
https://issues.apache.org/jira/browse/HBASE-20628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494693#comment-16494693
]
Anoop Sam John commented on HBASE-20628:
In fact I am having a patch for a custom scanner and SQM for flushes. I
totally forgot the details of that.. Need to revisit that.. IMHO, we can
commit this patch.
bq.replace the implementation of SegmentScanner::reseek(cell) with the code
from CollectionBackedScanner::reseek(cell) which does not create a new iterator
and simply runs next until reaching the cell.
I tried this once but not got perf improvement as with CollectionBackedScanner.
SegmentScanner::reseek - Lets work on that on another issue.
> SegmentScanner does over-comparing when one flushing
>
>
> Key: HBASE-20628
> URL: https://issues.apache.org/jira/browse/HBASE-20628
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: stack
>Priority: Critical
> Fix For: 2.0.1
>
> Attachments: HBASE-20628.branch-2.0.001 (1).patch,
> HBASE-20628.branch-2.0.001.patch, HBASE-20628.branch-2.0.001.patch,
> HBASE-20628.branch-2.0.002.patch, Screen Shot 2018-05-25 at 9.38.00 AM.png,
> hits-20628.png
>
>
> Flushing memstore is taking too long. It looks like we are doing a bunch of
> comparing out of a new facility in hbase2, the Segment scanner at flush time.
> Below is a patch from [~anoop.hbase]. I had a similar more hacky version.
> Both undo the extra comparing we were seeing in perf tests.
> [~anastas] and [~eshcar]. Need your help please.
> As I read it, we are trying to flush the memstore snapshot (default, no IMC
> case). There is only ever going to be one Segment involved (even if IMC is
> enabled); the snapshot Segment. But the getScanners is returning a list (of
> one) Scanners and the scan is via the generic SegmentScanner which is all
> about a bunch of stuff we don't need when doing a flush so it seems to do
> more work than is necessary. It also supports scanning backwards which is not
> needed when trying to flush memstore.
> Do you see a problem doing a version of Anoops patch (whether IMC or not)? It
> makes a big difference in general throughput when the below patch is in
> place. Thanks.
> {code}
> diff --git
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
>
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> index cbd60e5da3..c3dd972254 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> @@ -40,7 +40,8 @@ public class MemStoreSnapshot implements Closeable {
> this.cellsCount = snapshot.getCellsCount();
> this.memStoreSize = snapshot.getMemStoreSize();
> this.timeRangeTracker = snapshot.getTimeRangeTracker();
> -this.scanners = snapshot.getScanners(Long.MAX_VALUE, Long.MAX_VALUE);
> +//this.scanners = snapshot.getScanners(Long.MAX_VALUE, Long.MAX_VALUE);
> +this.scanners = snapshot.getScannersForSnapshot();
> this.tagsPresent = snapshot.isTagsPresent();
>}
> diff --git
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
>
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> index 70074bf3b4..279c4e50c8 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> @@ -33,6 +33,7 @@ import org.apache.hadoop.hbase.KeyValueUtil;
> import org.apache.hadoop.hbase.io.TimeRange;
> import org.apache.hadoop.hbase.util.Bytes;
> import org.apache.hadoop.hbase.util.ClassSize;
> +import org.apache.hadoop.hbase.util.CollectionBackedScanner;
> import org.apache.yetus.audience.InterfaceAudience;
> import org.slf4j.Logger;
> import
> org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
> @@ -130,6 +131,10 @@ public abstract class Segment {
> return Collections.singletonList(new SegmentScanner(this, readPoint,
> order));
>}
> + public List getScannersForSnapshot() {
> +return Collections.singletonList(new
> CollectionBackedScanner(this.cellSet.get(), comparator));
> + }
> +
>/**
> * @return whether the segment has any cells
> */
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20618) Skip large rows instead of throwing an exception to client
[
https://issues.apache.org/jira/browse/HBASE-20618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Swapna updated HBASE-20618:
---
Attachment: (was: HBASE-20618.hbasemaster.v01.patch)
> Skip large rows instead of throwing an exception to client
> --
>
> Key: HBASE-20618
> URL: https://issues.apache.org/jira/browse/HBASE-20618
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Swapna
>Priority: Minor
> Attachments: HBASE-20618.hbasemaster.v01.patch
>
>
> Currently HBase supports throwing RowTooBigException incase there is a row
> with one of the column family data exceeds the configured maximum
> https://issues.apache.org/jira/browse/HBASE-10925?attachmentOrder=desc
> We have some bad rows growing very large. We need a way to skip these rows
> for most of our jobs.
> Some of the options we considered:
> Option 1:
> Hbase client handle the exception and restart the scanner past bad row by
> capturing the row key where it failed. Can be by adding the rowkey to the
> exception stack trace, which seems brittle. Client would ignore the setting
> if its upgraded before server.
> Option 2:
> Skip through big rows on Server.Go with server level config similar to
> "hbase.table.max.rowsize" or request based by changing the scan request api.
> If allowed to do per request, based on the scan request config, Client will
> have to ignore the setting if its upgraded before server.
> {code}
> try {
> populateResult(results, this.storeHeap, scannerContext, current);
> } catch(RowTooBigException e) {
> LOG.info("Row exceeded the limit in storeheap. Skipping row with
> key:"+Bytes.toString(current.getRowArray()));
> this.storeHeap.reseek(PrivateCellUtil.createLastOnRow(current));
> results.clear();
> scannerContext.clearProgress();
> continue;
> }
> {code}
> Prefer the option 2 with server level config. Please share your inputs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20618) Skip large rows instead of throwing an exception to client
[
https://issues.apache.org/jira/browse/HBASE-20618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Swapna updated HBASE-20618:
---
Attachment: HBASE-20618.hbasemaster.v01.patch
Status: Patch Available (was: Open)
> Skip large rows instead of throwing an exception to client
> --
>
> Key: HBASE-20618
> URL: https://issues.apache.org/jira/browse/HBASE-20618
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Swapna
>Priority: Minor
> Attachments: HBASE-20618.hbasemaster.v01.patch
>
>
> Currently HBase supports throwing RowTooBigException incase there is a row
> with one of the column family data exceeds the configured maximum
> https://issues.apache.org/jira/browse/HBASE-10925?attachmentOrder=desc
> We have some bad rows growing very large. We need a way to skip these rows
> for most of our jobs.
> Some of the options we considered:
> Option 1:
> Hbase client handle the exception and restart the scanner past bad row by
> capturing the row key where it failed. Can be by adding the rowkey to the
> exception stack trace, which seems brittle. Client would ignore the setting
> if its upgraded before server.
> Option 2:
> Skip through big rows on Server.Go with server level config similar to
> "hbase.table.max.rowsize" or request based by changing the scan request api.
> If allowed to do per request, based on the scan request config, Client will
> have to ignore the setting if its upgraded before server.
> {code}
> try {
> populateResult(results, this.storeHeap, scannerContext, current);
> } catch(RowTooBigException e) {
> LOG.info("Row exceeded the limit in storeheap. Skipping row with
> key:"+Bytes.toString(current.getRowArray()));
> this.storeHeap.reseek(PrivateCellUtil.createLastOnRow(current));
> results.clear();
> scannerContext.clearProgress();
> continue;
> }
> {code}
> Prefer the option 2 with server level config. Please share your inputs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20618) Skip large rows instead of throwing an exception to client
[
https://issues.apache.org/jira/browse/HBASE-20618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Swapna updated HBASE-20618:
---
Attachment: HBASE-20618.hbasemaster.v01.patch
> Skip large rows instead of throwing an exception to client
> --
>
> Key: HBASE-20618
> URL: https://issues.apache.org/jira/browse/HBASE-20618
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Swapna
>Priority: Minor
> Attachments: HBASE-20618.hbasemaster.v01.patch
>
>
> Currently HBase supports throwing RowTooBigException incase there is a row
> with one of the column family data exceeds the configured maximum
> https://issues.apache.org/jira/browse/HBASE-10925?attachmentOrder=desc
> We have some bad rows growing very large. We need a way to skip these rows
> for most of our jobs.
> Some of the options we considered:
> Option 1:
> Hbase client handle the exception and restart the scanner past bad row by
> capturing the row key where it failed. Can be by adding the rowkey to the
> exception stack trace, which seems brittle. Client would ignore the setting
> if its upgraded before server.
> Option 2:
> Skip through big rows on Server.Go with server level config similar to
> "hbase.table.max.rowsize" or request based by changing the scan request api.
> If allowed to do per request, based on the scan request config, Client will
> have to ignore the setting if its upgraded before server.
> {code}
> try {
> populateResult(results, this.storeHeap, scannerContext, current);
> } catch(RowTooBigException e) {
> LOG.info("Row exceeded the limit in storeheap. Skipping row with
> key:"+Bytes.toString(current.getRowArray()));
> this.storeHeap.reseek(PrivateCellUtil.createLastOnRow(current));
> results.clear();
> scannerContext.clearProgress();
> continue;
> }
> {code}
> Prefer the option 2 with server level config. Please share your inputs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-15871) Memstore flush doesn't finish because of backwardseek() in memstore scanner.
[ https://issues.apache.org/jira/browse/HBASE-15871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494584#comment-16494584 ] Minwoo Kang edited comment on HBASE-15871 at 5/30/18 4:46 AM: -- Hi, All I am using HBase version 1.2.6, and I have been suffering this issue. Jeongdae Kim informed me that this issue might be related. It was perfectly in line with my situation. I think this issue should backport 1.2 branch. Is it possible? (I do not know backport-specific JIRA issue. So I left a comment here.) was (Author: minwoo.kang): Hi, All I am using HBase version 1.2.6, and I have been suffering this issue. Jeongdae Kim informed me that this issue might be related. It was perfectly in line with my situation. I think this issue should backport 1.2 branch. Is it possible? > Memstore flush doesn't finish because of backwardseek() in memstore scanner. > > > Key: HBASE-15871 > URL: https://issues.apache.org/jira/browse/HBASE-15871 > Project: HBase > Issue Type: Bug > Components: Scanners >Affects Versions: 1.1.2 >Reporter: Jeongdae Kim >Assignee: ramkrishna.s.vasudevan >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-15871-branch-1.patch, > HBASE-15871.branch-1.1.001.patch, HBASE-15871.branch-1.1.002.patch, > HBASE-15871.branch-1.1.003.patch, HBASE-15871.patch, HBASE-15871_1.patch, > HBASE-15871_1.patch, HBASE-15871_2.patch, HBASE-15871_3.patch, > HBASE-15871_4.patch, HBASE-15871_6.patch, memstore_backwardSeek().PNG > > > Sometimes in our production hbase cluster, it takes a long time to finish > memstore flush.( for about more than 30 minutes) > the reason is that a memstore flusher thread calls > StoreScanner.updateReaders(), waits for acquiring a lock that store scanner > holds in StoreScanner.next() and backwardseek() in memstore scanner runs for > a long time. > I think that this condition could occur in reverse scan by the following > process. > 1) create a reversed store scanner by requesting a reverse scan. > 2) flush a memstore in the same HStore. > 3) puts a lot of cells in memstore and memstore is almost full. > 4) call the reverse scanner.next() and re-create all scanners in this store > because all scanners was already closed by 2)'s flush() and backwardseek() > with store's lastTop for all new scanners. > 5) in this status, memstore is almost full by 2) and all cells in memstore > have sequenceID greater than this scanner's readPoint because of 2)'s > flush(). this condition causes searching all cells in memstore, and > seekToPreviousRow() repeatly seach cells that are already searched if a row > has one column. (described this in more detail in a attached file.) > 6) flush a memstore again in the same HStore, and wait until 4-5) process > finished, to update store files in the same HStore after flusing. > I searched HBase jira. and found a similar issue. (HBASE-14497) but, > HBASE-14497's fix can't solve this issue because that fix just changed > recursive call to loop.(and already applied to our HBase version) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20653) Add missing observer hooks for region server group to MasterObserver
[ https://issues.apache.org/jira/browse/HBASE-20653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-20653: --- Description: Currently the following region server group operations don't have corresponding hook in MasterObserver : * getRSGroupInfo * getRSGroupInfoOfServer * getRSGroupInfoOfTable * listRSGroup This JIRA is to * add them to MasterObserver * add pre/post hook calls in RSGroupAdminEndpoint thru master.getMasterCoprocessorHost for the above operations * add corresponding tests to TestRSGroups (in similar manner to that of HBASE-20627) was: Currently the following region server group operations don't have corresponding hook in MasterObserver : * getRSGroupInfo * getRSGroupInfoOfServer * getRSGroupInfoOfTable * listRSGroup This JIRA is to * add them to MasterObserver * add pre/post hook calls in RSGroupAdminEndpoint thru master.getMasterCoprocessorHost for the above operations * add corresponding tests to TestRSGroupsWithACL (or new test class, if needed) > Add missing observer hooks for region server group to MasterObserver > > > Key: HBASE-20653 > URL: https://issues.apache.org/jira/browse/HBASE-20653 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Nihal Jain >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20653.master.001.patch, > HBASE-20653.master.002.patch, HBASE-20653.master.003.patch > > > Currently the following region server group operations don't have > corresponding hook in MasterObserver : > * getRSGroupInfo > * getRSGroupInfoOfServer > * getRSGroupInfoOfTable > * listRSGroup > This JIRA is to > * add them to MasterObserver > * add pre/post hook calls in RSGroupAdminEndpoint thru > master.getMasterCoprocessorHost for the above operations > * add corresponding tests to TestRSGroups (in similar manner to that of > HBASE-20627) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20653) Add missing observer hooks for region server group to MasterObserver
[ https://issues.apache.org/jira/browse/HBASE-20653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494663#comment-16494663 ] Ted Yu commented on HBASE-20653: Thanks for the reminder, [~andrewcheng] I modified the description. Can you take a look ? > Add missing observer hooks for region server group to MasterObserver > > > Key: HBASE-20653 > URL: https://issues.apache.org/jira/browse/HBASE-20653 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Nihal Jain >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20653.master.001.patch, > HBASE-20653.master.002.patch, HBASE-20653.master.003.patch > > > Currently the following region server group operations don't have > corresponding hook in MasterObserver : > * getRSGroupInfo > * getRSGroupInfoOfServer > * getRSGroupInfoOfTable > * listRSGroup > This JIRA is to > * add them to MasterObserver > * add pre/post hook calls in RSGroupAdminEndpoint thru > master.getMasterCoprocessorHost for the above operations > * add corresponding tests to TestRSGroupsWithACL (or new test class, if > needed) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20653) Add missing observer hooks for region server group to MasterObserver
[
https://issues.apache.org/jira/browse/HBASE-20653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20653:
---
Description:
Currently the following region server group operations don't have corresponding
hook in MasterObserver :
* getRSGroupInfo
* getRSGroupInfoOfServer
* getRSGroupInfoOfTable
* listRSGroup
This JIRA is to
* add them to MasterObserver
* add pre/post hook calls in RSGroupAdminEndpoint thru
master.getMasterCoprocessorHost for the above operations
* add corresponding tests to TestRSGroupsWithACL (or new test class, if needed)
was:
Currently the following region server group operations don't have corresponding
hook in MasterObserver :
* getRSGroupInfo
* getRSGroupInfoOfServer
* getRSGroupInfoOfTable
* listRSGroup
This JIRA is to
* add them to MasterObserver
* add corresponding permission check in AccessController
* move the {{checkPermission}} out of RSGroupAdminEndpoint
* add corresponding tests to TestRSGroupsWithACL
> Add missing observer hooks for region server group to MasterObserver
>
>
> Key: HBASE-20653
> URL: https://issues.apache.org/jira/browse/HBASE-20653
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20653.master.001.patch,
> HBASE-20653.master.002.patch, HBASE-20653.master.003.patch
>
>
> Currently the following region server group operations don't have
> corresponding hook in MasterObserver :
> * getRSGroupInfo
> * getRSGroupInfoOfServer
> * getRSGroupInfoOfTable
> * listRSGroup
> This JIRA is to
> * add them to MasterObserver
> * add pre/post hook calls in RSGroupAdminEndpoint thru
> master.getMasterCoprocessorHost for the above operations
> * add corresponding tests to TestRSGroupsWithACL (or new test class, if
> needed)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494655#comment-16494655
]
Ted Yu commented on HBASE-20654:
Patch v3 only contained new classes.
Please combine v2 and v3 into v4, after addressing review comments.
If possible, please collect output from /jmx of master UI and attach relevant
snippet to this JIRA.
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20594) provide utility to compare old and new descriptors
[
https://issues.apache.org/jira/browse/HBASE-20594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494654#comment-16494654
]
Hudson commented on HBASE-20594:
Results for branch branch-2
[build #800 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/800/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/800//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/800//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/800//JDK8_Nightly_Build_Report_(Hadoop3)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> provide utility to compare old and new descriptors
> --
>
> Key: HBASE-20594
> URL: https://issues.apache.org/jira/browse/HBASE-20594
> Project: HBase
> Issue Type: Improvement
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Major
> Attachments: HBASE-20594.patch, HBASE-20594.v2.patch,
> HBASE-20594.v3.patch, HBASE-20594.v4.patch, HBASE-20594.v5.patch,
> HBASE-20594.v6.patch, HBASE-20594.v7.patch
>
>
> HBASE-20567 gives us hooks that give both the old and new descriptor in
> pre/postModify* events, but comparing them is still cumbersome. We should
> provide users some kind of utility for this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494641#comment-16494641
]
Ted Yu edited comment on HBASE-20654 at 5/30/18 3:49 AM:
-
Looks like patch v2 didn't include MetricsAssignmentWrapper since it is new
class.
Use this command and pipe its output to the end of next patch:
git diff --cached
was (Author: yuzhih...@gmail.com):
Looks like patch v2 didn't include MetricsAssignmentManagerSourceImpl since it
is new class.
Use this command and pipe its output to the end of next patch:
git diff --cached
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494641#comment-16494641
]
Ted Yu commented on HBASE-20654:
Looks like patch v2 didn't include MetricsAssignmentManagerSourceImpl since it
is new class.
Use this command and pipe its output to the end of next patch:
git diff --cached
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494638#comment-16494638
]
Ted Yu commented on HBASE-20654:
Can you take a look at the following compilation error based on patch v2 ?
{code}
[ERROR]
/Users/tyu/master/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsAssignmentManagerSourceImpl.java:[48,13]
cannot find symbol
symbol: class MetricsAssignmentWrapper
location: class
org.apache.hadoop.hbase.master.MetricsAssignmentManagerSourceImpl
[ERROR]
/Users/tyu/master/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsAssignmentManagerSourceImpl.java:[50,47]
cannot find symbol
symbol: class MetricsAssignmentWrapper
location: class
org.apache.hadoop.hbase.master.MetricsAssignmentManagerSourceImpl
{code}
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494633#comment-16494633
]
liubangchen commented on HBASE-20654:
-
Code review link : [https://reviews.apache.org/r/67376/diff/1#index_header]
Under your suggestions, I modified the code.Could you help me review it?
Thanks.
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HBASE-20533) Fix the flaky TestAssignmentManagerMetrics
[ https://issues.apache.org/jira/browse/HBASE-20533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu resolved HBASE-20533. -- Resolution: Fixed Hadoop Flags: Reviewed > Fix the flaky TestAssignmentManagerMetrics > -- > > Key: HBASE-20533 > URL: https://issues.apache.org/jira/browse/HBASE-20533 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.0, 2.0.0 >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.0.1 > > Attachments: HBASE-20533.v1.patch, HBASE-20533.v1.patch, > org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics-output.txt, > org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics.txt > > > See: > 1. > https://builds.apache.org/job/HBASE-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html > 2. https://builds.apache.org/job/HBASE-Flaky-Tests/30837/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20533) Fix the flaky TestAssignmentManagerMetrics
[ https://issues.apache.org/jira/browse/HBASE-20533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494630#comment-16494630 ] Zheng Hu commented on HBASE-20533: -- Not flaky any more, close this issue. > Fix the flaky TestAssignmentManagerMetrics > -- > > Key: HBASE-20533 > URL: https://issues.apache.org/jira/browse/HBASE-20533 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.0, 2.0.0 >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.0.1 > > Attachments: HBASE-20533.v1.patch, HBASE-20533.v1.patch, > org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics-output.txt, > org.apache.hadoop.hbase.master.TestAssignmentManagerMetrics.txt > > > See: > 1. > https://builds.apache.org/job/HBASE-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html > 2. https://builds.apache.org/job/HBASE-Flaky-Tests/30837/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494625#comment-16494625
]
Ted Yu commented on HBASE-20654:
Please upload next patch to https://reviews.apache.org/
Repository: hbase-git
Groups: hbase
Bugs: HBASE-20654
Thanks
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (HBASE-20639) Implement permission checking through AccessController instead of RSGroupAdminEndpoint
[
https://issues.apache.org/jira/browse/HBASE-20639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu reopened HBASE-20639:
> Implement permission checking through AccessController instead of
> RSGroupAdminEndpoint
> --
>
> Key: HBASE-20639
> URL: https://issues.apache.org/jira/browse/HBASE-20639
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Nihal Jain
>Priority: Major
> Attachments: HBASE-20639.master.001.patch,
> HBASE-20639.master.002.patch, HBASE-20639.master.002.patch
>
>
> Currently permission checking for various RS group operations is done via
> RSGroupAdminEndpoint.
> e.g. in RSGroupAdminServiceImpl#moveServers() :
> {code}
> checkPermission("moveServers");
> groupAdminServer.moveServers(hostPorts, request.getTargetGroup());
> {code}
> The practice in remaining parts of hbase is to perform permission checking
> within AccessController.
> Now that observer hooks for RS group operations are in right place, we should
> follow best practice and move permission checking to AccessController.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20639) Implement permission checking through AccessController instead of RSGroupAdminEndpoint
[
https://issues.apache.org/jira/browse/HBASE-20639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20639:
---
Fix Version/s: (was: 2.1.0)
(was: 3.0.0)
> Implement permission checking through AccessController instead of
> RSGroupAdminEndpoint
> --
>
> Key: HBASE-20639
> URL: https://issues.apache.org/jira/browse/HBASE-20639
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Nihal Jain
>Priority: Major
> Attachments: HBASE-20639.master.001.patch,
> HBASE-20639.master.002.patch, HBASE-20639.master.002.patch
>
>
> Currently permission checking for various RS group operations is done via
> RSGroupAdminEndpoint.
> e.g. in RSGroupAdminServiceImpl#moveServers() :
> {code}
> checkPermission("moveServers");
> groupAdminServer.moveServers(hostPorts, request.getTargetGroup());
> {code}
> The practice in remaining parts of hbase is to perform permission checking
> within AccessController.
> Now that observer hooks for RS group operations are in right place, we should
> follow best practice and move permission checking to AccessController.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20659) Implement a reopen table regions procedure
Duo Zhang created HBASE-20659: - Summary: Implement a reopen table regions procedure Key: HBASE-20659 URL: https://issues.apache.org/jira/browse/HBASE-20659 Project: HBase Issue Type: Bug Reporter: Duo Zhang For serial replication peer, and sync replication peer on branch HBASE-19064, sometimes we need to reopen the regions for a table to make sure that we have some wal edits which have not been replicated yet, or let the region switch a wal implementation. But for peer related procedures, we do not hold the table lock so if there are region split or merge than we may fail to reopen all the regions and cause problems. So I think we need to introduce a table procedure which is used to reopen all the regions for a table, and like other table procedures, such as ModifyTableProcedure, it will hold the exclusive lock on a table so we will not miss the new regions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20653) Add missing observer hooks for region server group to MasterObserver
[
https://issues.apache.org/jira/browse/HBASE-20653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494597#comment-16494597
]
Guangxu Cheng commented on HBASE-20653:
---
As HBASE-19483 Release Note:
{quote}Stripping out a public class AccessChecker from AccessController, using
ACL as core library in CPs.
Don't use CP hooks to check access in RSGroup.
{quote}
But this issue and HBASE-20639 move permission checking to AccessController
again. :(
> Add missing observer hooks for region server group to MasterObserver
>
>
> Key: HBASE-20653
> URL: https://issues.apache.org/jira/browse/HBASE-20653
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20653.master.001.patch,
> HBASE-20653.master.002.patch, HBASE-20653.master.003.patch
>
>
> Currently the following region server group operations don't have
> corresponding hook in MasterObserver :
> * getRSGroupInfo
> * getRSGroupInfoOfServer
> * getRSGroupInfoOfTable
> * listRSGroup
> This JIRA is to
> * add them to MasterObserver
> * add corresponding permission check in AccessController
> * move the {{checkPermission}} out of RSGroupAdminEndpoint
> * add corresponding tests to TestRSGroupsWithACL
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-15871) Memstore flush doesn't finish because of backwardseek() in memstore scanner.
[ https://issues.apache.org/jira/browse/HBASE-15871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494584#comment-16494584 ] Minwoo Kang commented on HBASE-15871: - Hi, All I am using HBase version 1.2.6, and I have been suffering this issue. Jeongdae Kim informed me that this issue might be related. It was perfectly in line with my situation. I think this issue should backport 1.2 branch. Is it possible? > Memstore flush doesn't finish because of backwardseek() in memstore scanner. > > > Key: HBASE-15871 > URL: https://issues.apache.org/jira/browse/HBASE-15871 > Project: HBase > Issue Type: Bug > Components: Scanners >Affects Versions: 1.1.2 >Reporter: Jeongdae Kim >Assignee: ramkrishna.s.vasudevan >Priority: Major > Fix For: 2.0.0 > > Attachments: HBASE-15871-branch-1.patch, > HBASE-15871.branch-1.1.001.patch, HBASE-15871.branch-1.1.002.patch, > HBASE-15871.branch-1.1.003.patch, HBASE-15871.patch, HBASE-15871_1.patch, > HBASE-15871_1.patch, HBASE-15871_2.patch, HBASE-15871_3.patch, > HBASE-15871_4.patch, HBASE-15871_6.patch, memstore_backwardSeek().PNG > > > Sometimes in our production hbase cluster, it takes a long time to finish > memstore flush.( for about more than 30 minutes) > the reason is that a memstore flusher thread calls > StoreScanner.updateReaders(), waits for acquiring a lock that store scanner > holds in StoreScanner.next() and backwardseek() in memstore scanner runs for > a long time. > I think that this condition could occur in reverse scan by the following > process. > 1) create a reversed store scanner by requesting a reverse scan. > 2) flush a memstore in the same HStore. > 3) puts a lot of cells in memstore and memstore is almost full. > 4) call the reverse scanner.next() and re-create all scanners in this store > because all scanners was already closed by 2)'s flush() and backwardseek() > with store's lastTop for all new scanners. > 5) in this status, memstore is almost full by 2) and all cells in memstore > have sequenceID greater than this scanner's readPoint because of 2)'s > flush(). this condition causes searching all cells in memstore, and > seekToPreviousRow() repeatly seach cells that are already searched if a row > has one column. (described this in more detail in a attached file.) > 6) flush a memstore again in the same HStore, and wait until 4-5) process > finished, to update store files in the same HStore after flusing. > I searched HBase jira. and found a similar issue. (HBASE-14497) but, > HBASE-14497's fix can't solve this issue because that fix just changed > recursive call to loop.(and already applied to our HBase version) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20637) Polish the WAL switching when transiting from A to S
[ https://issues.apache.org/jira/browse/HBASE-20637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494528#comment-16494528 ] Duo Zhang commented on HBASE-20637: --- TestSyncReplicationMoreLogsInLocalGiveUpSplitting is flakey, and is expected because of HBASE-20634. PTAL [~zghaobac] [~openinx]. Thanks. > Polish the WAL switching when transiting from A to S > > > Key: HBASE-20637 > URL: https://issues.apache.org/jira/browse/HBASE-20637 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: HBASE-19064 > > Attachments: HBASE-20637-HBASE-19064-v1.patch, > HBASE-20637-HBASE-19064-v2.patch, HBASE-20637-HBASE-19064-v3.patch, > HBASE-20637-HBASE-19064.patch > > > The design in HBASE-20424 implies that, when transiting from A to S, the RS > will abort and the WAL file will be split. > But in fact, there could be races that make a RS still alive after the > transiting which causes data inconsistency. And also, a full cluster restart > is expensive, we need to find a more smooth way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20634) Reopen region while server crash can cause the procedure to be stuck
[ https://issues.apache.org/jira/browse/HBASE-20634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494524#comment-16494524 ] Duo Zhang commented on HBASE-20634: --- Add a SERVER_CRASH_HANDLE_RIT2 state can not solve the problem, even if we do not hold the region lock on MoveRegionProcedure. It is possible that the MoveRegionProcedure is created but never scheduled. And after we execute SERVER_CRASH_HANDLE_RIT2 the UnassignProcedure is scheduled and then stuck... I think if we can not add a task to a given RS, then it means the RS is already dead, then we should finish the procedure. The problem is that whether we should fail the procedure. For example, for a MoveRegionProcedure, if the unassign and assign are both failed because of the region is not online, then what should we do? If it is used for reopening a region then it seems OK as the region has been reopened, but if it is used for moving a region... > Reopen region while server crash can cause the procedure to be stuck > > > Key: HBASE-20634 > URL: https://issues.apache.org/jira/browse/HBASE-20634 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 3.0.0, 2.1.0, 2.0.1 > > Attachments: HBASE-20634-UT.patch > > > Found this when implementing HBASE-20424, where we will transit the peer sync > replication state while there is server crash. > The problem is that, in ServerCrashAssign, we do not have the region lock, so > it is possible that after we call handleRIT to clear the existing > assign/unassign procedures related to this rs, and before we schedule the > assign procedures, it is possible that that we schedule a unassign procedure > for a region on the crashed rs. This procedure will not receive the > ServerCrashException, instead, in addToRemoteDispatcher, it will find that it > can not dispatch the remote call and then a FailedRemoteDispatchException > will be raised. But we do not treat this exception the same with > ServerCrashException, instead, we will try to expire the rs. Obviously the rs > has already been marked as expired, so this is almost a no-op. Then the > procedure will be stuck there for ever. > A possible way to fix it is to treat FailedRemoteDispatchException the same > with ServerCrashException, as it will be created in addToRemoteDispatcher > only, and the only reason we can not dispatch a remote call is that the rs > has already been dead. The nodeMap is a ConcurrentMap so I think we could use > it as a guard. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20658) Release latch early for ModifyTable and TruncateTable procedure.
[ https://issues.apache.org/jira/browse/HBASE-20658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494514#comment-16494514 ] Duo Zhang commented on HBASE-20658: --- The latch is used to simulate the old behavior for old client. > Release latch early for ModifyTable and TruncateTable procedure. > > > Key: HBASE-20658 > URL: https://issues.apache.org/jira/browse/HBASE-20658 > Project: HBase > Issue Type: Bug >Reporter: Ankit Singhal >Assignee: Ankit Singhal >Priority: Major > > It seems we are releasing latch in Modify and Truncate table procedures > during completion cleanup which make the whole procedure to execute > synchronously. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20658) Release latch early for ModifyTable and TruncateTable procedure.
Ankit Singhal created HBASE-20658: - Summary: Release latch early for ModifyTable and TruncateTable procedure. Key: HBASE-20658 URL: https://issues.apache.org/jira/browse/HBASE-20658 Project: HBase Issue Type: Bug Reporter: Ankit Singhal Assignee: Ankit Singhal It seems we are releasing latch in Modify and Truncate table procedures during completion cleanup which make the whole procedure to execute synchronously. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20597) Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint
[
https://issues.apache.org/jira/browse/HBASE-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494501#comment-16494501
]
Hudson commented on HBASE-20597:
Results for branch branch-2.0
[build #364 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/364/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/364//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/364//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/364//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Use a lock to serialize access to a shared reference to ZooKeeperWatcher in
> HBaseReplicationEndpoint
>
>
> Key: HBASE-20597
> URL: https://issues.apache.org/jira/browse/HBASE-20597
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.2, 1.4.4
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 2.0.1, 1.4.5
>
> Attachments: HBASE-20597-branch-1.addendum-v2.0.patch,
> HBASE-20597-branch-1.patch, HBASE-20597.addendum.0.patch, HBASE-20597.patch
>
>
> The code that closes down a ZKW that fails to initialize when attempting to
> connect to the remote cluster is not MT safe and can in theory leak
> ZooKeeperWatcher instances. The allocation of a new ZKW and store to the
> reference is not atomic. Might have concurrent allocations with only one
> winning store, leading to leaked ZKW instances.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20597) Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint
[
https://issues.apache.org/jira/browse/HBASE-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494500#comment-16494500
]
Hudson commented on HBASE-20597:
Results for branch branch-1.3
[build #345 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/345/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/345//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/345//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/345//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Use a lock to serialize access to a shared reference to ZooKeeperWatcher in
> HBaseReplicationEndpoint
>
>
> Key: HBASE-20597
> URL: https://issues.apache.org/jira/browse/HBASE-20597
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.2, 1.4.4
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 2.0.1, 1.4.5
>
> Attachments: HBASE-20597-branch-1.addendum-v2.0.patch,
> HBASE-20597-branch-1.patch, HBASE-20597.addendum.0.patch, HBASE-20597.patch
>
>
> The code that closes down a ZKW that fails to initialize when attempting to
> connect to the remote cluster is not MT safe and can in theory leak
> ZooKeeperWatcher instances. The allocation of a new ZKW and store to the
> reference is not atomic. Might have concurrent allocations with only one
> winning store, leading to leaked ZKW instances.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20390) IMC Default Parameters for 2.0.0
[
https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494465#comment-16494465
]
Hudson commented on HBASE-20390:
Results for branch master
[build #349 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/349/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/349//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/349//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/349//JDK8_Nightly_Build_Report_(Hadoop3)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> IMC Default Parameters for 2.0.0
>
>
> Key: HBASE-20390
> URL: https://issues.apache.org/jira/browse/HBASE-20390
> Project: HBase
> Issue Type: Sub-task
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
>Priority: Major
> Attachments: HBASE-20390-branch-2.0-01.patch,
> HBASE-20390-branch-2.0-01.patch, HBASE-20390.branch-2.0.002.patch,
> HBASE-20390.branch-2.0.003.patch, HBase 2.0 performance evaluation -
> throughput SSD_HDD.pdf, hits.ihc.png
>
>
> Setting new default parameters for in-memory compaction based on performance
> tests done in HBASE-20188
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20478) move import checks from hbaseanti to checkstyle
[
https://issues.apache.org/jira/browse/HBASE-20478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494466#comment-16494466
]
Hudson commented on HBASE-20478:
Results for branch master
[build #349 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/349/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/349//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/349//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/349//JDK8_Nightly_Build_Report_(Hadoop3)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> move import checks from hbaseanti to checkstyle
> ---
>
> Key: HBASE-20478
> URL: https://issues.apache.org/jira/browse/HBASE-20478
> Project: HBase
> Issue Type: Improvement
> Components: test
>Reporter: Sean Busbey
>Assignee: Mike Drob
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HBASE-20478.0.patch, HBASE-20478.1.patch,
> HBASE-20478.2.patch, HBASE-20478.3.patch, HBASE-20478.4.patch,
> HBASE-20478.5.patch, HBASE-20478.6.patch, HBASE-20478.8.patch,
> HBASE-20478.WIP.2.patch, HBASE-20478.WIP.2.patch, HBASE-20478.WIP.patch,
> HBASE-anti-check.patch
>
>
> came up in discussion on HBASE-20332. our check of "don't do this" things in
> the codebase doesn't log the specifics of complaints anywhere, which forces
> those who want to follow up to reverse engineer the check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20633) Dropping a table containing a disable violation policy fails to remove the quota upon table delete
[
https://issues.apache.org/jira/browse/HBASE-20633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494467#comment-16494467
]
Hudson commented on HBASE-20633:
Results for branch master
[build #349 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/349/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/349//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/349//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/349//JDK8_Nightly_Build_Report_(Hadoop3)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Dropping a table containing a disable violation policy fails to remove the
> quota upon table delete
> --
>
> Key: HBASE-20633
> URL: https://issues.apache.org/jira/browse/HBASE-20633
> Project: HBase
> Issue Type: Bug
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20633.master.001.patch,
> HBASE-20633.master.002.patch, HBASE-20633.master.002.patch
>
>
>
> {code:java}
> private void setQuotaAndThenDropTable(SpaceViolationPolicy policy) throws
> Exception {
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> SpaceViolationPolicy policy = SpaceViolationPolicy.DISABLE;
> // Do puts until we violate space policy
> final TableName tn = writeUntilViolationAndVerifyViolation(policy, put);
> // Now, drop the table
> TEST_UTIL.deleteTable(tn);
> LOG.debug("Successfully deleted table ", tn);
> // Now re-create the table
> TEST_UTIL.createTable(tn, Bytes.toBytes(SpaceQuotaHelperForTests.F1));
> LOG.debug("Successfully re-created table ", tn);
> // Put some rows now: should not violate as table/quota was dropped
> verifyNoViolation(policy, tn, put);
> }
> {code}
> * When we drop a table, upon completion the quota triggers removal of
> disable policy, thus causing the system to enable the table
> {noformat}
> 2018-05-18 18:08:58,189 DEBUG [PEWorker-13]
> procedure.DeleteTableProcedure(130): delete
> 'testSetQuotaAndThenDropTableWithDisable19' completed
> 2018-05-18 18:08:58,191 INFO [PEWorker-13]
> procedure2.ProcedureExecutor(1265): Finished pid=328, state=SUCCESS;
> DeleteTableProcedure table=testSetQuotaAndThenDropTableWithDisable19 in
> 271msec
> 2018-05-18 18:08:58,321 INFO [regionserver/ba4cba1aa13d:0.Chore.1]
> client.HBaseAdmin$14(844): Started enable of
> testSetQuotaAndThenDropTableWithDisable19{noformat}
> * But, since the table has already been dropped, enable procedure would
> rollback
> {noformat}
> 2018-05-18 18:08:58,427 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46443]
> procedure2.ProcedureExecutor(884): Stored pid=329,
> state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure
> table=testSetQuotaAndThenDropTableWithDisable19
> 2018-05-18 18:08:58,430 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46443]
> master.MasterRpcServices(1141): Checking to see if procedure is done pid=329
> 2018-05-18 18:08:58,451 INFO [PEWorker-10]
> procedure2.ProcedureExecutor(1359): Rolled back pid=329, state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.TableNotFoundException via
> master-enable-table:org.apache.hadoop.hbase.TableNotFoundException:
> testSetQuotaAndThenDropTableWithDisable19; EnableTableProcedure
> table=testSetQuotaAndThenDropTableWithDisable19 exec-time=124msec
> 2018-05-18 18:08:58,533 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46443]
> master.MasterRpcServices(1141): Checking to see if procedure is done pid=329
> 2018-05-18 18:08:58,535 INFO [regionserver/ba4cba1aa13d:0.Chore.1]
> client.HBaseAdmin$TableFuture(3652): Operation: ENABLE, Table Name:
> default:testSetQuotaAndThenDropTableWithDisable19 failed with
> testSetQuotaAndThenDropTableWithDisable19{noformat}
> * Since, quota manager fails to enable table (i.e disable violation policy),
> it would not remove the policy, causing problems if table re-created
> {noformat}
> 2018-05-18 18:08:58,536 ERROR [regionserver/ba4cba1aa13d:0.Chore.1]
> quotas.RegionServerSpaceQuotaManager(210): Failed to disable space violation
> policy for
[jira] [Commented] (HBASE-20597) Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint
[
https://issues.apache.org/jira/browse/HBASE-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494468#comment-16494468
]
Hudson commented on HBASE-20597:
Results for branch master
[build #349 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/349/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/349//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/349//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/349//JDK8_Nightly_Build_Report_(Hadoop3)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Use a lock to serialize access to a shared reference to ZooKeeperWatcher in
> HBaseReplicationEndpoint
>
>
> Key: HBASE-20597
> URL: https://issues.apache.org/jira/browse/HBASE-20597
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.2, 1.4.4
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 2.0.1, 1.4.5
>
> Attachments: HBASE-20597-branch-1.addendum-v2.0.patch,
> HBASE-20597-branch-1.patch, HBASE-20597.addendum.0.patch, HBASE-20597.patch
>
>
> The code that closes down a ZKW that fails to initialize when attempting to
> connect to the remote cluster is not MT safe and can in theory leak
> ZooKeeperWatcher instances. The allocation of a new ZKW and store to the
> reference is not atomic. Might have concurrent allocations with only one
> winning store, leading to leaked ZKW instances.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20653) Add missing observer hooks for region server group to MasterObserver
[
https://issues.apache.org/jira/browse/HBASE-20653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494456#comment-16494456
]
Ted Yu commented on HBASE-20653:
{code}
error: patch failed:
hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java:78
error:
hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdminEndpoint.java:
patch does not apply
Patch failed at 0001 HBASE-20653 Add missing observer hooks for region server
group to MasterObserver
{code}
Please attach patch for branch-2
> Add missing observer hooks for region server group to MasterObserver
>
>
> Key: HBASE-20653
> URL: https://issues.apache.org/jira/browse/HBASE-20653
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20653.master.001.patch,
> HBASE-20653.master.002.patch, HBASE-20653.master.003.patch
>
>
> Currently the following region server group operations don't have
> corresponding hook in MasterObserver :
> * getRSGroupInfo
> * getRSGroupInfoOfServer
> * getRSGroupInfoOfTable
> * listRSGroup
> This JIRA is to
> * add them to MasterObserver
> * add corresponding permission check in AccessController
> * move the {{checkPermission}} out of RSGroupAdminEndpoint
> * add corresponding tests to TestRSGroupsWithACL
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20597) Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint
[
https://issues.apache.org/jira/browse/HBASE-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494448#comment-16494448
]
Hudson commented on HBASE-20597:
Results for branch branch-2
[build #799 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/799/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/799//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/799//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/799//JDK8_Nightly_Build_Report_(Hadoop3)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Use a lock to serialize access to a shared reference to ZooKeeperWatcher in
> HBaseReplicationEndpoint
>
>
> Key: HBASE-20597
> URL: https://issues.apache.org/jira/browse/HBASE-20597
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.2, 1.4.4
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 2.0.1, 1.4.5
>
> Attachments: HBASE-20597-branch-1.addendum-v2.0.patch,
> HBASE-20597-branch-1.patch, HBASE-20597.addendum.0.patch, HBASE-20597.patch
>
>
> The code that closes down a ZKW that fails to initialize when attempting to
> connect to the remote cluster is not MT safe and can in theory leak
> ZooKeeperWatcher instances. The allocation of a new ZKW and store to the
> reference is not atomic. Might have concurrent allocations with only one
> winning store, leading to leaked ZKW instances.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20657) Retrying RPC call for ModifyTableProcedure may stuck
Sergey Soldatov created HBASE-20657:
---
Summary: Retrying RPC call for ModifyTableProcedure may stuck
Key: HBASE-20657
URL: https://issues.apache.org/jira/browse/HBASE-20657
Project: HBase
Issue Type: Bug
Components: Client, proc-v2
Affects Versions: 2.0.0
Reporter: Sergey Soldatov
Env: 2 masters, 1 RS.
Steps to reproduce: Active master is killed while ModifyTableProcedure is
executed.
If the table has enough regions it may come that when the secondary master get
active some of the regions may be closed, so once client retries the call to
the new active master, a new ModifyTableProcedure is created and get stuck
during MODIFY_TABLE_REOPEN_ALL_REGIONS state handling. That happens because:
1. When we are retrying from client side, we call modifyTableAsync which create
a procedure with a new nonce key:
{noformat}
ModifyTableRequest request = RequestConverter.buildModifyTableRequest(
td.getTableName(), td, ng.getNonceGroup(), ng.newNonce());
{noformat}
So on the server side, it's considered as a new procedure and starts executing
immediately.
2. When we are processing MODIFY_TABLE_REOPEN_ALL_REGIONS we create
MoveRegionProcedure for each region, but it checks whether the region is online
(and it's not), so it fails immediately, forcing the procedure to restart.
[~an...@apache.org] saw a similar case when two concurrent ModifyTable
procedures were running and got stuck in the similar way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20657) Retrying RPC call for ModifyTableProcedure may get stuck
[
https://issues.apache.org/jira/browse/HBASE-20657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Soldatov updated HBASE-20657:
Summary: Retrying RPC call for ModifyTableProcedure may get stuck (was:
Retrying RPC call for ModifyTableProcedure may stuck)
> Retrying RPC call for ModifyTableProcedure may get stuck
>
>
> Key: HBASE-20657
> URL: https://issues.apache.org/jira/browse/HBASE-20657
> Project: HBase
> Issue Type: Bug
> Components: Client, proc-v2
>Affects Versions: 2.0.0
>Reporter: Sergey Soldatov
>Priority: Major
>
> Env: 2 masters, 1 RS.
> Steps to reproduce: Active master is killed while ModifyTableProcedure is
> executed.
> If the table has enough regions it may come that when the secondary master
> get active some of the regions may be closed, so once client retries the call
> to the new active master, a new ModifyTableProcedure is created and get stuck
> during MODIFY_TABLE_REOPEN_ALL_REGIONS state handling. That happens because:
> 1. When we are retrying from client side, we call modifyTableAsync which
> create a procedure with a new nonce key:
> {noformat}
> ModifyTableRequest request =
> RequestConverter.buildModifyTableRequest(
> td.getTableName(), td, ng.getNonceGroup(), ng.newNonce());
> {noformat}
> So on the server side, it's considered as a new procedure and starts
> executing immediately.
> 2. When we are processing MODIFY_TABLE_REOPEN_ALL_REGIONS we create
> MoveRegionProcedure for each region, but it checks whether the region is
> online (and it's not), so it fails immediately, forcing the procedure to
> restart.
> [~an...@apache.org] saw a similar case when two concurrent ModifyTable
> procedures were running and got stuck in the similar way.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20642) IntegrationTestDDLMasterFailover throws 'InvalidFamilyOperationException
[ https://issues.apache.org/jira/browse/HBASE-20642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494416#comment-16494416 ] Ankit Singhal commented on HBASE-20642: --- bq. That is not my understanding. The nonces are in an in-memory-only map in the Master process. They will not be migrated from one Master to the new one so, even if you put calls behind a nonce-check, it'll fail since the nonce-map is empty on new Master. I just checked the code and see that while loading procedure(ProcedureExecutor#loadProcedures) from MasterProcWals during restart, we are already rebuilding in-memory nonce map from the persisted noncekey. so the retry should also get identified on the new master and ignored. Though there is different problem with the client which is generating new noncekey for every retry. bq. Because the Master is failing which broke the synchronous wait on add column? Maybe add a check if master is going down and if it is throw that for an exception instead of doing this pre-flight check against current state of table descriptor? Would that be more meaningful? We may not catch it If the master is killed, nevertheless user may not expect the exception if the procedure is completed successfully at the new master. bq. It is pretty cool that the call keeps going though the Master has crashed... I think it is a bit much to expect that this call can pick up where it left off on the old Master though. It has no reference to the original transaction (it does not have a Future ). The retry call was moved to the new master and the new master during initialization will pick up procedure from the state at which it was persisted last by an old master in ProcWals and ignore the retry. so we should not have any problem if the operations are idempotent for a state. Will not this process happen when standby become active? bq. We want to move folks over to the async calls where they check to see if the Procedure is completed. It seems after HBASE-19953, Asynchronous call for DDLs also wait for the procedure to complete( as countDown() will happen when the procedure is completed). Thanks for bearing with me, I know there are too many questions. > IntegrationTestDDLMasterFailover throws 'InvalidFamilyOperationException > - > > Key: HBASE-20642 > URL: https://issues.apache.org/jira/browse/HBASE-20642 > Project: HBase > Issue Type: Bug >Reporter: Ankit Singhal >Assignee: Ankit Singhal >Priority: Major > Attachments: HBASE-20642.patch > > > [~romil.choksi] reported that IntegrationTestDDLMasterFailover is failing > while adding column family during the time master is restarting. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20331) clean up shaded packaging for 2.1
[
https://issues.apache.org/jira/browse/HBASE-20331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494415#comment-16494415
]
Hudson commented on HBASE-20331:
Results for branch HBASE-20331
[build #24 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/24/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/24//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/24//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/24//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/24//artifacts/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> clean up shaded packaging for 2.1
> -
>
> Key: HBASE-20331
> URL: https://issues.apache.org/jira/browse/HBASE-20331
> Project: HBase
> Issue Type: Umbrella
> Components: Client, mapreduce, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
>
> polishing pass on shaded modules for 2.0 based on trying to use them in more
> contexts.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20597) Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint
[
https://issues.apache.org/jira/browse/HBASE-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494412#comment-16494412
]
Hudson commented on HBASE-20597:
Results for branch branch-1.2
[build #349 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/349/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/349//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/349//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/349//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Use a lock to serialize access to a shared reference to ZooKeeperWatcher in
> HBaseReplicationEndpoint
>
>
> Key: HBASE-20597
> URL: https://issues.apache.org/jira/browse/HBASE-20597
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Affects Versions: 1.3.2, 1.4.4
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 2.0.1, 1.4.5
>
> Attachments: HBASE-20597-branch-1.addendum-v2.0.patch,
> HBASE-20597-branch-1.patch, HBASE-20597.addendum.0.patch, HBASE-20597.patch
>
>
> The code that closes down a ZKW that fails to initialize when attempting to
> connect to the remote cluster is not MT safe and can in theory leak
> ZooKeeperWatcher instances. The allocation of a new ZKW and store to the
> reference is not atomic. Might have concurrent allocations with only one
> winning store, leading to leaked ZKW instances.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20634) Reopen region while server crash can cause the procedure to be stuck
[ https://issues.apache.org/jira/browse/HBASE-20634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494378#comment-16494378 ] stack commented on HBASE-20634: --- This is a good one. We've seen it before. In SCP we have a step SERVER_CRASH_HANDLE_RIT2 that reruns the handleRIT for a case similar to this where we've managed to assign AFTER SCP has called handleRIT and it has split logs. What is interesting here is that SCP is locked out because of the concurrent MoveProcedure's assign which has managed to grab the region lock first. > Reopen region while server crash can cause the procedure to be stuck > > > Key: HBASE-20634 > URL: https://issues.apache.org/jira/browse/HBASE-20634 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 3.0.0, 2.1.0, 2.0.1 > > Attachments: HBASE-20634-UT.patch > > > Found this when implementing HBASE-20424, where we will transit the peer sync > replication state while there is server crash. > The problem is that, in ServerCrashAssign, we do not have the region lock, so > it is possible that after we call handleRIT to clear the existing > assign/unassign procedures related to this rs, and before we schedule the > assign procedures, it is possible that that we schedule a unassign procedure > for a region on the crashed rs. This procedure will not receive the > ServerCrashException, instead, in addToRemoteDispatcher, it will find that it > can not dispatch the remote call and then a FailedRemoteDispatchException > will be raised. But we do not treat this exception the same with > ServerCrashException, instead, we will try to expire the rs. Obviously the rs > has already been marked as expired, so this is almost a no-op. Then the > procedure will be stuck there for ever. > A possible way to fix it is to treat FailedRemoteDispatchException the same > with ServerCrashException, as it will be created in addToRemoteDispatcher > only, and the only reason we can not dispatch a remote call is that the rs > has already been dead. The nodeMap is a ConcurrentMap so I think we could use > it as a guard. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20582) Bump up JRuby version because of some reported vulnerabilities
[
https://issues.apache.org/jira/browse/HBASE-20582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494360#comment-16494360
]
Hadoop QA commented on HBASE-20582:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
55s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
51s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
17m 25s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}181m
3s{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}238m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20582 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12925217/HBASE-20582.addendum.patch
|
| Optional Tests | asflicense javac javadoc unit shadedjars hadoopcheck
xml compile |
| uname | Linux a2ad9b9afd19 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 7ff29d8e00 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13001/testReport/ |
| Max. process+thread count | 4539 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13001/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Bump up JRuby version because of some reported vulnerabilities
> --
>
> Key: HBASE-20582
> URL: https://issues.apache.org/jira/browse/HBASE-20582
> Project: HBase
> Issue Type: Bug
> Components: dependencies, shell
>Reporter: Ankit Singhal
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20582.002.patch,
[jira] [Commented] (HBASE-20605) Exclude new Azure Storage FileSystem from SecureBulkLoadEndpoint permission check
[ https://issues.apache.org/jira/browse/HBASE-20605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494335#comment-16494335 ] Sean Busbey commented on HBASE-20605: - bq. Would you be OK with me doing this separately from this issue? I think it will be complicated since it will be different changes from 1.x to 2.x release (whereas this one doesn't even apply to 2.x right now). yeah that makes sense. > Exclude new Azure Storage FileSystem from SecureBulkLoadEndpoint permission > check > - > > Key: HBASE-20605 > URL: https://issues.apache.org/jira/browse/HBASE-20605 > Project: HBase > Issue Type: Improvement > Components: security >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 1.5.0, 1.3.3, 1.4.5 > > Attachments: HBASE-20605.001.branch-1.patch, > HBASE-20605.002.branch-1.patch > > > Some folks in Hadoop are working on landing a new FileSystem from the Azure > team: HADOOP-15407 > At present, this FileSystem doesn't support permissions which causes the > SecureBulkLoadEndpoint to balk because it the staging directory doesn't have > the proper 711 permissions. > We have a static list of FileSystem schemes which we ignore this check on. I > have a patch on an HBase 1.1ish which: > # Adds the new FileSystem scheme > # Makes this list configurable for the future -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20605) Exclude new Azure Storage FileSystem from SecureBulkLoadEndpoint permission check
[
https://issues.apache.org/jira/browse/HBASE-20605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494313#comment-16494313
]
Josh Elser commented on HBASE-20605:
{quote}So ideally I'd like a WARN message that says something like "we aren't
sure about FileSystem %FOO, here's where to look in the docs:
[https://s.apache.org/sofhoiwjef];
{quote}
We're on the same page. This is the type of addition to the book I was thinking
about. I'm not sure I agree with you about operators actually trolling enough
to find this, but am happy to try to add it.
Would you be OK with me doing this separately from this issue? I think it will
be complicated since it will be different changes from 1.x to 2.x release
(whereas this one doesn't even apply to 2.x right now).
> Exclude new Azure Storage FileSystem from SecureBulkLoadEndpoint permission
> check
> -
>
> Key: HBASE-20605
> URL: https://issues.apache.org/jira/browse/HBASE-20605
> Project: HBase
> Issue Type: Improvement
> Components: security
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 1.5.0, 1.3.3, 1.4.5
>
> Attachments: HBASE-20605.001.branch-1.patch,
> HBASE-20605.002.branch-1.patch
>
>
> Some folks in Hadoop are working on landing a new FileSystem from the Azure
> team: HADOOP-15407
> At present, this FileSystem doesn't support permissions which causes the
> SecureBulkLoadEndpoint to balk because it the staging directory doesn't have
> the proper 711 permissions.
> We have a static list of FileSystem schemes which we ignore this check on. I
> have a patch on an HBase 1.1ish which:
> # Adds the new FileSystem scheme
> # Makes this list configurable for the future
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20605) Exclude new Azure Storage FileSystem from SecureBulkLoadEndpoint permission check
[ https://issues.apache.org/jira/browse/HBASE-20605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494288#comment-16494288 ] Sean Busbey commented on HBASE-20605: - I personally think a log message is more likely to get operator attention than docs, so even if that log message just points to docs that's a boon. So ideally I'd like a WARN message that says something like "we aren't sure about FileSystem %FOO, here's where to look in the docs: https://s.apache.org/sofhoiwjef;. But some folks get alerts whenever anything logs at WARN or above. So a "I know what I'm doing" config that turns off the WARN would probably be needed. > Exclude new Azure Storage FileSystem from SecureBulkLoadEndpoint permission > check > - > > Key: HBASE-20605 > URL: https://issues.apache.org/jira/browse/HBASE-20605 > Project: HBase > Issue Type: Improvement > Components: security >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 1.5.0, 1.3.3, 1.4.5 > > Attachments: HBASE-20605.001.branch-1.patch, > HBASE-20605.002.branch-1.patch > > > Some folks in Hadoop are working on landing a new FileSystem from the Azure > team: HADOOP-15407 > At present, this FileSystem doesn't support permissions which causes the > SecureBulkLoadEndpoint to balk because it the staging directory doesn't have > the proper 711 permissions. > We have a static list of FileSystem schemes which we ignore this check on. I > have a patch on an HBase 1.1ish which: > # Adds the new FileSystem scheme > # Makes this list configurable for the future -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20653) Add missing observer hooks for region server group to MasterObserver
[
https://issues.apache.org/jira/browse/HBASE-20653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494271#comment-16494271
]
Hadoop QA commented on HBASE-20653:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
12s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
7s{color} | {color:blue} hbase-server in master has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
57s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
15m 47s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}125m
53s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
26s{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}181m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20653 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12925602/HBASE-20653.master.003.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux a4d9862ad871 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 7ff29d8e00 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z)
[jira] [Commented] (HBASE-20653) Add missing observer hooks for region server group to MasterObserver
[
https://issues.apache.org/jira/browse/HBASE-20653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494250#comment-16494250
]
Hadoop QA commented on HBASE-20653:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
59s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
11s{color} | {color:blue} hbase-server in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
58s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
15m 25s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}124m
52s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
41s{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}180m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20653 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12925602/HBASE-20653.master.003.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux f83e23ca8840 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 7ff29d8e00 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
[jira] [Commented] (HBASE-20597) Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494171#comment-16494171 ] Hudson commented on HBASE-20597: SUCCESS: Integrated in Jenkins build HBase-1.2-IT #1115 (See [https://builds.apache.org/job/HBase-1.2-IT/1115/]) Revert "HBASE-20597 Use a lock to serialize access to a shared reference (apurtell: rev 419ee99e0239e680b4cfeb5c4650a54a3a662452) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HBaseReplicationEndpoint.java HBASE-20597 Serialize access to a shared reference to ZooKeeperWatcher (apurtell: rev 6594f2656bc23b33370106c94648ec1abb1f62e7) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HBaseReplicationEndpoint.java > Use a lock to serialize access to a shared reference to ZooKeeperWatcher in > HBaseReplicationEndpoint > > > Key: HBASE-20597 > URL: https://issues.apache.org/jira/browse/HBASE-20597 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.3.2, 1.4.4 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 2.0.1, 1.4.5 > > Attachments: HBASE-20597-branch-1.addendum-v2.0.patch, > HBASE-20597-branch-1.patch, HBASE-20597.addendum.0.patch, HBASE-20597.patch > > > The code that closes down a ZKW that fails to initialize when attempting to > connect to the remote cluster is not MT safe and can in theory leak > ZooKeeperWatcher instances. The allocation of a new ZKW and store to the > reference is not atomic. Might have concurrent allocations with only one > winning store, leading to leaked ZKW instances. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-11625) Reading datablock throws "Invalid HFile block magic" and can not switch to hdfs checksum
[ https://issues.apache.org/jira/browse/HBASE-11625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez reassigned HBASE-11625: - Assignee: Appy (was: Esteban Gutierrez) > Reading datablock throws "Invalid HFile block magic" and can not switch to > hdfs checksum > - > > Key: HBASE-11625 > URL: https://issues.apache.org/jira/browse/HBASE-11625 > Project: HBase > Issue Type: Bug > Components: HFile >Affects Versions: 0.94.21, 0.98.4, 0.98.5, 1.0.1.1, 1.0.3 >Reporter: qian wang >Assignee: Appy >Priority: Major > Fix For: 1.3.0, 1.2.2, 1.1.6, 2.0.0 > > Attachments: 2711de1fdf73419d9f8afc6a8b86ce64.gz, > HBASE-11625-branch-1-v1.patch, HBASE-11625-branch-1.2-v1.patch, > HBASE-11625-branch-1.2-v2.patch, HBASE-11625-branch-1.2-v3.patch, > HBASE-11625-branch-1.2-v4.patch, HBASE-11625-master-v2.patch, > HBASE-11625-master-v3.patch, HBASE-11625-master.patch, > HBASE-11625.branch-1.1.001.patch, HBASE-11625.patch, correct-hfile, > corrupted-header-hfile > > > when using hbase checksum,call readBlockDataInternal() in hfileblock.java, it > could happen file corruption but it only can switch to hdfs checksum > inputstream till validateBlockChecksum(). If the datablock's header corrupted > when b = new HFileBlock(),it throws the exception "Invalid HFile block magic" > and the rpc call fail -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-11625) Reading datablock throws "Invalid HFile block magic" and can not switch to hdfs checksum
[ https://issues.apache.org/jira/browse/HBASE-11625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez reassigned HBASE-11625: - Assignee: Esteban Gutierrez (was: Appy) > Reading datablock throws "Invalid HFile block magic" and can not switch to > hdfs checksum > - > > Key: HBASE-11625 > URL: https://issues.apache.org/jira/browse/HBASE-11625 > Project: HBase > Issue Type: Bug > Components: HFile >Affects Versions: 0.94.21, 0.98.4, 0.98.5, 1.0.1.1, 1.0.3 >Reporter: qian wang >Assignee: Esteban Gutierrez >Priority: Major > Fix For: 1.3.0, 1.2.2, 1.1.6, 2.0.0 > > Attachments: 2711de1fdf73419d9f8afc6a8b86ce64.gz, > HBASE-11625-branch-1-v1.patch, HBASE-11625-branch-1.2-v1.patch, > HBASE-11625-branch-1.2-v2.patch, HBASE-11625-branch-1.2-v3.patch, > HBASE-11625-branch-1.2-v4.patch, HBASE-11625-master-v2.patch, > HBASE-11625-master-v3.patch, HBASE-11625-master.patch, > HBASE-11625.branch-1.1.001.patch, HBASE-11625.patch, correct-hfile, > corrupted-header-hfile > > > when using hbase checksum,call readBlockDataInternal() in hfileblock.java, it > could happen file corruption but it only can switch to hdfs checksum > inputstream till validateBlockChecksum(). If the datablock's header corrupted > when b = new HFileBlock(),it throws the exception "Invalid HFile block magic" > and the rpc call fail -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20597) Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494142#comment-16494142 ] Hudson commented on HBASE-20597: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #414 (See [https://builds.apache.org/job/HBase-1.3-IT/414/]) Revert "HBASE-20597 Use a lock to serialize access to a shared reference (apurtell: rev 8e79429f4c09ea9bb3c9638e83bb4addfe85) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HBaseReplicationEndpoint.java HBASE-20597 Serialize access to a shared reference to ZooKeeperWatcher (apurtell: rev 405a30404bde4e6256f20542008a25e33b6b732e) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/HBaseReplicationEndpoint.java > Use a lock to serialize access to a shared reference to ZooKeeperWatcher in > HBaseReplicationEndpoint > > > Key: HBASE-20597 > URL: https://issues.apache.org/jira/browse/HBASE-20597 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.3.2, 1.4.4 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 2.0.1, 1.4.5 > > Attachments: HBASE-20597-branch-1.addendum-v2.0.patch, > HBASE-20597-branch-1.patch, HBASE-20597.addendum.0.patch, HBASE-20597.patch > > > The code that closes down a ZKW that fails to initialize when attempting to > connect to the remote cluster is not MT safe and can in theory leak > ZooKeeperWatcher instances. The allocation of a new ZKW and store to the > reference is not atomic. Might have concurrent allocations with only one > winning store, leading to leaked ZKW instances. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20390) IMC Default Parameters for 2.0.0
[
https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494107#comment-16494107
]
Hudson commented on HBASE-20390:
Results for branch branch-2.0
[build #363 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/363/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/363//console].
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/363//console].
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/363//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> IMC Default Parameters for 2.0.0
>
>
> Key: HBASE-20390
> URL: https://issues.apache.org/jira/browse/HBASE-20390
> Project: HBase
> Issue Type: Sub-task
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
>Priority: Major
> Attachments: HBASE-20390-branch-2.0-01.patch,
> HBASE-20390-branch-2.0-01.patch, HBASE-20390.branch-2.0.002.patch,
> HBASE-20390.branch-2.0.003.patch, HBase 2.0 performance evaluation -
> throughput SSD_HDD.pdf, hits.ihc.png
>
>
> Setting new default parameters for in-memory compaction based on performance
> tests done in HBASE-20188
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20645) Fix security_available method in security.rb
[
https://issues.apache.org/jira/browse/HBASE-20645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494108#comment-16494108
]
Hudson commented on HBASE-20645:
Results for branch branch-2.0
[build #363 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/363/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/363//console].
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/363//console].
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/363//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Fix security_available method in security.rb
> -
>
> Key: HBASE-20645
> URL: https://issues.apache.org/jira/browse/HBASE-20645
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20645.patch
>
>
> "exists?" method expects parameter tableName to be String but ACL_TABLE_NAME
> is of org.apache.hadoop.hbase.TableName form.
> {code}
> raise(ArgumentError, 'DISABLED: Security features are not available') unless \
>
> exists?(org.apache.hadoop.hbase.security.access.AccessControlLists::ACL_TABLE_NAME.getNameAsString)
> {code}
> Impact of the bug:-
> So , if a user is running any security related
> command(revoke,user_permission) and there is an exception(MasterNotRunning)
> while checking security capabilities, then instead of seeing the underlying
> exception, user is seeing
> {code}
> ERROR: no method 'valueOf' for arguments (org.apache.hadoop.hbase.TableName)
> on Java::OrgApacheHadoopHbase::TableName
> available overloads:
> (java.lang.String)
> (byte[])
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20633) Dropping a table containing a disable violation policy fails to remove the quota upon table delete
[
https://issues.apache.org/jira/browse/HBASE-20633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494109#comment-16494109
]
Hudson commented on HBASE-20633:
Results for branch branch-2.0
[build #363 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/363/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/363//console].
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/363//console].
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/363//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Dropping a table containing a disable violation policy fails to remove the
> quota upon table delete
> --
>
> Key: HBASE-20633
> URL: https://issues.apache.org/jira/browse/HBASE-20633
> Project: HBase
> Issue Type: Bug
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20633.master.001.patch,
> HBASE-20633.master.002.patch, HBASE-20633.master.002.patch
>
>
>
> {code:java}
> private void setQuotaAndThenDropTable(SpaceViolationPolicy policy) throws
> Exception {
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> SpaceViolationPolicy policy = SpaceViolationPolicy.DISABLE;
> // Do puts until we violate space policy
> final TableName tn = writeUntilViolationAndVerifyViolation(policy, put);
> // Now, drop the table
> TEST_UTIL.deleteTable(tn);
> LOG.debug("Successfully deleted table ", tn);
> // Now re-create the table
> TEST_UTIL.createTable(tn, Bytes.toBytes(SpaceQuotaHelperForTests.F1));
> LOG.debug("Successfully re-created table ", tn);
> // Put some rows now: should not violate as table/quota was dropped
> verifyNoViolation(policy, tn, put);
> }
> {code}
> * When we drop a table, upon completion the quota triggers removal of
> disable policy, thus causing the system to enable the table
> {noformat}
> 2018-05-18 18:08:58,189 DEBUG [PEWorker-13]
> procedure.DeleteTableProcedure(130): delete
> 'testSetQuotaAndThenDropTableWithDisable19' completed
> 2018-05-18 18:08:58,191 INFO [PEWorker-13]
> procedure2.ProcedureExecutor(1265): Finished pid=328, state=SUCCESS;
> DeleteTableProcedure table=testSetQuotaAndThenDropTableWithDisable19 in
> 271msec
> 2018-05-18 18:08:58,321 INFO [regionserver/ba4cba1aa13d:0.Chore.1]
> client.HBaseAdmin$14(844): Started enable of
> testSetQuotaAndThenDropTableWithDisable19{noformat}
> * But, since the table has already been dropped, enable procedure would
> rollback
> {noformat}
> 2018-05-18 18:08:58,427 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46443]
> procedure2.ProcedureExecutor(884): Stored pid=329,
> state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure
> table=testSetQuotaAndThenDropTableWithDisable19
> 2018-05-18 18:08:58,430 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46443]
> master.MasterRpcServices(1141): Checking to see if procedure is done pid=329
> 2018-05-18 18:08:58,451 INFO [PEWorker-10]
> procedure2.ProcedureExecutor(1359): Rolled back pid=329, state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.TableNotFoundException via
> master-enable-table:org.apache.hadoop.hbase.TableNotFoundException:
> testSetQuotaAndThenDropTableWithDisable19; EnableTableProcedure
> table=testSetQuotaAndThenDropTableWithDisable19 exec-time=124msec
> 2018-05-18 18:08:58,533 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46443]
> master.MasterRpcServices(1141): Checking to see if procedure is done pid=329
> 2018-05-18 18:08:58,535 INFO [regionserver/ba4cba1aa13d:0.Chore.1]
> client.HBaseAdmin$TableFuture(3652): Operation: ENABLE, Table Name:
> default:testSetQuotaAndThenDropTableWithDisable19 failed with
> testSetQuotaAndThenDropTableWithDisable19{noformat}
> * Since, quota manager fails to enable table (i.e disable violation policy),
> it would not remove the policy, causing problems if table re-created
> {noformat}
> 2018-05-18 18:08:58,536 ERROR [regionserver/ba4cba1aa13d:0.Chore.1]
> quotas.RegionServerSpaceQuotaManager(210): Failed to disable space
[jira] [Updated] (HBASE-20597) Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-20597: --- Resolution: Fixed Status: Resolved (was: Patch Available) > Use a lock to serialize access to a shared reference to ZooKeeperWatcher in > HBaseReplicationEndpoint > > > Key: HBASE-20597 > URL: https://issues.apache.org/jira/browse/HBASE-20597 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.3.2, 1.4.4 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 2.0.1, 1.4.5 > > Attachments: HBASE-20597-branch-1.addendum-v2.0.patch, > HBASE-20597-branch-1.patch, HBASE-20597.addendum.0.patch, HBASE-20597.patch > > > The code that closes down a ZKW that fails to initialize when attempting to > connect to the remote cluster is not MT safe and can in theory leak > ZooKeeperWatcher instances. The allocation of a new ZKW and store to the > reference is not atomic. Might have concurrent allocations with only one > winning store, leading to leaked ZKW instances. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20594) provide utility to compare old and new descriptors
[
https://issues.apache.org/jira/browse/HBASE-20594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494048#comment-16494048
]
Sean Busbey commented on HBASE-20594:
-
+1 on v7
{code}
+
+// Equality here relies on implementation detail of the returned Set being
a TreeSet
{code}
nit: too in the details, but I think "Set that compares content of byte arrays
for element equality" would be more accurate. We don't need TreeSet internally,
but lots of stuff will break if we use a set that compares eauality of byte
arrays by instance rather than their contents.
> provide utility to compare old and new descriptors
> --
>
> Key: HBASE-20594
> URL: https://issues.apache.org/jira/browse/HBASE-20594
> Project: HBase
> Issue Type: Improvement
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Major
> Attachments: HBASE-20594.patch, HBASE-20594.v2.patch,
> HBASE-20594.v3.patch, HBASE-20594.v4.patch, HBASE-20594.v5.patch,
> HBASE-20594.v6.patch, HBASE-20594.v7.patch
>
>
> HBASE-20567 gives us hooks that give both the old and new descriptor in
> pre/postModify* events, but comparing them is still cumbersome. We should
> provide users some kind of utility for this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20594) provide utility to compare old and new descriptors
[ https://issues.apache.org/jira/browse/HBASE-20594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494032#comment-16494032 ] Sean Busbey commented on HBASE-20594: - Reviewing now; sorry missed the notification. > provide utility to compare old and new descriptors > -- > > Key: HBASE-20594 > URL: https://issues.apache.org/jira/browse/HBASE-20594 > Project: HBase > Issue Type: Improvement >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Attachments: HBASE-20594.patch, HBASE-20594.v2.patch, > HBASE-20594.v3.patch, HBASE-20594.v4.patch, HBASE-20594.v5.patch, > HBASE-20594.v6.patch, HBASE-20594.v7.patch > > > HBASE-20567 gives us hooks that give both the old and new descriptor in > pre/postModify* events, but comparing them is still cumbersome. We should > provide users some kind of utility for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20605) Exclude new Azure Storage FileSystem from SecureBulkLoadEndpoint permission check
[
https://issues.apache.org/jira/browse/HBASE-20605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494031#comment-16494031
]
Josh Elser commented on HBASE-20605:
{quote}Maybe something to do on init of the FileSystem overlay we do, rather
than during a permission check.
{quote}
I was thinking that I'd actually prefer adding somethign to the documentation
instead of adding another code-based check. Do you think there's still value in
a WARN/INFO?
> Exclude new Azure Storage FileSystem from SecureBulkLoadEndpoint permission
> check
> -
>
> Key: HBASE-20605
> URL: https://issues.apache.org/jira/browse/HBASE-20605
> Project: HBase
> Issue Type: Improvement
> Components: security
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 1.5.0, 1.3.3, 1.4.5
>
> Attachments: HBASE-20605.001.branch-1.patch,
> HBASE-20605.002.branch-1.patch
>
>
> Some folks in Hadoop are working on landing a new FileSystem from the Azure
> team: HADOOP-15407
> At present, this FileSystem doesn't support permissions which causes the
> SecureBulkLoadEndpoint to balk because it the staging directory doesn't have
> the proper 711 permissions.
> We have a static list of FileSystem schemes which we ignore this check on. I
> have a patch on an HBase 1.1ish which:
> # Adds the new FileSystem scheme
> # Makes this list configurable for the future
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20582) Bump up JRuby version because of some reported vulnerabilities
[
https://issues.apache.org/jira/browse/HBASE-20582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493989#comment-16493989
]
Sean Busbey commented on HBASE-20582:
-
restarted precommit job, since a failure in hbase-server means we don't run
hbase-shell tests.
> Bump up JRuby version because of some reported vulnerabilities
> --
>
> Key: HBASE-20582
> URL: https://issues.apache.org/jira/browse/HBASE-20582
> Project: HBase
> Issue Type: Bug
> Components: dependencies, shell
>Reporter: Ankit Singhal
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20582.002.patch, HBASE-20582.addendum.patch,
> HBASE-20582.patch
>
>
> There are some vulnerabilities reported with two of the libraries used in
> HBase.
> {code:java}
> Jruby(version:9.1.10.0):
> CVE-2009-5147
> CVE-2013-4363
> CVE-2014-4975
> CVE-2014-8080
> CVE-2014-8090
> CVE-2015-3900
> CVE-2015-7551
> CVE-2015-9096
> CVE-2017-0899
> CVE-2017-0900
> CVE-2017-0901
> CVE-2017-0902
> CVE-2017-0903
> CVE-2017-10784
> CVE-2017-14064
> CVE-2017-9224
> CVE-2017-9225
> CVE-2017-9226
> CVE-2017-9227
> CVE-2017-9228
> {code}
> Tool somehow able to relate the vulnerability of Ruby with JRuby(Java
> implementation). (Jackson will be handled in a different issue.)
> Not all of them directly affects HBase but [~elserj] suggested that it is
> better to be on the updated version to avoid issues during an audit in
> security sensitive organization.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20605) Exclude new Azure Storage FileSystem from SecureBulkLoadEndpoint permission check
[ https://issues.apache.org/jira/browse/HBASE-20605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493971#comment-16493971 ] Sean Busbey commented on HBASE-20605: - It would be good to have a check somewhere to warn about FS that won't behave as expected in master/branch-2 as well. Maybe something to do on init of the FileSystem overlay we do, rather than during a permission check. Also we could do this as a whitelist of things that we've tested to work as expected rather than a blacklist, since we'd just be doing a WARN / INFO message. > Exclude new Azure Storage FileSystem from SecureBulkLoadEndpoint permission > check > - > > Key: HBASE-20605 > URL: https://issues.apache.org/jira/browse/HBASE-20605 > Project: HBase > Issue Type: Improvement > Components: security >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 1.5.0, 1.3.3, 1.4.5 > > Attachments: HBASE-20605.001.branch-1.patch, > HBASE-20605.002.branch-1.patch > > > Some folks in Hadoop are working on landing a new FileSystem from the Azure > team: HADOOP-15407 > At present, this FileSystem doesn't support permissions which causes the > SecureBulkLoadEndpoint to balk because it the staging directory doesn't have > the proper 711 permissions. > We have a static list of FileSystem schemes which we ignore this check on. I > have a patch on an HBase 1.1ish which: > # Adds the new FileSystem scheme > # Makes this list configurable for the future -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20653) Add missing observer hooks for region server group to MasterObserver
[
https://issues.apache.org/jira/browse/HBASE-20653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493966#comment-16493966
]
Nihal Jain commented on HBASE-20653:
[^HBASE-20653.master.003.patch] fixes the following:
* {{checkstyle}}: removes unused imports
* adds missing {{@Override}} annotations
> Add missing observer hooks for region server group to MasterObserver
>
>
> Key: HBASE-20653
> URL: https://issues.apache.org/jira/browse/HBASE-20653
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20653.master.001.patch,
> HBASE-20653.master.002.patch, HBASE-20653.master.003.patch
>
>
> Currently the following region server group operations don't have
> corresponding hook in MasterObserver :
> * getRSGroupInfo
> * getRSGroupInfoOfServer
> * getRSGroupInfoOfTable
> * listRSGroup
> This JIRA is to
> * add them to MasterObserver
> * add corresponding permission check in AccessController
> * move the {{checkPermission}} out of RSGroupAdminEndpoint
> * add corresponding tests to TestRSGroupsWithACL
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20653) Add missing observer hooks for region server group to MasterObserver
[
https://issues.apache.org/jira/browse/HBASE-20653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain updated HBASE-20653:
---
Attachment: HBASE-20653.master.003.patch
> Add missing observer hooks for region server group to MasterObserver
>
>
> Key: HBASE-20653
> URL: https://issues.apache.org/jira/browse/HBASE-20653
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20653.master.001.patch,
> HBASE-20653.master.002.patch, HBASE-20653.master.003.patch
>
>
> Currently the following region server group operations don't have
> corresponding hook in MasterObserver :
> * getRSGroupInfo
> * getRSGroupInfoOfServer
> * getRSGroupInfoOfTable
> * listRSGroup
> This JIRA is to
> * add them to MasterObserver
> * add corresponding permission check in AccessController
> * move the {{checkPermission}} out of RSGroupAdminEndpoint
> * add corresponding tests to TestRSGroupsWithACL
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20582) Bump up JRuby version because of some reported vulnerabilities
[
https://issues.apache.org/jira/browse/HBASE-20582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493960#comment-16493960
]
Hadoop QA commented on HBASE-20582:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
18s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
23s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
13m 15s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}125m 23s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}174m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.locking.TestLockManager |
| | hadoop.hbase.replication.TestReplicationDroppedTables |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20582 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12925217/HBASE-20582.addendum.patch
|
| Optional Tests | asflicense javac javadoc unit shadedjars hadoopcheck
xml compile |
| uname | Linux 714d71c13aa2 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 42be553433 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/12997/artifact/patchprocess/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/12997/testReport/ |
| Max. process+thread count | 4836 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/12997/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Bump up JRuby version because of some reported vulnerabilities
> --
>
> Key: HBASE-20582
> URL: https://issues.apache.org/jira/browse/HBASE-20582
>
[jira] [Commented] (HBASE-20628) SegmentScanner does over-comparing when one flushing
[
https://issues.apache.org/jira/browse/HBASE-20628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493937#comment-16493937
]
stack commented on HBASE-20628:
---
I just tripped over an old, related [~ram_krish] issue where he suggests a
custom scanner for flushes... It can afford to be dumber is his suggestion:
HBASE-17355. This seems like natural progression from where we are here.
> SegmentScanner does over-comparing when one flushing
>
>
> Key: HBASE-20628
> URL: https://issues.apache.org/jira/browse/HBASE-20628
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: stack
>Priority: Critical
> Fix For: 2.0.1
>
> Attachments: HBASE-20628.branch-2.0.001 (1).patch,
> HBASE-20628.branch-2.0.001.patch, HBASE-20628.branch-2.0.001.patch,
> HBASE-20628.branch-2.0.002.patch, Screen Shot 2018-05-25 at 9.38.00 AM.png,
> hits-20628.png
>
>
> Flushing memstore is taking too long. It looks like we are doing a bunch of
> comparing out of a new facility in hbase2, the Segment scanner at flush time.
> Below is a patch from [~anoop.hbase]. I had a similar more hacky version.
> Both undo the extra comparing we were seeing in perf tests.
> [~anastas] and [~eshcar]. Need your help please.
> As I read it, we are trying to flush the memstore snapshot (default, no IMC
> case). There is only ever going to be one Segment involved (even if IMC is
> enabled); the snapshot Segment. But the getScanners is returning a list (of
> one) Scanners and the scan is via the generic SegmentScanner which is all
> about a bunch of stuff we don't need when doing a flush so it seems to do
> more work than is necessary. It also supports scanning backwards which is not
> needed when trying to flush memstore.
> Do you see a problem doing a version of Anoops patch (whether IMC or not)? It
> makes a big difference in general throughput when the below patch is in
> place. Thanks.
> {code}
> diff --git
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
>
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> index cbd60e5da3..c3dd972254 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> @@ -40,7 +40,8 @@ public class MemStoreSnapshot implements Closeable {
> this.cellsCount = snapshot.getCellsCount();
> this.memStoreSize = snapshot.getMemStoreSize();
> this.timeRangeTracker = snapshot.getTimeRangeTracker();
> -this.scanners = snapshot.getScanners(Long.MAX_VALUE, Long.MAX_VALUE);
> +//this.scanners = snapshot.getScanners(Long.MAX_VALUE, Long.MAX_VALUE);
> +this.scanners = snapshot.getScannersForSnapshot();
> this.tagsPresent = snapshot.isTagsPresent();
>}
> diff --git
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
>
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> index 70074bf3b4..279c4e50c8 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> @@ -33,6 +33,7 @@ import org.apache.hadoop.hbase.KeyValueUtil;
> import org.apache.hadoop.hbase.io.TimeRange;
> import org.apache.hadoop.hbase.util.Bytes;
> import org.apache.hadoop.hbase.util.ClassSize;
> +import org.apache.hadoop.hbase.util.CollectionBackedScanner;
> import org.apache.yetus.audience.InterfaceAudience;
> import org.slf4j.Logger;
> import
> org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
> @@ -130,6 +131,10 @@ public abstract class Segment {
> return Collections.singletonList(new SegmentScanner(this, readPoint,
> order));
>}
> + public List getScannersForSnapshot() {
> +return Collections.singletonList(new
> CollectionBackedScanner(this.cellSet.get(), comparator));
> + }
> +
>/**
> * @return whether the segment has any cells
> */
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20653) Add missing observer hooks for region server group to MasterObserver
[
https://issues.apache.org/jira/browse/HBASE-20653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493932#comment-16493932
]
Hadoop QA commented on HBASE-20653:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
53s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
1s{color} | {color:blue} hbase-server in master has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 33s{color}
| {color:red} hbase-rsgroup generated 8 new + 107 unchanged - 0 fixed = 115
total (was 107) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} hbase-rsgroup: The patch generated 3 new + 0 unchanged
- 0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
51s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 47s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}106m
26s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
23s{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}160m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20653 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/1292/HBASE-20653.master.002.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux f54e32f6d1ec 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master /
[jira] [Commented] (HBASE-20597) Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493921#comment-16493921 ] Andrew Purtell commented on HBASE-20597: Ah, we raced. Ok I will commit. Thanks [~busbey] > Use a lock to serialize access to a shared reference to ZooKeeperWatcher in > HBaseReplicationEndpoint > > > Key: HBASE-20597 > URL: https://issues.apache.org/jira/browse/HBASE-20597 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.3.2, 1.4.4 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 2.0.1, 1.4.5 > > Attachments: HBASE-20597-branch-1.addendum-v2.0.patch, > HBASE-20597-branch-1.patch, HBASE-20597.addendum.0.patch, HBASE-20597.patch > > > The code that closes down a ZKW that fails to initialize when attempting to > connect to the remote cluster is not MT safe and can in theory leak > ZooKeeperWatcher instances. The allocation of a new ZKW and store to the > reference is not atomic. Might have concurrent allocations with only one > winning store, leading to leaked ZKW instances. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20597) Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493918#comment-16493918 ] Andrew Purtell commented on HBASE-20597: Ok, I can't detect any issues with the change with local tests. Let me know what you think of the addendum (really: revision #2 ) > Use a lock to serialize access to a shared reference to ZooKeeperWatcher in > HBaseReplicationEndpoint > > > Key: HBASE-20597 > URL: https://issues.apache.org/jira/browse/HBASE-20597 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.3.2, 1.4.4 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 2.0.1, 1.4.5 > > Attachments: HBASE-20597-branch-1.addendum-v2.0.patch, > HBASE-20597-branch-1.patch, HBASE-20597.addendum.0.patch, HBASE-20597.patch > > > The code that closes down a ZKW that fails to initialize when attempting to > connect to the remote cluster is not MT safe and can in theory leak > ZooKeeperWatcher instances. The allocation of a new ZKW and store to the > reference is not atomic. Might have concurrent allocations with only one > winning store, leading to leaked ZKW instances. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-16228) Add read/write HDFS size metrics for flush/compact/handler
[ https://issues.apache.org/jira/browse/HBASE-16228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16228: -- Priority: Critical (was: Major) > Add read/write HDFS size metrics for flush/compact/handler > -- > > Key: HBASE-16228 > URL: https://issues.apache.org/jira/browse/HBASE-16228 > Project: HBase > Issue Type: New Feature >Reporter: binlijin >Priority: Critical > Attachments: HBASE-16228.patch > > > Flush/Compact/Handler and other threads read or write to HDFS, we can get > this metrics to see read/write amplification in hbase, and test how a > compaction algorithm will affect it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-16228) Add read/write HDFS size metrics for flush/compact/handler
[ https://issues.apache.org/jira/browse/HBASE-16228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493916#comment-16493916 ] stack commented on HBASE-16228: --- Wonderying why this stalled. It looks really useful. > Add read/write HDFS size metrics for flush/compact/handler > -- > > Key: HBASE-16228 > URL: https://issues.apache.org/jira/browse/HBASE-16228 > Project: HBase > Issue Type: New Feature >Reporter: binlijin >Priority: Critical > Attachments: HBASE-16228.patch > > > Flush/Compact/Handler and other threads read or write to HDFS, we can get > this metrics to see read/write amplification in hbase, and test how a > compaction algorithm will affect it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20597) Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493912#comment-16493912 ] Sean Busbey commented on HBASE-20597: - if tests pass locally for you [~apurtell] I'm +1 on your addendum. > Use a lock to serialize access to a shared reference to ZooKeeperWatcher in > HBaseReplicationEndpoint > > > Key: HBASE-20597 > URL: https://issues.apache.org/jira/browse/HBASE-20597 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.3.2, 1.4.4 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 2.0.1, 1.4.5 > > Attachments: HBASE-20597-branch-1.addendum-v2.0.patch, > HBASE-20597-branch-1.patch, HBASE-20597.addendum.0.patch, HBASE-20597.patch > > > The code that closes down a ZKW that fails to initialize when attempting to > connect to the remote cluster is not MT safe and can in theory leak > ZooKeeperWatcher instances. The allocation of a new ZKW and store to the > reference is not atomic. Might have concurrent allocations with only one > winning store, leading to leaked ZKW instances. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19761) Fix Checkstyle errors in hbase-zookeeper
[
https://issues.apache.org/jira/browse/HBASE-19761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493887#comment-16493887
]
Hadoop QA commented on HBASE-19761:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 20 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
24s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
15s{color} | {color:blue} hbase-server in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
38s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} hbase-zookeeper: The patch generated 0 new + 0
unchanged - 13 fixed = 0 total (was 13) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch hbase-replication passed checkstyle
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
37s{color} | {color:green} hbase-server: The patch generated 0 new + 478
unchanged - 4 fixed = 478 total (was 482) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch hbase-rsgroup passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} The patch hbase-it passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
30s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
17m 3s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
48s{color} | {color:green} hbase-zookeeper in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
24s{color} | {color:green} hbase-replication in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}112m 9s{color}
| {color:red} hbase-server in the patch failed. {color} |
|
[jira] [Commented] (HBASE-20390) IMC Default Parameters for 2.0.0
[
https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493877#comment-16493877
]
Mike Drob commented on HBASE-20390:
---
Thanks for updating the docs describing the new defaults, [~eshcar]! One
question about this paragraph -
{quote}
When an in-memory flush happens is calculated by dividing the configured region
flush size (Set in the table descriptor or read from
hbase.hregion.memstore.flush.size) by the number of column families and then
multiplying by hbase.memstore.inmemoryflush.threshold.factor (default is 0.014
- a little less than the size of a chunk).
{quote}
I don't see a 'chunk' defined anywhere, can we clarify what that is for readers
who may be unfamiliar?
> IMC Default Parameters for 2.0.0
>
>
> Key: HBASE-20390
> URL: https://issues.apache.org/jira/browse/HBASE-20390
> Project: HBase
> Issue Type: Sub-task
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
>Priority: Major
> Attachments: HBASE-20390-branch-2.0-01.patch,
> HBASE-20390-branch-2.0-01.patch, HBASE-20390.branch-2.0.002.patch,
> HBASE-20390.branch-2.0.003.patch, HBase 2.0 performance evaluation -
> throughput SSD_HDD.pdf, hits.ihc.png
>
>
> Setting new default parameters for in-memory compaction based on performance
> tests done in HBASE-20188
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20390) IMC Default Parameters for 2.0.0
[
https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493852#comment-16493852
]
Hudson commented on HBASE-20390:
Results for branch branch-2.0
[build #362 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/362/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/362//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/362//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/362//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> IMC Default Parameters for 2.0.0
>
>
> Key: HBASE-20390
> URL: https://issues.apache.org/jira/browse/HBASE-20390
> Project: HBase
> Issue Type: Sub-task
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
>Priority: Major
> Attachments: HBASE-20390-branch-2.0-01.patch,
> HBASE-20390-branch-2.0-01.patch, HBASE-20390.branch-2.0.002.patch,
> HBASE-20390.branch-2.0.003.patch, HBase 2.0 performance evaluation -
> throughput SSD_HDD.pdf, hits.ihc.png
>
>
> Setting new default parameters for in-memory compaction based on performance
> tests done in HBASE-20188
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20597) Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493845#comment-16493845 ] Sean Busbey commented on HBASE-20597: - bookeeper jobs were stomping on system resources last week, though I don't know if that's specifically what caused process killing here. > Use a lock to serialize access to a shared reference to ZooKeeperWatcher in > HBaseReplicationEndpoint > > > Key: HBASE-20597 > URL: https://issues.apache.org/jira/browse/HBASE-20597 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.3.2, 1.4.4 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 2.0.1, 1.4.5 > > Attachments: HBASE-20597-branch-1.addendum-v2.0.patch, > HBASE-20597-branch-1.patch, HBASE-20597.addendum.0.patch, HBASE-20597.patch > > > The code that closes down a ZKW that fails to initialize when attempting to > connect to the remote cluster is not MT safe and can in theory leak > ZooKeeperWatcher instances. The allocation of a new ZKW and store to the > reference is not atomic. Might have concurrent allocations with only one > winning store, leading to leaked ZKW instances. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20636) Introduce two bloom filter type : ROWPREFIX and ROWPREFIX_DELIMITED
[ https://issues.apache.org/jira/browse/HBASE-20636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493835#comment-16493835 ] Andrew Purtell commented on HBASE-20636: bq. I also think that It is necessary to keep compatibility. But, even if the HFile has a version increment, the old version of HBase still unable to read the HFile. Yes, this is why I said there needs to be a fallback so the read can proceed even when the BF type is not recognized. At we need to at least increment the minor HFile version. I think that would be enough. > Introduce two bloom filter type : ROWPREFIX and ROWPREFIX_DELIMITED > --- > > Key: HBASE-20636 > URL: https://issues.apache.org/jira/browse/HBASE-20636 > Project: HBase > Issue Type: New Feature > Components: HFile, regionserver, scan >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Attachments: HBASE-20636.master.001.patch, > HBASE-20636.master.002.patch, HBASE-20636.master.003.patch > > > As we all know, HBase uses BloomFilter(ROW and ROWCOL) to filter unnecessary > files to improve read performance. But they only support Get and do not > support Scan. > In our company(Tencent), many users need to scan all rows with the same > prefix, such as Tencent Game. Game user's some operational record will be > written into HBase, each game user will have a lot of records, the rowkey is > constructed as userid+'#'+timestamps. So we can scan all records for a given > user for a specified period. > For this scenario, we designed the prefix Bloom filter. If the startRow and > stopRow of the Scan has a valid common prefix, the scan will be allowed to > use BloomFilter to filter files which will enhance the performance of the > scan. > Now, this feature has been running on our cluster over a year, and scan > performance for this scenario has been improved by more than one times than > before. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20597) Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493832#comment-16493832 ] Andrew Purtell commented on HBASE-20597: A number of processes were killed while running the precommit, this is not related to this patch. > Use a lock to serialize access to a shared reference to ZooKeeperWatcher in > HBaseReplicationEndpoint > > > Key: HBASE-20597 > URL: https://issues.apache.org/jira/browse/HBASE-20597 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.3.2, 1.4.4 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 3.0.0, 2.1.0, 1.5.0, 1.3.3, 2.0.1, 1.4.5 > > Attachments: HBASE-20597-branch-1.addendum-v2.0.patch, > HBASE-20597-branch-1.patch, HBASE-20597.addendum.0.patch, HBASE-20597.patch > > > The code that closes down a ZKW that fails to initialize when attempting to > connect to the remote cluster is not MT safe and can in theory leak > ZooKeeperWatcher instances. The allocation of a new ZKW and store to the > reference is not atomic. Might have concurrent allocations with only one > winning store, leading to leaked ZKW instances. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20633) Dropping a table containing a disable violation policy fails to remove the quota upon table delete
[
https://issues.apache.org/jira/browse/HBASE-20633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-20633:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.0.1
2.1.0
Status: Resolved (was: Patch Available)
> Dropping a table containing a disable violation policy fails to remove the
> quota upon table delete
> --
>
> Key: HBASE-20633
> URL: https://issues.apache.org/jira/browse/HBASE-20633
> Project: HBase
> Issue Type: Bug
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20633.master.001.patch,
> HBASE-20633.master.002.patch, HBASE-20633.master.002.patch
>
>
>
> {code:java}
> private void setQuotaAndThenDropTable(SpaceViolationPolicy policy) throws
> Exception {
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> SpaceViolationPolicy policy = SpaceViolationPolicy.DISABLE;
> // Do puts until we violate space policy
> final TableName tn = writeUntilViolationAndVerifyViolation(policy, put);
> // Now, drop the table
> TEST_UTIL.deleteTable(tn);
> LOG.debug("Successfully deleted table ", tn);
> // Now re-create the table
> TEST_UTIL.createTable(tn, Bytes.toBytes(SpaceQuotaHelperForTests.F1));
> LOG.debug("Successfully re-created table ", tn);
> // Put some rows now: should not violate as table/quota was dropped
> verifyNoViolation(policy, tn, put);
> }
> {code}
> * When we drop a table, upon completion the quota triggers removal of
> disable policy, thus causing the system to enable the table
> {noformat}
> 2018-05-18 18:08:58,189 DEBUG [PEWorker-13]
> procedure.DeleteTableProcedure(130): delete
> 'testSetQuotaAndThenDropTableWithDisable19' completed
> 2018-05-18 18:08:58,191 INFO [PEWorker-13]
> procedure2.ProcedureExecutor(1265): Finished pid=328, state=SUCCESS;
> DeleteTableProcedure table=testSetQuotaAndThenDropTableWithDisable19 in
> 271msec
> 2018-05-18 18:08:58,321 INFO [regionserver/ba4cba1aa13d:0.Chore.1]
> client.HBaseAdmin$14(844): Started enable of
> testSetQuotaAndThenDropTableWithDisable19{noformat}
> * But, since the table has already been dropped, enable procedure would
> rollback
> {noformat}
> 2018-05-18 18:08:58,427 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46443]
> procedure2.ProcedureExecutor(884): Stored pid=329,
> state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure
> table=testSetQuotaAndThenDropTableWithDisable19
> 2018-05-18 18:08:58,430 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46443]
> master.MasterRpcServices(1141): Checking to see if procedure is done pid=329
> 2018-05-18 18:08:58,451 INFO [PEWorker-10]
> procedure2.ProcedureExecutor(1359): Rolled back pid=329, state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.TableNotFoundException via
> master-enable-table:org.apache.hadoop.hbase.TableNotFoundException:
> testSetQuotaAndThenDropTableWithDisable19; EnableTableProcedure
> table=testSetQuotaAndThenDropTableWithDisable19 exec-time=124msec
> 2018-05-18 18:08:58,533 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46443]
> master.MasterRpcServices(1141): Checking to see if procedure is done pid=329
> 2018-05-18 18:08:58,535 INFO [regionserver/ba4cba1aa13d:0.Chore.1]
> client.HBaseAdmin$TableFuture(3652): Operation: ENABLE, Table Name:
> default:testSetQuotaAndThenDropTableWithDisable19 failed with
> testSetQuotaAndThenDropTableWithDisable19{noformat}
> * Since, quota manager fails to enable table (i.e disable violation policy),
> it would not remove the policy, causing problems if table re-created
> {noformat}
> 2018-05-18 18:08:58,536 ERROR [regionserver/ba4cba1aa13d:0.Chore.1]
> quotas.RegionServerSpaceQuotaManager(210): Failed to disable space violation
> policy for testSetQuotaAndThenDropTableWithDisable19. This table will remain
> in violation.
> org.apache.hadoop.hbase.TableNotFoundException:
> testSetQuotaAndThenDropTableWithDisable19
> at
> org.apache.hadoop.hbase.master.procedure.EnableTableProcedure.prepareEnable(EnableTableProcedure.java:323)
> at
> org.apache.hadoop.hbase.master.procedure.EnableTableProcedure.executeFromState(EnableTableProcedure.java:98)
> at
> org.apache.hadoop.hbase.master.procedure.EnableTableProcedure.executeFromState(EnableTableProcedure.java:49)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:184)
> at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:850)
> at
>
[jira] [Updated] (HBASE-20645) Fix security_available method in security.rb
[
https://issues.apache.org/jira/browse/HBASE-20645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-20645:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Pushed to 2.0+
Thanks for the patch, Ankit!
> Fix security_available method in security.rb
> -
>
> Key: HBASE-20645
> URL: https://issues.apache.org/jira/browse/HBASE-20645
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20645.patch
>
>
> "exists?" method expects parameter tableName to be String but ACL_TABLE_NAME
> is of org.apache.hadoop.hbase.TableName form.
> {code}
> raise(ArgumentError, 'DISABLED: Security features are not available') unless \
>
> exists?(org.apache.hadoop.hbase.security.access.AccessControlLists::ACL_TABLE_NAME.getNameAsString)
> {code}
> Impact of the bug:-
> So , if a user is running any security related
> command(revoke,user_permission) and there is an exception(MasterNotRunning)
> while checking security capabilities, then instead of seeing the underlying
> exception, user is seeing
> {code}
> ERROR: no method 'valueOf' for arguments (org.apache.hadoop.hbase.TableName)
> on Java::OrgApacheHadoopHbase::TableName
> available overloads:
> (java.lang.String)
> (byte[])
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20390) IMC Default Parameters for 2.0.0
[ https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493739#comment-16493739 ] Eshcar Hillel commented on HBASE-20390: --- This is still the report following the first addendum, waiting for the report after the second addendum > IMC Default Parameters for 2.0.0 > > > Key: HBASE-20390 > URL: https://issues.apache.org/jira/browse/HBASE-20390 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20390-branch-2.0-01.patch, > HBASE-20390-branch-2.0-01.patch, HBASE-20390.branch-2.0.002.patch, > HBASE-20390.branch-2.0.003.patch, HBase 2.0 performance evaluation - > throughput SSD_HDD.pdf, hits.ihc.png > > > Setting new default parameters for in-memory compaction based on performance > tests done in HBASE-20188 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20390) IMC Default Parameters for 2.0.0
[
https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493728#comment-16493728
]
Hudson commented on HBASE-20390:
Results for branch branch-2
[build #797 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/797/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/797//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/797//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/797//JDK8_Nightly_Build_Report_(Hadoop3)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> IMC Default Parameters for 2.0.0
>
>
> Key: HBASE-20390
> URL: https://issues.apache.org/jira/browse/HBASE-20390
> Project: HBase
> Issue Type: Sub-task
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
>Priority: Major
> Attachments: HBASE-20390-branch-2.0-01.patch,
> HBASE-20390-branch-2.0-01.patch, HBASE-20390.branch-2.0.002.patch,
> HBASE-20390.branch-2.0.003.patch, HBase 2.0 performance evaluation -
> throughput SSD_HDD.pdf, hits.ihc.png
>
>
> Setting new default parameters for in-memory compaction based on performance
> tests done in HBASE-20188
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19724) Fix Checkstyle errors in hbase-hadoop2-compat
[
https://issues.apache.org/jira/browse/HBASE-19724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493727#comment-16493727
]
Hadoop QA commented on HBASE-19724:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HBASE-19724 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.7.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-19724 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904976/HBASE-19724.master.001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/12998/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Fix Checkstyle errors in hbase-hadoop2-compat
> -
>
> Key: HBASE-19724
> URL: https://issues.apache.org/jira/browse/HBASE-19724
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jan Hentschel
>Assignee: Jan Hentschel
>Priority: Minor
> Attachments: HBASE-19724.master.001.patch
>
>
> Fix the remaining Checkstyle errors in the *hbase-hadoop2-compat* module and
> enable Checkstyle to fail on violations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19724) Fix Checkstyle errors in hbase-hadoop2-compat
[ https://issues.apache.org/jira/browse/HBASE-19724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493719#comment-16493719 ] Mike Drob commented on HBASE-19724: --- I just pushed an update from checkstyle 6 to checkstyle 8 to master branch. I don't think this will complicate the work here, but wanted to give folks a heads up. > Fix Checkstyle errors in hbase-hadoop2-compat > - > > Key: HBASE-19724 > URL: https://issues.apache.org/jira/browse/HBASE-19724 > Project: HBase > Issue Type: Sub-task >Reporter: Jan Hentschel >Assignee: Jan Hentschel >Priority: Minor > Attachments: HBASE-19724.master.001.patch > > > Fix the remaining Checkstyle errors in the *hbase-hadoop2-compat* module and > enable Checkstyle to fail on violations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19761) Fix Checkstyle errors in hbase-zookeeper
[ https://issues.apache.org/jira/browse/HBASE-19761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493718#comment-16493718 ] Mike Drob commented on HBASE-19761: --- I just pushed an update from checkstyle 6 to checkstyle 8 to master branch. I don't think this will complicate the work here, but wanted to give folks a heads up. > Fix Checkstyle errors in hbase-zookeeper > > > Key: HBASE-19761 > URL: https://issues.apache.org/jira/browse/HBASE-19761 > Project: HBase > Issue Type: Sub-task >Reporter: Jan Hentschel >Assignee: maoling >Priority: Minor > Attachments: HBASE-19761-master-v0.patch, > HBASE-19761-master-v1.patch, HBASE-19761-master-v2.patch, > HBASE-19761-master-v3.patch, HBASE-19761.master.001.patch > > > Fix the remaining Checkstyle errors in the *hbase-zookeeper* module and > enable Checkstyle to fail on violations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20636) Introduce two bloom filter type : ROWPREFIX and ROWPREFIX_DELIMITED
[
https://issues.apache.org/jira/browse/HBASE-20636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493714#comment-16493714
]
Hadoop QA commented on HBASE-20636:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
54s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
3s{color} | {color:blue} hbase-server in master has 2 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
14s{color} | {color:red} hbase-server: The patch generated 1 new + 235
unchanged - 9 fixed = 236 total (was 244) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
49s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 26s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
52s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}118m 58s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
49s{color} | {color:green} hbase-mapreduce in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
55s{color} | {color:green} hbase-it in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}194m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.TestMasterFailoverBalancerPersistence |
| |
hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpointNoMaster
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20636 |
| JIRA Patch URL |
[jira] [Commented] (HBASE-20633) Dropping a table containing a disable violation policy fails to remove the quota upon table delete
[
https://issues.apache.org/jira/browse/HBASE-20633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493711#comment-16493711
]
Josh Elser commented on HBASE-20633:
Ya, another good one, [~nihaljain.cs]!
Committing.
> Dropping a table containing a disable violation policy fails to remove the
> quota upon table delete
> --
>
> Key: HBASE-20633
> URL: https://issues.apache.org/jira/browse/HBASE-20633
> Project: HBase
> Issue Type: Bug
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20633.master.001.patch,
> HBASE-20633.master.002.patch, HBASE-20633.master.002.patch
>
>
>
> {code:java}
> private void setQuotaAndThenDropTable(SpaceViolationPolicy policy) throws
> Exception {
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> SpaceViolationPolicy policy = SpaceViolationPolicy.DISABLE;
> // Do puts until we violate space policy
> final TableName tn = writeUntilViolationAndVerifyViolation(policy, put);
> // Now, drop the table
> TEST_UTIL.deleteTable(tn);
> LOG.debug("Successfully deleted table ", tn);
> // Now re-create the table
> TEST_UTIL.createTable(tn, Bytes.toBytes(SpaceQuotaHelperForTests.F1));
> LOG.debug("Successfully re-created table ", tn);
> // Put some rows now: should not violate as table/quota was dropped
> verifyNoViolation(policy, tn, put);
> }
> {code}
> * When we drop a table, upon completion the quota triggers removal of
> disable policy, thus causing the system to enable the table
> {noformat}
> 2018-05-18 18:08:58,189 DEBUG [PEWorker-13]
> procedure.DeleteTableProcedure(130): delete
> 'testSetQuotaAndThenDropTableWithDisable19' completed
> 2018-05-18 18:08:58,191 INFO [PEWorker-13]
> procedure2.ProcedureExecutor(1265): Finished pid=328, state=SUCCESS;
> DeleteTableProcedure table=testSetQuotaAndThenDropTableWithDisable19 in
> 271msec
> 2018-05-18 18:08:58,321 INFO [regionserver/ba4cba1aa13d:0.Chore.1]
> client.HBaseAdmin$14(844): Started enable of
> testSetQuotaAndThenDropTableWithDisable19{noformat}
> * But, since the table has already been dropped, enable procedure would
> rollback
> {noformat}
> 2018-05-18 18:08:58,427 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46443]
> procedure2.ProcedureExecutor(884): Stored pid=329,
> state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure
> table=testSetQuotaAndThenDropTableWithDisable19
> 2018-05-18 18:08:58,430 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46443]
> master.MasterRpcServices(1141): Checking to see if procedure is done pid=329
> 2018-05-18 18:08:58,451 INFO [PEWorker-10]
> procedure2.ProcedureExecutor(1359): Rolled back pid=329, state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.TableNotFoundException via
> master-enable-table:org.apache.hadoop.hbase.TableNotFoundException:
> testSetQuotaAndThenDropTableWithDisable19; EnableTableProcedure
> table=testSetQuotaAndThenDropTableWithDisable19 exec-time=124msec
> 2018-05-18 18:08:58,533 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46443]
> master.MasterRpcServices(1141): Checking to see if procedure is done pid=329
> 2018-05-18 18:08:58,535 INFO [regionserver/ba4cba1aa13d:0.Chore.1]
> client.HBaseAdmin$TableFuture(3652): Operation: ENABLE, Table Name:
> default:testSetQuotaAndThenDropTableWithDisable19 failed with
> testSetQuotaAndThenDropTableWithDisable19{noformat}
> * Since, quota manager fails to enable table (i.e disable violation policy),
> it would not remove the policy, causing problems if table re-created
> {noformat}
> 2018-05-18 18:08:58,536 ERROR [regionserver/ba4cba1aa13d:0.Chore.1]
> quotas.RegionServerSpaceQuotaManager(210): Failed to disable space violation
> policy for testSetQuotaAndThenDropTableWithDisable19. This table will remain
> in violation.
> org.apache.hadoop.hbase.TableNotFoundException:
> testSetQuotaAndThenDropTableWithDisable19
> at
> org.apache.hadoop.hbase.master.procedure.EnableTableProcedure.prepareEnable(EnableTableProcedure.java:323)
> at
> org.apache.hadoop.hbase.master.procedure.EnableTableProcedure.executeFromState(EnableTableProcedure.java:98)
> at
> org.apache.hadoop.hbase.master.procedure.EnableTableProcedure.executeFromState(EnableTableProcedure.java:49)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:184)
> at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:850)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1472)
> at
>
[jira] [Commented] (HBASE-19761) Fix Checkstyle errors in hbase-zookeeper
[
https://issues.apache.org/jira/browse/HBASE-19761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493693#comment-16493693
]
Jan Hentschel commented on HBASE-19761:
---
Thanks [~maoling]. When running it locally, Checkstyle complains about the
following things:
{code}
[INFO] --- maven-checkstyle-plugin:3.0.0:check (checkstyle) @ hbase-zookeeper
---
[INFO] There are 49 errors reported by Checkstyle 8.2 with hbase/checkstyle.xml
ruleset.
[ERROR]
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKNodeTracker.java:[105]
(indentation) Indentation: 'throws' has incorrect indentation level 2, expected
level should be 4.
[ERROR]
src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterStatusTracker.java:[68]
(indentation) Indentation: 'throws' has incorrect indentation level 2, expected
level should be 4.
[ERROR]
src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterStatusTracker.java:[82]
(indentation) Indentation: 'throws' has incorrect indentation level 2, expected
level should be 4.
[ERROR]
src/main/java/org/apache/hadoop/hbase/zookeeper/LoadBalancerTracker.java:[85]
(indentation) Indentation: 'throws' has incorrect indentation level 2, expected
level should be 4.
[ERROR] src/main/java/org/apache/hadoop/hbase/zookeeper/ZKMainServer.java:[49]
(indentation) Indentation: 'throws' has incorrect indentation level 4, expected
level should be 6.
[ERROR]
src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java:[88]
(indentation) Indentation: 'throws' has incorrect indentation level 2, expected
level should be 4.
[ERROR]
src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java:[97]
(indentation) Indentation: 'throws' has incorrect indentation level 2, expected
level should be 4.
[ERROR]
src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java:[303]
(indentation) Indentation: 'throws' has incorrect indentation level 2, expected
level should be 4.
[ERROR]
src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java:[334]
(indentation) Indentation: 'throws' has incorrect indentation level 2, expected
level should be 4.
[ERROR]
src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java:[365]
(indentation) Indentation: 'throws' has incorrect indentation level 2, expected
level should be 4.
[ERROR]
src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java:[398]
(indentation) Indentation: 'throws' has incorrect indentation level 2, expected
level should be 4.
[ERROR]
src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java:[448]
(indentation) Indentation: 'throws' has incorrect indentation level 2, expected
level should be 4.
[ERROR]
src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java:[479]
(indentation) Indentation: 'throws' has incorrect indentation level 2, expected
level should be 4.
[ERROR]
src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java:[522]
(indentation) Indentation: 'throws' has incorrect indentation level 2, expected
level should be 4.
[ERROR]
src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java:[590]
(indentation) Indentation: 'throws' has incorrect indentation level 2, expected
level should be 4.
[ERROR]
src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java:[656]
(indentation) Indentation: 'throws' has incorrect indentation level 2, expected
level should be 4.
[ERROR] src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java:[64]
(indentation) Indentation: 'throws' has incorrect indentation level 2, expected
level should be 4.
[ERROR]
src/main/java/org/apache/hadoop/hbase/zookeeper/MasterAddressTracker.java:[211]
(indentation) Indentation: 'throws' has incorrect indentation level 2, expected
level should be 4.
[ERROR] src/main/java/org/apache/hadoop/hbase/zookeeper/ZKWatcher.java:[118]
(indentation) Indentation: 'throws' has incorrect indentation level 2, expected
level should be 4.
[ERROR] src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java:[111]
(indentation) Indentation: 'throws' has incorrect indentation level 2, expected
level should be 4.
[ERROR] src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java:[118]
(indentation) Indentation: 'throws' has incorrect indentation level 2, expected
level should be 4.
[ERROR] src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java:[124]
(indentation) Indentation: 'throws' has incorrect indentation level 2, expected
level should be 4.
[ERROR] src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java:[350]
(indentation) Indentation: 'throws' has incorrect indentation level 2, expected
level should be 4.
[ERROR] src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java:[404]
(indentation) Indentation: 'throws' has incorrect indentation level 2, expected
level should be 4.
[ERROR]
[jira] [Commented] (HBASE-20478) move import checks from hbaseanti to checkstyle
[ https://issues.apache.org/jira/browse/HBASE-20478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493692#comment-16493692 ] Sean Busbey commented on HBASE-20478: - lol. I dunno, I can see how that sounds painful. I'm willing to let it sit for now and pick things up myself if it becomes a problem on the earlier branches, presuming no one else objects. > move import checks from hbaseanti to checkstyle > --- > > Key: HBASE-20478 > URL: https://issues.apache.org/jira/browse/HBASE-20478 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Sean Busbey >Assignee: Mike Drob >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-20478.0.patch, HBASE-20478.1.patch, > HBASE-20478.2.patch, HBASE-20478.3.patch, HBASE-20478.4.patch, > HBASE-20478.5.patch, HBASE-20478.6.patch, HBASE-20478.8.patch, > HBASE-20478.WIP.2.patch, HBASE-20478.WIP.2.patch, HBASE-20478.WIP.patch, > HBASE-anti-check.patch > > > came up in discussion on HBASE-20332. our check of "don't do this" things in > the codebase doesn't log the specifics of complaints anywhere, which forces > those who want to follow up to reverse engineer the check. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20390) IMC Default Parameters for 2.0.0
[ https://issues.apache.org/jira/browse/HBASE-20390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493691#comment-16493691 ] stack commented on HBASE-20390: --- Thanks for being on top of it [~eshcar] > IMC Default Parameters for 2.0.0 > > > Key: HBASE-20390 > URL: https://issues.apache.org/jira/browse/HBASE-20390 > Project: HBase > Issue Type: Sub-task >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel >Priority: Major > Attachments: HBASE-20390-branch-2.0-01.patch, > HBASE-20390-branch-2.0-01.patch, HBASE-20390.branch-2.0.002.patch, > HBASE-20390.branch-2.0.003.patch, HBase 2.0 performance evaluation - > throughput SSD_HDD.pdf, hits.ihc.png > > > Setting new default parameters for in-memory compaction based on performance > tests done in HBASE-20188 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20637) Polish the WAL switching when transiting from A to S
[
https://issues.apache.org/jira/browse/HBASE-20637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493690#comment-16493690
]
Hadoop QA commented on HBASE-20637:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} HBASE-19064 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
42s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
12s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
49s{color} | {color:blue} hbase-server in HBASE-19064 has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} HBASE-19064 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} hbase-server: The patch generated 0 new + 19
unchanged - 2 fixed = 19 total (was 21) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
25s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
14m 4s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}116m 1s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}158m 46s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.replication.TestSyncReplicationMoreLogsInLocalGiveUpSplitting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:d8b550f |
| JIRA Issue | HBASE-20637 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12925540/HBASE-20637-HBASE-19064-v3.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 5bc918814278 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | HBASE-19064 / 5999f5a7f7 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/12993/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Commented] (HBASE-20478) move import checks from hbaseanti to checkstyle
[ https://issues.apache.org/jira/browse/HBASE-20478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493689#comment-16493689 ] Mike Drob commented on HBASE-20478: --- [~busbey] - I have no desire to pull this back to any other branches, lmk how strongly you disagree with that (I'm presuming that you do disagree) > move import checks from hbaseanti to checkstyle > --- > > Key: HBASE-20478 > URL: https://issues.apache.org/jira/browse/HBASE-20478 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Sean Busbey >Assignee: Mike Drob >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-20478.0.patch, HBASE-20478.1.patch, > HBASE-20478.2.patch, HBASE-20478.3.patch, HBASE-20478.4.patch, > HBASE-20478.5.patch, HBASE-20478.6.patch, HBASE-20478.8.patch, > HBASE-20478.WIP.2.patch, HBASE-20478.WIP.2.patch, HBASE-20478.WIP.patch, > HBASE-anti-check.patch > > > came up in discussion on HBASE-20332. our check of "don't do this" things in > the codebase doesn't log the specifics of complaints anywhere, which forces > those who want to follow up to reverse engineer the check. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20478) move import checks from hbaseanti to checkstyle
[ https://issues.apache.org/jira/browse/HBASE-20478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-20478: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) > move import checks from hbaseanti to checkstyle > --- > > Key: HBASE-20478 > URL: https://issues.apache.org/jira/browse/HBASE-20478 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Sean Busbey >Assignee: Mike Drob >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-20478.0.patch, HBASE-20478.1.patch, > HBASE-20478.2.patch, HBASE-20478.3.patch, HBASE-20478.4.patch, > HBASE-20478.5.patch, HBASE-20478.6.patch, HBASE-20478.8.patch, > HBASE-20478.WIP.2.patch, HBASE-20478.WIP.2.patch, HBASE-20478.WIP.patch, > HBASE-anti-check.patch > > > came up in discussion on HBASE-20332. our check of "don't do this" things in > the codebase doesn't log the specifics of complaints anywhere, which forces > those who want to follow up to reverse engineer the check. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20478) move import checks from hbaseanti to checkstyle
[ https://issues.apache.org/jira/browse/HBASE-20478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493681#comment-16493681 ] Mike Drob commented on HBASE-20478: --- I restored the comment and added it to the other applicable methods in the class as well before pushing. > move import checks from hbaseanti to checkstyle > --- > > Key: HBASE-20478 > URL: https://issues.apache.org/jira/browse/HBASE-20478 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Sean Busbey >Assignee: Mike Drob >Priority: Minor > Attachments: HBASE-20478.0.patch, HBASE-20478.1.patch, > HBASE-20478.2.patch, HBASE-20478.3.patch, HBASE-20478.4.patch, > HBASE-20478.5.patch, HBASE-20478.6.patch, HBASE-20478.8.patch, > HBASE-20478.WIP.2.patch, HBASE-20478.WIP.2.patch, HBASE-20478.WIP.patch, > HBASE-anti-check.patch > > > came up in discussion on HBASE-20332. our check of "don't do this" things in > the codebase doesn't log the specifics of complaints anywhere, which forces > those who want to follow up to reverse engineer the check. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493665#comment-16493665
]
Ted Yu edited comment on HBASE-20654 at 5/29/18 3:08 PM:
-
{code}
+
metricsRecordBuilder.tag(Interns.info(REGIONS_STATE_FAILED_OPEN,REGIONS_STATE_FAILED_OPEN_DESC),metricsAssignmentWrapper.getFailedOpenRegions());
{code}
Please limit line length to 100 characters.
{code}
+//assignmentManagerSource = CompatibilitySingletonFactory.getInstance(
+//MetricsAssignmentManagerSource.class);
{code}
Drop commented out code.
{code}
+ public static final String
METRIC_MAXNUM_REGIONS_IN_EACH_STATE="hbase.assignment.metric.maxmum.regions.in.each.state";
{code}
typo: maxnum -> maximum. If you meant maximum number of regions, use "max.num"
instead.
{code}
+ private static final int DEFAULT_METRIC_MAXNUM_REGIONS_IN_EACH_STATE = 200;
{code}
Consider lowering the default count since the count is per state. How about
using 100 as the default ?
{code}
+ protected int getMaxnumRegionInEachState(){
{code}
num -> Num
Region -> Regions
{code}
+return StringUtils.join(regions, ",");
{code}
I think using ', ' as delimiter would make the output more readable.
was (Author: yuzhih...@gmail.com):
{code}
+
metricsRecordBuilder.tag(Interns.info(REGIONS_STATE_FAILED_OPEN,REGIONS_STATE_FAILED_OPEN_DESC),metricsAssignmentWrapper.getFailedOpenRegions());
{code}
Please limit line length to 100 characters.
{code}
+//assignmentManagerSource = CompatibilitySingletonFactory.getInstance(
+//MetricsAssignmentManagerSource.class);
{code}
Drop commented out code.
{code}
+ public static final String
METRIC_MAXNUM_REGIONS_IN_EACH_STATE="hbase.assignment.metric.maxmum.regions.in.each.state";
typo: maxnum -> maximum. If you meant maximum number of regions, use "max.num"
instead.
{code}
+ private static final int DEFAULT_METRIC_MAXNUM_REGIONS_IN_EACH_STATE = 200;
{code}
Consider lowering the default count since the count is per state. How about
using 100 as the default ?
{code}
+ protected int getMaxnumRegionInEachState(){
{code}
num -> Num
Region -> Regions
{code}
+return StringUtils.join(regions, ",");
{code}
I think using ', ' as delimiter would make the output more readable.
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18118) Default storage policy if not configured cannot be "NONE"
[ https://issues.apache.org/jira/browse/HBASE-18118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493664#comment-16493664 ] Sean Busbey commented on HBASE-18118: - [~carp84] would you like to handle creating a Jira and working on the follow-up? Or would you rather review something I put up? > Default storage policy if not configured cannot be "NONE" > - > > Key: HBASE-18118 > URL: https://issues.apache.org/jira/browse/HBASE-18118 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-18118.patch > > > HBase can't use 'NONE' as default storage policy if not configured because > HDFS supports no such policy. This policy name was probably available in a > precommit or early version of the HDFS side support for heterogeneous > storage. Now the best default is 'HOT'. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20654) Expose regions in transition thru JMX
[
https://issues.apache.org/jira/browse/HBASE-20654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493665#comment-16493665
]
Ted Yu commented on HBASE-20654:
{code}
+
metricsRecordBuilder.tag(Interns.info(REGIONS_STATE_FAILED_OPEN,REGIONS_STATE_FAILED_OPEN_DESC),metricsAssignmentWrapper.getFailedOpenRegions());
{code}
Please limit line length to 100 characters.
{code}
+//assignmentManagerSource = CompatibilitySingletonFactory.getInstance(
+//MetricsAssignmentManagerSource.class);
{code}
Drop commented out code.
{code}
+ public static final String
METRIC_MAXNUM_REGIONS_IN_EACH_STATE="hbase.assignment.metric.maxmum.regions.in.each.state";
typo: maxnum -> maximum. If you meant maximum number of regions, use "max.num"
instead.
{code}
+ private static final int DEFAULT_METRIC_MAXNUM_REGIONS_IN_EACH_STATE = 200;
{code}
Consider lowering the default count since the count is per state. How about
using 100 as the default ?
{code}
+ protected int getMaxnumRegionInEachState(){
{code}
num -> Num
Region -> Regions
{code}
+return StringUtils.join(regions, ",");
{code}
I think using ', ' as delimiter would make the output more readable.
> Expose regions in transition thru JMX
> -
>
> Key: HBASE-20654
> URL: https://issues.apache.org/jira/browse/HBASE-20654
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: liubangchen
>Priority: Major
> Attachments: HBASE-20654-1.patch
>
>
> Currently only the count of regions in transition is exposed thru JMX.
> Here is a sample snippet of the /jmx output:
> {code}
> {
> "beans" : [ {
> ...
> }, {
> "name" : "Hadoop:service=HBase,name=Master,sub=AssignmentManager",
> "modelerType" : "Master,sub=AssignmentManager",
> "tag.Context" : "master",
> ...
> "ritCount" : 3
> {code}
> It would be desirable to expose region name, state for the regions in
> transition as well.
> We can place configurable upper bound on the number of entries returned in
> case there're a lot of regions in transition.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19761) Fix Checkstyle errors in hbase-zookeeper
[ https://issues.apache.org/jira/browse/HBASE-19761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493663#comment-16493663 ] maoling commented on HBASE-19761: - [~Jan Hentschel] 1.I exec the *dev-support/submit-patch.py -b master -jid HBASE-19761*,but cannot upload the patch automatically and create a review request for me(i''ll try it tomorrow).so I do this manually.[Review Board|https://reviews.apache.org/r/67353/] is reopen. 2.the latest patch is:HBASE-19761.master.001.patch > Fix Checkstyle errors in hbase-zookeeper > > > Key: HBASE-19761 > URL: https://issues.apache.org/jira/browse/HBASE-19761 > Project: HBase > Issue Type: Sub-task >Reporter: Jan Hentschel >Assignee: maoling >Priority: Minor > Attachments: HBASE-19761-master-v0.patch, > HBASE-19761-master-v1.patch, HBASE-19761-master-v2.patch, > HBASE-19761-master-v3.patch, HBASE-19761.master.001.patch > > > Fix the remaining Checkstyle errors in the *hbase-zookeeper* module and > enable Checkstyle to fail on violations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20628) SegmentScanner does over-comparing when one flushing
[
https://issues.apache.org/jira/browse/HBASE-20628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493659#comment-16493659
]
stack commented on HBASE-20628:
---
bq. Reasonable?
Mostly.
bq. ...seems to be inefficient with no real reason.
SegmentScanner goes against active Segments right? If Segment is active, won't
we want to get a new tailset iterator on a reseek at least for the active
Segment in case a Cell is inserted since we last got the Iterator? If so, we
should make the change for ImmutableSegment only?
INCLUDE_AND_SEEK_NEXT_COL (See description on HBASE-4433) interesting was added
to save on seeks (I was going to argue that it does not need to apply when all
data is loaded and up in memory as is case with SegmentScanner but memory
seeking costs tooa nd probably more trouble than its worth undoing it for this
one case in SQM...)
> SegmentScanner does over-comparing when one flushing
>
>
> Key: HBASE-20628
> URL: https://issues.apache.org/jira/browse/HBASE-20628
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: stack
>Priority: Critical
> Fix For: 2.0.1
>
> Attachments: HBASE-20628.branch-2.0.001 (1).patch,
> HBASE-20628.branch-2.0.001.patch, HBASE-20628.branch-2.0.001.patch,
> HBASE-20628.branch-2.0.002.patch, Screen Shot 2018-05-25 at 9.38.00 AM.png,
> hits-20628.png
>
>
> Flushing memstore is taking too long. It looks like we are doing a bunch of
> comparing out of a new facility in hbase2, the Segment scanner at flush time.
> Below is a patch from [~anoop.hbase]. I had a similar more hacky version.
> Both undo the extra comparing we were seeing in perf tests.
> [~anastas] and [~eshcar]. Need your help please.
> As I read it, we are trying to flush the memstore snapshot (default, no IMC
> case). There is only ever going to be one Segment involved (even if IMC is
> enabled); the snapshot Segment. But the getScanners is returning a list (of
> one) Scanners and the scan is via the generic SegmentScanner which is all
> about a bunch of stuff we don't need when doing a flush so it seems to do
> more work than is necessary. It also supports scanning backwards which is not
> needed when trying to flush memstore.
> Do you see a problem doing a version of Anoops patch (whether IMC or not)? It
> makes a big difference in general throughput when the below patch is in
> place. Thanks.
> {code}
> diff --git
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
>
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> index cbd60e5da3..c3dd972254 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreSnapshot.java
> @@ -40,7 +40,8 @@ public class MemStoreSnapshot implements Closeable {
> this.cellsCount = snapshot.getCellsCount();
> this.memStoreSize = snapshot.getMemStoreSize();
> this.timeRangeTracker = snapshot.getTimeRangeTracker();
> -this.scanners = snapshot.getScanners(Long.MAX_VALUE, Long.MAX_VALUE);
> +//this.scanners = snapshot.getScanners(Long.MAX_VALUE, Long.MAX_VALUE);
> +this.scanners = snapshot.getScannersForSnapshot();
> this.tagsPresent = snapshot.isTagsPresent();
>}
> diff --git
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
>
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> index 70074bf3b4..279c4e50c8 100644
> ---
> a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> +++
> b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
> @@ -33,6 +33,7 @@ import org.apache.hadoop.hbase.KeyValueUtil;
> import org.apache.hadoop.hbase.io.TimeRange;
> import org.apache.hadoop.hbase.util.Bytes;
> import org.apache.hadoop.hbase.util.ClassSize;
> +import org.apache.hadoop.hbase.util.CollectionBackedScanner;
> import org.apache.yetus.audience.InterfaceAudience;
> import org.slf4j.Logger;
> import
> org.apache.hbase.thirdparty.com.google.common.annotations.VisibleForTesting;
> @@ -130,6 +131,10 @@ public abstract class Segment {
> return Collections.singletonList(new SegmentScanner(this, readPoint,
> order));
>}
> + public List getScannersForSnapshot() {
> +return Collections.singletonList(new
> CollectionBackedScanner(this.cellSet.get(), comparator));
> + }
> +
>/**
> * @return whether the segment has any cells
> */
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20582) Bump up JRuby version because of some reported vulnerabilities
[
https://issues.apache.org/jira/browse/HBASE-20582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-20582:
---
Status: Patch Available (was: Reopened)
Forgot to re-click the button..
> Bump up JRuby version because of some reported vulnerabilities
> --
>
> Key: HBASE-20582
> URL: https://issues.apache.org/jira/browse/HBASE-20582
> Project: HBase
> Issue Type: Bug
> Components: dependencies, shell
>Reporter: Ankit Singhal
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20582.002.patch, HBASE-20582.addendum.patch,
> HBASE-20582.patch
>
>
> There are some vulnerabilities reported with two of the libraries used in
> HBase.
> {code:java}
> Jruby(version:9.1.10.0):
> CVE-2009-5147
> CVE-2013-4363
> CVE-2014-4975
> CVE-2014-8080
> CVE-2014-8090
> CVE-2015-3900
> CVE-2015-7551
> CVE-2015-9096
> CVE-2017-0899
> CVE-2017-0900
> CVE-2017-0901
> CVE-2017-0902
> CVE-2017-0903
> CVE-2017-10784
> CVE-2017-14064
> CVE-2017-9224
> CVE-2017-9225
> CVE-2017-9226
> CVE-2017-9227
> CVE-2017-9228
> {code}
> Tool somehow able to relate the vulnerability of Ruby with JRuby(Java
> implementation). (Jackson will be handled in a different issue.)
> Not all of them directly affects HBase but [~elserj] suggested that it is
> better to be on the updated version to avoid issues during an audit in
> security sensitive organization.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20653) Add missing observer hooks for region server group to MasterObserver
[
https://issues.apache.org/jira/browse/HBASE-20653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493644#comment-16493644
]
Nihal Jain edited comment on HBASE-20653 at 5/29/18 2:57 PM:
-
[^HBASE-20653.master.002.patch] This patch takes care of the following:
* Fixes doc-string problem
* Removes redundant method {{checkPermission}}
* Adds new tests in {{TestRSGroups }} to validate the CP hooks are called
Still need to figure out why tests failed in {{TestRSGroups}} in
[^HBASE-20653.master.001.patch]}}
was (Author: nihaljain.cs):
[^HBASE-20653.master.002.patch] This patch takes care of the following:
* Fixes doc-string problem
* Removes redundant method {{checkPermission}}
* Adds new tests in {{TestRSGroups }}to validate the CP hooks are called
Still need to figure out why tests failed in {{TestRSGroups in
[^HBASE-20653.master.001.patch]}}
> Add missing observer hooks for region server group to MasterObserver
>
>
> Key: HBASE-20653
> URL: https://issues.apache.org/jira/browse/HBASE-20653
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20653.master.001.patch,
> HBASE-20653.master.002.patch
>
>
> Currently the following region server group operations don't have
> corresponding hook in MasterObserver :
> * getRSGroupInfo
> * getRSGroupInfoOfServer
> * getRSGroupInfoOfTable
> * listRSGroup
> This JIRA is to
> * add them to MasterObserver
> * add corresponding permission check in AccessController
> * move the {{checkPermission}} out of RSGroupAdminEndpoint
> * add corresponding tests to TestRSGroupsWithACL
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20653) Add missing observer hooks for region server group to MasterObserver
[
https://issues.apache.org/jira/browse/HBASE-20653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493644#comment-16493644
]
Nihal Jain commented on HBASE-20653:
[^HBASE-20653.master.002.patch] This patch takes care of the following:
* Fixes doc-string problem
* Removes redundant method {{checkPermission}}
* Adds new tests in {{TestRSGroups }}to validate the CP hooks are called
Still need to figure out why tests failed in {{TestRSGroups in
[^HBASE-20653.master.001.patch]}}
> Add missing observer hooks for region server group to MasterObserver
>
>
> Key: HBASE-20653
> URL: https://issues.apache.org/jira/browse/HBASE-20653
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20653.master.001.patch,
> HBASE-20653.master.002.patch
>
>
> Currently the following region server group operations don't have
> corresponding hook in MasterObserver :
> * getRSGroupInfo
> * getRSGroupInfoOfServer
> * getRSGroupInfoOfTable
> * listRSGroup
> This JIRA is to
> * add them to MasterObserver
> * add corresponding permission check in AccessController
> * move the {{checkPermission}} out of RSGroupAdminEndpoint
> * add corresponding tests to TestRSGroupsWithACL
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20478) move import checks from hbaseanti to checkstyle
[ https://issues.apache.org/jira/browse/HBASE-20478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493642#comment-16493642 ] Sean Busbey commented on HBASE-20478: - okay. I'd rather have that kind of comment still in place (or maybe just a class level explanation?), since I don't think it's immediately obvious what's going on. I'm +1 on v8, either as-is or with the comment restored. > move import checks from hbaseanti to checkstyle > --- > > Key: HBASE-20478 > URL: https://issues.apache.org/jira/browse/HBASE-20478 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Sean Busbey >Assignee: Mike Drob >Priority: Minor > Attachments: HBASE-20478.0.patch, HBASE-20478.1.patch, > HBASE-20478.2.patch, HBASE-20478.3.patch, HBASE-20478.4.patch, > HBASE-20478.5.patch, HBASE-20478.6.patch, HBASE-20478.8.patch, > HBASE-20478.WIP.2.patch, HBASE-20478.WIP.2.patch, HBASE-20478.WIP.patch, > HBASE-anti-check.patch > > > came up in discussion on HBASE-20332. our check of "don't do this" things in > the codebase doesn't log the specifics of complaints anywhere, which forces > those who want to follow up to reverse engineer the check. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
