[jira] [Commented] (HBASE-20769) getSplits() has a out of bounds problem in TableSnapshotInputFormatImpl
[
https://issues.apache.org/jira/browse/HBASE-20769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522239#comment-16522239
]
Zheng Hu commented on HBASE-20769:
--
I've addressed my comment in RB Thanks [~yuzhih...@gmail.com] for pinging
me..
> getSplits() has a out of bounds problem in TableSnapshotInputFormatImpl
> ---
>
> Key: HBASE-20769
> URL: https://issues.apache.org/jira/browse/HBASE-20769
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0, 2.0.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-20769.master.001.patch,
> HBASE-20769.master.002.patch, HBASE-20769.master.003.patch
>
>
> When numSplits > 1, getSplits may create split that has start row smaller
> than user specified scan's start row or stop row larger than user specified
> scan's stop row.
> {code}
> byte[][] sp = sa.split(hri.getStartKey(), hri.getEndKey(), numSplits,
> true);
> for (int i = 0; i < sp.length - 1; i++) {
> if (PrivateCellUtil.overlappingKeys(scan.getStartRow(),
> scan.getStopRow(), sp[i],
> sp[i + 1])) {
> List hosts =
> calculateLocationsForInputSplit(conf, htd, hri, tableDir,
> localityEnabled);
> Scan boundedScan = new Scan(scan);
> boundedScan.setStartRow(sp[i]);
> boundedScan.setStopRow(sp[i + 1]);
> splits.add(new InputSplit(htd, hri, hosts, boundedScan,
> restoreDir));
> }
> }
> {code}
> Since we split keys by the range of regions, when sp[i] < scan.getStartRow()
> or sp[i + 1] > scan.getStopRow(), the created bounded scan may contain range
> that over user defined scan.
> fix should be simple:
> {code}
> boundedScan.setStartRow(
> Bytes.compareTo(scan.getStartRow(), sp[i]) > 0 ? scan.getStartRow() : sp[i]);
> boundedScan.setStopRow(
> Bytes.compareTo(scan.getStopRow(), sp[i + 1]) < 0 ? scan.getStopRow() : sp[i
> + 1]);
> {code}
> I will also try to add UTs to help discover this problem
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19997) [rolling upgrade] 1.x => 2.x
[
https://issues.apache.org/jira/browse/HBASE-19997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522303#comment-16522303
]
Duo Zhang commented on HBASE-19997:
---
OK the problem is that I forgot to use zk less assignment for 1.4.3. After
setting {{hbase.assignment.usezk}} to true, the rolling upgrade works fine. I
tried the unload/load way, and also killed a RS directly without unloading its
regions, both worked fine, the 1.x master can work together with the 2.x RS.
And at last I restarted the master and upgrade them to 2.1.0, the rolling
upgrade finished successfully.
Will write up a doc about how to do rolling upgrade from 1.x to 2.x soon. And
you can also try it [~stack]. And there is a known issue for 2.1.0-SNAPSHOT,
the SNAPSHOT suffix will be lost in the master page. Filed HBASE-20784 for it,
not a big problem I think.
> [rolling upgrade] 1.x => 2.x
>
>
> Key: HBASE-19997
> URL: https://issues.apache.org/jira/browse/HBASE-19997
> Project: HBase
> Issue Type: Umbrella
>Reporter: stack
>Priority: Blocker
> Fix For: 2.1.0
>
> Attachments: Screenshot from 2018-05-03 14-43-46.png
>
>
> An umbrella issue of issues needed so folks can do a rolling upgrade from
> hbase-1.x to hbase-2.x.
> (Recent) Notables:
> * hbase-1.x can't read hbase-2.x WALs -- hbase-1.x doesn't know the
> AsyncProtobufLogWriter class used writing the WAL -- see
> https://issues.apache.org/jira/browse/HBASE-19166?focusedCommentId=16362897=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16362897
> for exception.
> ** Might be ok... means WAL split fails on an hbase1 RS... must wait till an
> hbase-2.x RS picks up the WAL for it to be split.
> * hbase-1 can't open regions from tables created by hbase-2; it can't find
> the Table descriptor. See
> https://issues.apache.org/jira/browse/HBASE-19116?focusedCommentId=16363276=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16363276
> ** This might be ok if the tables we are doing rolling upgrade over were
> written with hbase-1.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-18201) add UT and docs for DataBlockEncodingTool
[ https://issues.apache.org/jira/browse/HBASE-18201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuan-Po Tseng updated HBASE-18201: -- Attachment: HBASE-18201.master.003.patch > add UT and docs for DataBlockEncodingTool > - > > Key: HBASE-18201 > URL: https://issues.apache.org/jira/browse/HBASE-18201 > Project: HBase > Issue Type: Task > Components: tooling >Reporter: Chia-Ping Tsai >Assignee: Kuan-Po Tseng >Priority: Minor > Labels: beginner > Attachments: HBASE-18201.master.001.patch, > HBASE-18201.master.002.patch, HBASE-18201.master.002.patch, > HBASE-18201.master.003.patch > > > There is no example, documents, or tests for DataBlockEncodingTool. We should > have it friendly if any use case exists. Otherwise, we should just get rid of > it because DataBlockEncodingTool presumes that the implementation of cell > returned from DataBlockEncoder is KeyValue. The presume may obstruct the > cleanup of KeyValue references in the code base of read/write path. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20766) Verify Replication Tool Has Typo "remove cluster"
[ https://issues.apache.org/jira/browse/HBASE-20766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi updated HBASE-20766: -- Labels: beginner (was: ) > Verify Replication Tool Has Typo "remove cluster" > - > > Key: HBASE-20766 > URL: https://issues.apache.org/jira/browse/HBASE-20766 > Project: HBase > Issue Type: Bug >Reporter: Clay B. >Priority: Trivial > Labels: beginner > > The verify replication tool has a trivial typo "remove cluster" instead of > "remote cluster": > https://github.com/apache/hbase/blob/a6eeb26cc0b4d0af3fff50b5b931b6847df1f9d2/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.java#L355 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20783) NPE encountered when rolling update from master with an async peer to branch HBASE-19064
[
https://issues.apache.org/jira/browse/HBASE-20783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522291#comment-16522291
]
Hadoop QA commented on HBASE-20783:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HBASE-19064 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
43s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
55s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} HBASE-19064 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
32s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 14s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
59s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20783 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12929026/HBASE-20783-HBASE-19064-v1.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 1c9eb2a7c55e 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | HBASE-19064 / 25add19a08 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13380/testReport/ |
| Max. process+thread count | 248 (vs. ulimit of 1) |
| modules | C: hbase-client U: hbase-client |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13380/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically
[jira] [Commented] (HBASE-13147) Load actual META table descriptor, don't use statically defined one.
[ https://issues.apache.org/jira/browse/HBASE-13147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522317#comment-16522317 ] Duo Zhang commented on HBASE-13147: --- I think this could be done by upgrading an rs first and then move the meta region to the new RS. At xiaomi we always do it like this, and with HBASE-17931 it will be done automatically. And when upgrading from 1.x to 2.x, for now a possible way is to upgrade RS before master, so this will not a problem. > Load actual META table descriptor, don't use statically defined one. > > > Key: HBASE-13147 > URL: https://issues.apache.org/jira/browse/HBASE-13147 > Project: HBase > Issue Type: Bug > Components: master, regionserver >Affects Versions: 2.0.0 >Reporter: Andrey Stepachev >Assignee: Andrey Stepachev >Priority: Major > Attachments: HBASE-13147-branch-1.patch, > HBASE-13147-branch-1.v2.patch, HBASE-13147.patch, HBASE-13147.v2.patch, > HBASE-13147.v3.patch, HBASE-13147.v4.patch, HBASE-13147.v4.patch, > HBASE-13147.v5.patch, HBASE-13147.v6.patch, HBASE-13147.v7.patch > > > In HBASE-13087 stumbled on the fact, that region servers don't see actual > meta descriptor, they use their own, statically compiled. > Need to fix that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20691) Storage policy should allow deferring to HDFS
[
https://issues.apache.org/jira/browse/HBASE-20691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522348#comment-16522348
]
Sean Busbey commented on HBASE-20691:
-
{quote}
Is it too tight coupling to do something like...
{code}
static void setStoragePolicy(final FileSystem fs, final Path path, boolean
throwException) throws IOException {
final String storagePolicy = fs.getConf().get(HConstants.WAL_STORAGE_POLICY,
HConstants.DEFAULT_WAL_STORAGE_POLICY);
{code}
Can see argument going either way, probably personal preference at that point.
{quote}
Too tight. Doesn't the per-CF storage policy code use this same method?
> Storage policy should allow deferring to HDFS
> -
>
> Key: HBASE-20691
> URL: https://issues.apache.org/jira/browse/HBASE-20691
> Project: HBase
> Issue Type: Bug
> Components: Filesystem Integration, wal
>Affects Versions: 1.5.0, 2.0.0
>Reporter: Sean Busbey
>Assignee: Yu Li
>Priority: Minor
> Fix For: 2.1.0, 1.5.0
>
> Attachments: HBASE-20691.patch, HBASE-20691.v2.patch,
> HBASE-20691.v3.patch, HBASE-20691.v4.patch, HBASE-20691.v5.patch
>
>
> In HBase 1.1 - 1.4 we can defer storage policy decisions to HDFS by using
> "NONE" as the storage policy in hbase configs.
> As described on this [dev@hbase thread "WAL storage policies and interactions
> with Hadoop admin
> tools."|https://lists.apache.org/thread.html/d220726fab4bb4c9e117ecc8f44246402dd97bfc986a57eb2237@%3Cdev.hbase.apache.org%3E]
> we no longer have that option in 2.0.0 and 1.5.0 (as the branch is now).
> Additionally, we can't set the policy to HOT in the event that HDFS has
> changed the policy for a parent directory of our WALs.
> We should put back that ability. Presuming this is done by re-adopting the
> "NONE" placeholder variable, we need to ensure that value doesn't get passed
> to HDFS APIs. Since it isn't a valid storage policy attempting to use it will
> cause a bunch of logging churn (which will be a regression of the problem
> HBASE-18118 sought to fix).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20769) getSplits() has a out of bounds problem in TableSnapshotInputFormatImpl
[
https://issues.apache.org/jira/browse/HBASE-20769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522195#comment-16522195
]
Ted Yu commented on HBASE-20769:
[~openinx]:
Can you take a look as well ?
If this is good by you, can you help commit ?
Thanks
> getSplits() has a out of bounds problem in TableSnapshotInputFormatImpl
> ---
>
> Key: HBASE-20769
> URL: https://issues.apache.org/jira/browse/HBASE-20769
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0, 2.0.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-20769.master.001.patch,
> HBASE-20769.master.002.patch, HBASE-20769.master.003.patch
>
>
> When numSplits > 1, getSplits may create split that has start row smaller
> than user specified scan's start row or stop row larger than user specified
> scan's stop row.
> {code}
> byte[][] sp = sa.split(hri.getStartKey(), hri.getEndKey(), numSplits,
> true);
> for (int i = 0; i < sp.length - 1; i++) {
> if (PrivateCellUtil.overlappingKeys(scan.getStartRow(),
> scan.getStopRow(), sp[i],
> sp[i + 1])) {
> List hosts =
> calculateLocationsForInputSplit(conf, htd, hri, tableDir,
> localityEnabled);
> Scan boundedScan = new Scan(scan);
> boundedScan.setStartRow(sp[i]);
> boundedScan.setStopRow(sp[i + 1]);
> splits.add(new InputSplit(htd, hri, hosts, boundedScan,
> restoreDir));
> }
> }
> {code}
> Since we split keys by the range of regions, when sp[i] < scan.getStartRow()
> or sp[i + 1] > scan.getStopRow(), the created bounded scan may contain range
> that over user defined scan.
> fix should be simple:
> {code}
> boundedScan.setStartRow(
> Bytes.compareTo(scan.getStartRow(), sp[i]) > 0 ? scan.getStartRow() : sp[i]);
> boundedScan.setStopRow(
> Bytes.compareTo(scan.getStopRow(), sp[i + 1]) < 0 ? scan.getStopRow() : sp[i
> + 1]);
> {code}
> I will also try to add UTs to help discover this problem
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20783) NPE encountered when rolling update from master with an async peer to branch HBASE-19064
[
https://issues.apache.org/jira/browse/HBASE-20783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-20783:
-
Attachment: HBASE-20783-HBASE-19064-v1.patch
> NPE encountered when rolling update from master with an async peer to branch
> HBASE-19064
>
>
> Key: HBASE-20783
> URL: https://issues.apache.org/jira/browse/HBASE-20783
> Project: HBase
> Issue Type: Sub-task
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Attachments: HBASE-20783-HBASE-19064-v1.patch,
> HBASE-20783-HBASE-19064-v1.patch
>
>
> {code}
> 2018-06-25 16:25:04,261 ERROR [Thread-14] master.HMaster: Failed to become
> active master
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.replication.SyncReplicationState.parseFrom(SyncReplicationState.java:72)
> at
> org.apache.hadoop.hbase.replication.ZKReplicationPeerStorage.getSyncReplicationState(ZKReplicationPeerStorage.java:224)
> at
> org.apache.hadoop.hbase.replication.ZKReplicationPeerStorage.getPeerSyncReplicationState(ZKReplicationPeerStorage.java:240)
> at
> org.apache.hadoop.hbase.master.replication.ReplicationPeerManager.create(ReplicationPeerManager.java:479)
> at
> org.apache.hadoop.hbase.master.HMaster.initializeZKBasedSystemTrackers(HMaster.java:755)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:895)
> at
> org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2126)
> at
> org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:571)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18366) Fix flaky test hbase.master.procedure.TestServerCrashProcedure#testRecoveryAndDoubleExecutionOnRsWithMeta
[ https://issues.apache.org/jira/browse/HBASE-18366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522310#comment-16522310 ] Duo Zhang commented on HBASE-18366: --- Seems not flakey any more? > Fix flaky test > hbase.master.procedure.TestServerCrashProcedure#testRecoveryAndDoubleExecutionOnRsWithMeta > - > > Key: HBASE-18366 > URL: https://issues.apache.org/jira/browse/HBASE-18366 > Project: HBase > Issue Type: Bug >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Major > Attachments: hbase-18366.fix1.patch, hbase-18366.fix2.patch > > > It worked for a few days after enabling it with HBASE-18278. But started > failing after commits: > 6786b2b > 68436c9 > 75d2eca > 50bb045 > df93c13 > It works with one commit before: c5abb6c. Need to see what changed with those > commits. > Currently it fails with TableNotFoundException. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20357) AccessControlClient API Enhancement
[ https://issues.apache.org/jira/browse/HBASE-20357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522157#comment-16522157 ] Ted Yu commented on HBASE-20357: Please fill out release note. > AccessControlClient API Enhancement > --- > > Key: HBASE-20357 > URL: https://issues.apache.org/jira/browse/HBASE-20357 > Project: HBase > Issue Type: Improvement > Components: security >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Major > Attachments: HBASE-20357.master.001.patch, > HBASE-20357.master.002.patch, HBASE-20357.master.003.patch > > > *Background:* > Currently HBase ACLs can be retrieved based on the namespace or table name > only. There is no direct API available to retrieve the permissions based on > the namespace, table name, column family and column qualifier for specific > user. > Client has to write application logic in multiple steps to retrieve ACLs > based on table name, column name and column qualifier for specific user. > HBase should enhance AccessControlClient APIs to simplyfy this. > *AccessControlClient API should be extended with following APIs,* > # To retrieve permissions based on the namespace, table name, column family > and column qualifier for specific user. > Permissions can be retrieved based on the following inputs, > - Namespace/Table (already available) > - Namespace/Table + UserName > - Table + CF > - Table + CF + UserName > - Table + CF + CQ > - Table + CF + CQ + UserName > Scope of retrieving permission will be as follows, > - Same as existing > 2. To validate whether a user is allowed to perform specified > operations on a particular table, will be useful to check user privilege > instead of getting ACD during client > operation. > User validation can be performed based on following inputs, > - Table + CF + CQ + UserName + Actions > Scope of validating user privilege, > User can perform self check without any special privilege > but ADMIN privilege will be required to perform check for other users. > For example, suppose there are two users "userA" & > "userB" then there can be below scenarios, > - when userA want to check whether userA have > privilege to perform mentioned actions > > userA don't need ADMIN privilege, as it's a > self query. > - when userA want to check whether userB have > privilege to perform mentioned actions, > > userA must have ADMIN or superuser > privilege, as it's trying to query for other user. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20193) Basic Replication Web UI - Regionserver
[ https://issues.apache.org/jira/browse/HBASE-20193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522199#comment-16522199 ] Jingyun Tian commented on HBASE-20193: -- [~Apache9] TestReplicationMetricsforUI This failed UT is related, I'm working on fixing it right now. I'll upload new patch asap. > Basic Replication Web UI - Regionserver > > > Key: HBASE-20193 > URL: https://issues.apache.org/jira/browse/HBASE-20193 > Project: HBase > Issue Type: Sub-task > Components: Replication, Usability >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Critical > Fix For: 2.1.0 > > Attachments: HBASE-20193.master.001.patch, > HBASE-20193.master.002.patch, HBASE-20193.master.003.patch, > HBASE-20193.master.004.patch, HBASE-20193.master.004.patch, > HBASE-20193.master.005.patch, HBASE-20193.master.006.patch, > HBASE-20193.master.006.patch, HBASE-20193.master.007.patch > > > subtask of HBASE-15809. Implementation of replication UI on Regionserver web > page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20771) PUT operation fail with "No server address listed in hbase:meta for region xxxxx"
[
https://issues.apache.org/jira/browse/HBASE-20771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522137#comment-16522137
]
Hadoop QA commented on HBASE-20771:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
36s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
34s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 34s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
21s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:1f3957d |
| JIRA Issue | HBASE-20771 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12929016/HBASE-20771.branch-1.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux a542f0e7b6b9 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-20357) AccessControlClient API Enhancement
[ https://issues.apache.org/jira/browse/HBASE-20357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522152#comment-16522152 ] Ted Yu commented on HBASE-20357: +1 > AccessControlClient API Enhancement > --- > > Key: HBASE-20357 > URL: https://issues.apache.org/jira/browse/HBASE-20357 > Project: HBase > Issue Type: Improvement > Components: security >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Major > Attachments: HBASE-20357.master.001.patch, > HBASE-20357.master.002.patch, HBASE-20357.master.003.patch > > > *Background:* > Currently HBase ACLs can be retrieved based on the namespace or table name > only. There is no direct API available to retrieve the permissions based on > the namespace, table name, column family and column qualifier for specific > user. > Client has to write application logic in multiple steps to retrieve ACLs > based on table name, column name and column qualifier for specific user. > HBase should enhance AccessControlClient APIs to simplyfy this. > *AccessControlClient API should be extended with following APIs,* > # To retrieve permissions based on the namespace, table name, column family > and column qualifier for specific user. > Permissions can be retrieved based on the following inputs, > - Namespace/Table (already available) > - Namespace/Table + UserName > - Table + CF > - Table + CF + UserName > - Table + CF + CQ > - Table + CF + CQ + UserName > Scope of retrieving permission will be as follows, > - Same as existing > 2. To validate whether a user is allowed to perform specified > operations on a particular table, will be useful to check user privilege > instead of getting ACD during client > operation. > User validation can be performed based on following inputs, > - Table + CF + CQ + UserName + Actions > Scope of validating user privilege, > User can perform self check without any special privilege > but ADMIN privilege will be required to perform check for other users. > For example, suppose there are two users "userA" & > "userB" then there can be below scenarios, > - when userA want to check whether userA have > privilege to perform mentioned actions > > userA don't need ADMIN privilege, as it's a > self query. > - when userA want to check whether userB have > privilege to perform mentioned actions, > > userA must have ADMIN or superuser > privilege, as it's trying to query for other user. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20691) Storage policy should allow deferring to HDFS
[
https://issues.apache.org/jira/browse/HBASE-20691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522326#comment-16522326
]
Mike Drob commented on HBASE-20691:
---
{code}
+ if (LOG.isTraceEnabled()) {
+LOG.trace("We were passed the defer-to-hdfs policy " + storagePolicy +
", exiting early.");
+ }
{code}
please use parameterized logging, and I don't think we need the conditional
check.
{code}
+ toThrow = e;
{code}
Do we want to be rethrowing the reflection related exceptions too? Not just the
ones related to HDFS execution? This confuses me. Also, will need to update the
comment about "// This swallows FNFE, should we be throwing it?"
{code}
+ static void setStoragePolicy(final FileSystem fs, final Path path, final
String storagePolicy,
+ boolean throwException) throws IOException {
{code}
Is it too tight coupling to do something like...
{code}
static void setStoragePolicy(final FileSystem fs, final Path path, boolean
throwException) throws IOException {
final String storagePolicy = fs.getConf().get(HConstants.WAL_STORAGE_POLICY,
HConstants.DEFAULT_WAL_STORAGE_POLICY);
{code}
Can see argument going either way, probably personal preference at that point.
> Storage policy should allow deferring to HDFS
> -
>
> Key: HBASE-20691
> URL: https://issues.apache.org/jira/browse/HBASE-20691
> Project: HBase
> Issue Type: Bug
> Components: Filesystem Integration, wal
>Affects Versions: 1.5.0, 2.0.0
>Reporter: Sean Busbey
>Assignee: Yu Li
>Priority: Minor
> Fix For: 2.1.0, 1.5.0
>
> Attachments: HBASE-20691.patch, HBASE-20691.v2.patch,
> HBASE-20691.v3.patch, HBASE-20691.v4.patch, HBASE-20691.v5.patch
>
>
> In HBase 1.1 - 1.4 we can defer storage policy decisions to HDFS by using
> "NONE" as the storage policy in hbase configs.
> As described on this [dev@hbase thread "WAL storage policies and interactions
> with Hadoop admin
> tools."|https://lists.apache.org/thread.html/d220726fab4bb4c9e117ecc8f44246402dd97bfc986a57eb2237@%3Cdev.hbase.apache.org%3E]
> we no longer have that option in 2.0.0 and 1.5.0 (as the branch is now).
> Additionally, we can't set the policy to HOT in the event that HDFS has
> changed the policy for a parent directory of our WALs.
> We should put back that ability. Presuming this is done by re-adopting the
> "NONE" placeholder variable, we need to ensure that value doesn't get passed
> to HDFS APIs. Since it isn't a valid storage policy attempting to use it will
> cause a bunch of logging churn (which will be a regression of the problem
> HBASE-18118 sought to fix).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-14112) VerifyReplication may fail to count bad rows when regions of table in source cluster are empty
[
https://issues.apache.org/jira/browse/HBASE-14112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522542#comment-16522542
]
Hadoop QA commented on HBASE-14112:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
11s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 37s{color}
| {color:red} hbase-mapreduce generated 1 new + 158 unchanged - 1 fixed = 159
total (was 159) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
19s{color} | {color:red} hbase-mapreduce: The patch generated 1 new + 19
unchanged - 0 fixed = 20 total (was 19) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
1s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 42s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 51s{color}
| {color:red} hbase-mapreduce in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.replication.TestVerifyReplication |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-14112 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12929043/HBASE-14112.master.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux a93360f4ce96 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 9640ebacd4 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| javac |
https://builds.apache.org/job/PreCommit-HBASE-Build/13382/artifact/patchprocess/diff-compile-javac-hbase-mapreduce.txt
|
| checkstyle |
[jira] [Updated] (HBASE-14112) VerifyReplication may fail to count bad rows when regions of table in source cluster are empty
[ https://issues.apache.org/jira/browse/HBASE-14112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-14112: -- Attachment: HBASE-14112.master.001.patch > VerifyReplication may fail to count bad rows when regions of table in source > cluster are empty > -- > > Key: HBASE-14112 > URL: https://issues.apache.org/jira/browse/HBASE-14112 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 0.99.2 >Reporter: Jianwei Cui >Priority: Minor > Attachments: HBASE-14112-trunk-v1.patch, HBASE-14112.master.001.patch > > > VerifyReplication will start the scanner for the table of peer cluster in > Verifier#map(...). If the regions of the table in source cluster are empty, > the Verifier#map(...) won't be invoked, so that the tool can not count the > bad rows in the table of peer cluster. A possible improvement is to check > whether the scanner for the table of peer cluster has been opened in > Verifier#cleanup(...), and if not, open the scanner and count the rows only > in the table of peer cluster as bad rows. Suggestions and discussions are > welcomed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-20635) Support to convert the shaded user permission proto to client user permission object
[
https://issues.apache.org/jira/browse/HBASE-20635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajeshbabu Chintaguntla resolved HBASE-20635.
-
Resolution: Fixed
bq. You understand the difference between hbase-protocol and
hbase-protocol-shaded and that the shaded utils are for internal use only?
Yes understood [~stack]. Thanks.
> Support to convert the shaded user permission proto to client user permission
> object
>
>
> Key: HBASE-20635
> URL: https://issues.apache.org/jira/browse/HBASE-20635
> Project: HBase
> Issue Type: Bug
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20635.patch, HBASE-20635_v2.patch,
> PHOENIX-4528_5.x-HBase-2.0_v2.patch
>
>
> Currently we have API to build the protobuf UserPermission to client user
> permission in AccessControlUtil but we cannot do the same when we use shaded
> protobufs.
> {noformat}
> /**
>* Converts a user permission proto to a client user permission object.
>*
>* @param proto the protobuf UserPermission
>* @return the converted UserPermission
>*/
> public static UserPermission
> toUserPermission(AccessControlProtos.UserPermission proto) {
> return new UserPermission(proto.getUser().toByteArray(),
> toTablePermission(proto.getPermission()));
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-14112) VerifyReplication may fail to count bad rows when regions of table in source cluster are empty
[ https://issues.apache.org/jira/browse/HBASE-14112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob reassigned HBASE-14112: - Assignee: Mike Drob > VerifyReplication may fail to count bad rows when regions of table in source > cluster are empty > -- > > Key: HBASE-14112 > URL: https://issues.apache.org/jira/browse/HBASE-14112 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 1.0.0, 0.99.2, 2.0.0 >Reporter: Jianwei Cui >Assignee: Mike Drob >Priority: Minor > Attachments: HBASE-14112-trunk-v1.patch, HBASE-14112.master.001.patch > > > VerifyReplication will start the scanner for the table of peer cluster in > Verifier#map(...). If the regions of the table in source cluster are empty, > the Verifier#map(...) won't be invoked, so that the tool can not count the > bad rows in the table of peer cluster. A possible improvement is to check > whether the scanner for the table of peer cluster has been opened in > Verifier#cleanup(...), and if not, open the scanner and count the rows only > in the table of peer cluster as bad rows. Suggestions and discussions are > welcomed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-14112) VerifyReplication may fail to count bad rows when regions of table in source cluster are empty
[ https://issues.apache.org/jira/browse/HBASE-14112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-14112: -- Affects Version/s: 1.0.0 2.0.0 Status: Patch Available (was: Open) Rebased original patch, needed to make some minor tweaks to it to apply. > VerifyReplication may fail to count bad rows when regions of table in source > cluster are empty > -- > > Key: HBASE-14112 > URL: https://issues.apache.org/jira/browse/HBASE-14112 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 2.0.0, 0.99.2, 1.0.0 >Reporter: Jianwei Cui >Assignee: Mike Drob >Priority: Minor > Attachments: HBASE-14112-trunk-v1.patch, HBASE-14112.master.001.patch > > > VerifyReplication will start the scanner for the table of peer cluster in > Verifier#map(...). If the regions of the table in source cluster are empty, > the Verifier#map(...) won't be invoked, so that the tool can not count the > bad rows in the table of peer cluster. A possible improvement is to check > whether the scanner for the table of peer cluster has been opened in > Verifier#cleanup(...), and if not, open the scanner and count the rows only > in the table of peer cluster as bad rows. Suggestions and discussions are > welcomed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20553) Add dependency CVE checking to nightly tests
[
https://issues.apache.org/jira/browse/HBASE-20553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522413#comment-16522413
]
Sean Busbey commented on HBASE-20553:
-
sure. if you've got it working locally, please post a patch and I'll make a
branch based on this issue. the patch will need to include the jira name and
the branch name, so it'll have something terrible like
"HBASE-20553-HBASE-20553.v0.patch" where the "v0" part increments as you make
changes.
> Add dependency CVE checking to nightly tests
>
>
> Key: HBASE-20553
> URL: https://issues.apache.org/jira/browse/HBASE-20553
> Project: HBase
> Issue Type: Umbrella
> Components: dependencies
>Affects Versions: 3.0.0
>Reporter: Sean Busbey
>Assignee: Sakthi
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
>
> We should proactively work to flag dependencies with known CVEs so that we
> can then update them early in our development instead of near a release.
> YETUS-441 is working to add a plugin for this, we should grab a copy early to
> make sure it works for us.
> Rough outline:
> 1. [install yetus locally|http://yetus.apache.org/downloads/]
> 2. [install the dependency-check
> cli|https://www.owasp.org/index.php/OWASP_Dependency_Check] (homebrew
> instructions on right hand margin)
> 3. Get a local copy of the OWASP datafile ({{dependency-check --updateonly
> --data /some/local/path/to/dir}})
> 4. Run {{hbase_nightly_yetus.sh}} using matching environment variables from
> the “yetus general check” (currently [line #126 in our nightly
> Jenkinsfile|https://github.com/apache/hbase/blob/master/dev-support/Jenkinsfile#L126])
> 5. Grab the plugin definition and suppression file from from YETUS-441
> 6. put the plugin definition either in a directory of dev-support or into the
> hbase-personality.sh directly
> 7. Re-run {{hbase_nightly_yetus.sh}} to verify that the plugin results show
> up. (Probably this will involve adding new pointers for “where is the
> suppression file”, “where is the OWASP datafile” and pointing them somewhere
> locally.)
> Once all of that is in place we’ll get the changes needed into a branch that
> we can test out. Over in YETUS-441 I’ll need to add a jenkins job that’ll
> handle periodically updating a copy of the datafile for the OWASP dependency
> checker. Presuming I have that in place by the time we have a nightly branch
> to check this out, then we’ll also need to update our nightly Jenkinsfile to
> fetch the data file from that job.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-20649) Validate HFiles do not have PREFIX_TREE DataBlockEncoding
[ https://issues.apache.org/jira/browse/HBASE-20649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Balazs Meszaros reassigned HBASE-20649: --- Assignee: Balazs Meszaros > Validate HFiles do not have PREFIX_TREE DataBlockEncoding > - > > Key: HBASE-20649 > URL: https://issues.apache.org/jira/browse/HBASE-20649 > Project: HBase > Issue Type: New Feature >Reporter: Peter Somogyi >Assignee: Balazs Meszaros >Priority: Minor > Attachments: HBASE-20649.master.001.patch, > HBASE-20649.master.002.patch, HBASE-20649.master.003.patch > > > HBASE-20592 adds a tool to check column families on the cluster do not have > PREFIX_TREE encoding. > Since it is possible that DataBlockEncoding was already changed but HFiles > are not rewritten yet we would need a tool that can verify the content of > hfiles in the cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20777) RpcConnection could still remain opened after we shutdown the NettyRpcServer
[
https://issues.apache.org/jira/browse/HBASE-20777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522459#comment-16522459
]
Hudson commented on HBASE-20777:
Results for branch master
[build #376 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/376/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/376//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/376//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/376//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> RpcConnection could still remain opened after we shutdown the NettyRpcServer
>
>
> Key: HBASE-20777
> URL: https://issues.apache.org/jira/browse/HBASE-20777
> Project: HBase
> Issue Type: Bug
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HBASE-20777-v1.patch, HBASE-20777.patch,
> org.apache.hadoop.hbase.client.TestAsyncTableBatch-output.txt
>
>
> The log is very strange, we keep sending request to a dead RS, and the result
> is not connection refused, but rpc timeout, and later it becomes
> CallQueueTooBig...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20763) Update guava >=24.1.1
[ https://issues.apache.org/jira/browse/HBASE-20763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522577#comment-16522577 ] Mike Drob commented on HBASE-20763: --- +1 > Update guava >=24.1.1 > - > > Key: HBASE-20763 > URL: https://issues.apache.org/jira/browse/HBASE-20763 > Project: HBase > Issue Type: Task > Components: thirdparty >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: thirdparty-2.2.0 > > Attachments: HBASE-20763.001.patch > > > We should update Guava in hbase-thirdparty to stop shipping the code cited as > vulnerable in CVE-2018-10237. We do not invoke this code ourselves and users > would have to try pretty hard to use it themselves, but we've seen more > strange things before ;) > Let's just bump up the dependency and move on. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-20762) precommit should archive generated LICENSE file
[
https://issues.apache.org/jira/browse/HBASE-20762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Drob resolved HBASE-20762.
---
Resolution: Not A Problem
> precommit should archive generated LICENSE file
> ---
>
> Key: HBASE-20762
> URL: https://issues.apache.org/jira/browse/HBASE-20762
> Project: HBase
> Issue Type: Bug
> Components: build, community
>Reporter: Mike Drob
>Priority: Major
>
> When a precommit run fails due to license issues, we get pointed to a file in
> our maven logs:
> {noformat}
> /testptch/hbase/hbase-assembly/target/maven-shared-archive-resources/META-INF/LICENSE
> {noformat}
> But we don't have that file saved, so we don't know what the actual failure
> was. So we should save that in our build artifacts. Or maybe we can print a
> snippet from that file directly into the maven log. Both would be acceptable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20781) Save recalculating families in a WALEdit batch of Cells
[ https://issues.apache.org/jira/browse/HBASE-20781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20781: -- Attachment: HBASE-20781.branch-2.0.001.patch > Save recalculating families in a WALEdit batch of Cells > --- > > Key: HBASE-20781 > URL: https://issues.apache.org/jira/browse/HBASE-20781 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Major > Attachments: 2.0621.2.12782.cpu.svg, 2.0621.256k.51351.cpu.svg, > 2.0623.families.121250.cpu.svg, HBASE-20781.branch-2.0.001.patch, > HBASE-20781.master.001.patch > > > Doing a doMiniBatchMutate, we calculate the set of families that the WALEdit > Cells belong to up front but down after the RingBuffer when we make an > FSWALEdit, we spin through all the Cells again to figure the set of families > in a particularly painful way. Just pass the calculated family set in the > WALEdit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20785) NPE getting metrics in PE testing scans
[ https://issues.apache.org/jira/browse/HBASE-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20785: -- Attachment: HBASE-20785.branch-1.4.001.patch > NPE getting metrics in PE testing scans > --- > > Key: HBASE-20785 > URL: https://issues.apache.org/jira/browse/HBASE-20785 > Project: HBase > Issue Type: Bug > Components: Performance >Affects Versions: 1.4.4 >Reporter: stack >Assignee: stack >Priority: Major > Attachments: HBASE-20785.branch-1.4.001.patch, > HBASE-20785.branch-1.4.001.patch > > > Comparing scans using PE. In branch-1 at least, I was getting an NPE when we > tried to use a null metrics instance. Seems transient around startup. > One-liner patch coming. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20787) Rebase the HBASE-18477 onto the current master to continue dev
[ https://issues.apache.org/jira/browse/HBASE-20787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522979#comment-16522979 ] Zach York commented on HBASE-20787: --- I will also remove the various commits/reverts of the initial patch to simplify things. > Rebase the HBASE-18477 onto the current master to continue dev > -- > > Key: HBASE-20787 > URL: https://issues.apache.org/jira/browse/HBASE-20787 > Project: HBase > Issue Type: Sub-task >Affects Versions: HBASE-18477 >Reporter: Zach York >Assignee: Zach York >Priority: Minor > Fix For: HBASE-18477 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20788) Write up a doc about how to rolling upgrade from 1.x to 2.x
Duo Zhang created HBASE-20788: - Summary: Write up a doc about how to rolling upgrade from 1.x to 2.x Key: HBASE-20788 URL: https://issues.apache.org/jira/browse/HBASE-20788 Project: HBase Issue Type: Sub-task Reporter: Duo Zhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19722) Meta query statistics metrics source
[ https://issues.apache.org/jira/browse/HBASE-19722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522939#comment-16522939 ] stack commented on HBASE-19722: --- [~apurtell] in hbase-2.0 too please. > Meta query statistics metrics source > > > Key: HBASE-19722 > URL: https://issues.apache.org/jira/browse/HBASE-19722 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Xu Cang >Priority: Major > Attachments: HBASE-19722.branch-1.v001.patch, > HBASE-19722.master.010.patch, HBASE-19722.master.011.patch, > HBASE-19722.master.012.patch, HBASE-19722.master.013.patch, > HBASE-19722.master.014.patch, HBASE-19722.master.015.patch, > HBASE-19722.master.016.patch > > > Implement a meta query statistics metrics source, created whenever a > regionserver starts hosting meta, removed when meta hosting moves. Provide > views on top tables by request counts, top meta rowkeys by request count, top > clients making requests by their hostname. > Can be implemented as a coprocessor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19722) Meta query statistics metrics source
[ https://issues.apache.org/jira/browse/HBASE-19722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522940#comment-16522940 ] stack commented on HBASE-19722: --- [~xucang] Needs a nice release note on how to deploy and some output on what gets reported to get folks excited about this nice new facility. > Meta query statistics metrics source > > > Key: HBASE-19722 > URL: https://issues.apache.org/jira/browse/HBASE-19722 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Xu Cang >Priority: Major > Attachments: HBASE-19722.branch-1.v001.patch, > HBASE-19722.master.010.patch, HBASE-19722.master.011.patch, > HBASE-19722.master.012.patch, HBASE-19722.master.013.patch, > HBASE-19722.master.014.patch, HBASE-19722.master.015.patch, > HBASE-19722.master.016.patch > > > Implement a meta query statistics metrics source, created whenever a > regionserver starts hosting meta, removed when meta hosting moves. Provide > views on top tables by request counts, top meta rowkeys by request count, top > clients making requests by their hostname. > Can be implemented as a coprocessor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20787) Rebase the HBASE-18477 onto the current master to continue dev
[ https://issues.apache.org/jira/browse/HBASE-20787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-20787: -- Issue Type: Sub-task (was: Task) Parent: HBASE-18477 > Rebase the HBASE-18477 onto the current master to continue dev > -- > > Key: HBASE-20787 > URL: https://issues.apache.org/jira/browse/HBASE-20787 > Project: HBase > Issue Type: Sub-task >Affects Versions: HBASE-18477 >Reporter: Zach York >Assignee: Zach York >Priority: Minor > Fix For: HBASE-18477 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20787) Rebase the HBASE-18477 onto the current master to continue dev
Zach York created HBASE-20787: - Summary: Rebase the HBASE-18477 onto the current master to continue dev Key: HBASE-20787 URL: https://issues.apache.org/jira/browse/HBASE-20787 Project: HBase Issue Type: Task Affects Versions: HBASE-18477 Reporter: Zach York Assignee: Zach York Fix For: HBASE-18477 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20701) too much logging when balancer runs from BaseLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-20701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-20701:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> too much logging when balancer runs from BaseLoadBalancer
> -
>
> Key: HBASE-20701
> URL: https://issues.apache.org/jira/browse/HBASE-20701
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Reporter: Monani Mihir
>Assignee: Monani Mihir
>Priority: Trivial
> Fix For: 1.5.0, 1.3.3, 1.4.6
>
> Attachments: HBASE-20701-branch-1.3.patch,
> HBASE-20701-branch-1.3.patch, HBASE-20701-branch-1.3.patch,
> HBASE-20701-branch-1.4.patch, HBASE-20701-branch-1.4.patch,
> HBASE-20701.branch-1.001.patch
>
>
> When balancer runs, it tries to find least loaded server with better locality
> for current region. During this, we make debug level logging for each of
> those regions. It creates too much amount of logging at debug level , we
> should move this to trace level logging.
> {code:java}
> int getLeastLoadedTopServerForRegion (int region, int currentServer) {
> ...
> if (leastLoadedServerIndex != -1) {
> LOG.debug("Pick the least loaded server " +
> servers[leastLoadedServerIndex].getHostname()
> + " with better locality for region " + regions[region]);
> }
> ...
> }{code}
> This was fixed in branch-2.0 as part of -HBASE-14614-
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19722) Meta query statistics metrics source
[
https://issues.apache.org/jira/browse/HBASE-19722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523038#comment-16523038
]
Andrew Purtell commented on HBASE-19722:
The branch-1 patch fails this test
{noformat}
[INFO] Running org.apache.hadoop.hbase.coprocessor.TestMetaTableMetrics
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 28.562
s <<< FAILURE! - in org.apache.hadoop.hbase.coprocessor.TestMetaTableMetrics
[ERROR] test(org.apache.hadoop.hbase.coprocessor.TestMetaTableMetrics) Time
elapsed: 14.202 s <<< FAILURE!
java.lang.AssertionError
at
org.apache.hadoop.hbase.coprocessor.TestMetaTableMetrics.test(TestMetaTableMetrics.java:202)
{noformat}
> Meta query statistics metrics source
>
>
> Key: HBASE-19722
> URL: https://issues.apache.org/jira/browse/HBASE-19722
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Purtell
>Assignee: Xu Cang
>Priority: Major
> Attachments: HBASE-19722.branch-1.v001.patch,
> HBASE-19722.master.010.patch, HBASE-19722.master.011.patch,
> HBASE-19722.master.012.patch, HBASE-19722.master.013.patch,
> HBASE-19722.master.014.patch, HBASE-19722.master.015.patch,
> HBASE-19722.master.016.patch
>
>
> Implement a meta query statistics metrics source, created whenever a
> regionserver starts hosting meta, removed when meta hosting moves. Provide
> views on top tables by request counts, top meta rowkeys by request count, top
> clients making requests by their hostname.
> Can be implemented as a coprocessor.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20781) Save recalculating families in a WALEdit batch of Cells
[ https://issues.apache.org/jira/browse/HBASE-20781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522907#comment-16522907 ] stack commented on HBASE-20781: --- So, FSWALEntry should be good but perhaps you are worried about WALEdit [~Apache9]. Yes, CPs can add edits for inclusion in a WALEdit and they *could* be edits that have a Column Family outside of the current transaction Set. Let me accommodate inside this hack by making it so we auto-accumulate families even if the custom add done for CPs. Related, there is HBASE-19134, Make WALKey an Interface; expose Read-Only version to CPs, which is in 2.0.0. WALKey is read-only. CPs cannot #add directly to WALEdit which means we should be good with our internal handling of adds to WALEdit. > Save recalculating families in a WALEdit batch of Cells > --- > > Key: HBASE-20781 > URL: https://issues.apache.org/jira/browse/HBASE-20781 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Major > Attachments: 2.0621.2.12782.cpu.svg, 2.0621.256k.51351.cpu.svg, > 2.0623.families.121250.cpu.svg, HBASE-20781.branch-2.0.001.patch, > HBASE-20781.master.001.patch > > > Doing a doMiniBatchMutate, we calculate the set of families that the WALEdit > Cells belong to up front but down after the RingBuffer when we make an > FSWALEdit, we spin through all the Cells again to figure the set of families > in a particularly painful way. Just pass the calculated family set in the > WALEdit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20631) B: Merge command enhancements
[
https://issues.apache.org/jira/browse/HBASE-20631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523002#comment-16523002
]
Hadoop QA commented on HBASE-20631:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
40s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} hbase-backup: The patch generated 1 new + 3 unchanged
- 0 fixed = 4 total (was 3) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
37s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 20s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
27s{color} | {color:green} hbase-backup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20631 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12929104/HBASE-20631-v3.patch |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux e772cee1a6f8 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4ba6242a62 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/13384/artifact/patchprocess/diff-checkstyle-hbase-backup.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13384/testReport/ |
| Max. process+thread count | 4196 (vs. ulimit of 1) |
| modules | C: hbase-backup U: hbase-backup |
| Console output |
[jira] [Commented] (HBASE-20781) Save recalculating families in a WALEdit batch of Cells
[ https://issues.apache.org/jira/browse/HBASE-20781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522909#comment-16522909 ] stack commented on HBASE-20781: --- [^HBASE-20781.branch-2.0.001.patch] accumulates families in the WALEdit even if a CP adds Cells that have families outside of those in the current transaction set. Added some cleanup too of WALEdit (it could do w/ more). Removed some duplication of WALEdit methods. > Save recalculating families in a WALEdit batch of Cells > --- > > Key: HBASE-20781 > URL: https://issues.apache.org/jira/browse/HBASE-20781 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Major > Attachments: 2.0621.2.12782.cpu.svg, 2.0621.256k.51351.cpu.svg, > 2.0623.families.121250.cpu.svg, HBASE-20781.branch-2.0.001.patch, > HBASE-20781.master.001.patch > > > Doing a doMiniBatchMutate, we calculate the set of families that the WALEdit > Cells belong to up front but down after the RingBuffer when we make an > FSWALEdit, we spin through all the Cells again to figure the set of families > in a particularly painful way. Just pass the calculated family set in the > WALEdit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20786) Table create with thousands of regions takes too long
[ https://issues.apache.org/jira/browse/HBASE-20786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522941#comment-16522941 ] stack commented on HBASE-20786: --- bq. Does all the creation happen on master? Yes. bq. Can it be farmed out to region servers? RSs are participating in region deploy. The creation of regions is bottlenecked on back and forth w/ NN. M has an executor running so can do a bunch of creates against the NN at a time. Could farm this out but RS will just be asking NN. bq. Is this a result of all splits needing to go through master becoming a bottleneck? M is running the process so is likely bottleneck. But, we've not spent time on this. Lets dig in on where we are bottlenecking. Let me add some more notes. > Table create with thousands of regions takes too long > - > > Key: HBASE-20786 > URL: https://issues.apache.org/jira/browse/HBASE-20786 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Priority: Major > > Internal testing has create of a table with 33k regions taking 18 minutes. > Let me provide more info below. We have an executor with default ten threads > handling the creation of the regions in HDFS which helps distribute out the > load but its not enough. This cluster had >600 servers. Let me add detail. > Need to spend some time on speeding up create/assigns. Made this an umbrella > issue so can pick off pieces of the problem as subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20780) ServerRpcConnection logging cleanup
[ https://issues.apache.org/jira/browse/HBASE-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20780: -- Resolution: Fixed Fix Version/s: 2.0.2 2.1.0 3.0.0 Status: Resolved (was: Patch Available) Pushed logging change to branch-2.0+ > ServerRpcConnection logging cleanup > --- > > Key: HBASE-20780 > URL: https://issues.apache.org/jira/browse/HBASE-20780 > Project: HBase > Issue Type: Sub-task > Components: logging, Performance >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.0.2 > > Attachments: 2.0621.2.12782.lock.svg, 2.0621.2M.46340.lock.svg, > 2.0623.111354.lock.svg, HBASE-20780.branch-2.0.001.patch, > HBASE-20780.branch-2.0.001.patch > > > The logging we do inside in connection header parsing shows as worst offender > in the perf locking profiles. Its odd, but easy to clean up. I'd doubt it > makes any difference in throughput but lets get it out of the way. Let me > load up a few samples of what it current looks like. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20701) too much logging when balancer runs from BaseLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-20701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522999#comment-16522999
]
Hudson commented on HBASE-20701:
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #424 (See
[https://builds.apache.org/job/HBase-1.3-IT/424/])
HBASE-20701 too much logging when balancer runs from BaseLoadBalancer
(apurtell: rev 3a5d00398df14a10dcb1a95df0c6c68a2bafd8e9)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
> too much logging when balancer runs from BaseLoadBalancer
> -
>
> Key: HBASE-20701
> URL: https://issues.apache.org/jira/browse/HBASE-20701
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Reporter: Monani Mihir
>Assignee: Monani Mihir
>Priority: Trivial
> Fix For: 1.5.0, 1.3.3, 1.4.6
>
> Attachments: HBASE-20701-branch-1.3.patch,
> HBASE-20701-branch-1.3.patch, HBASE-20701-branch-1.3.patch,
> HBASE-20701-branch-1.4.patch, HBASE-20701-branch-1.4.patch,
> HBASE-20701.branch-1.001.patch
>
>
> When balancer runs, it tries to find least loaded server with better locality
> for current region. During this, we make debug level logging for each of
> those regions. It creates too much amount of logging at debug level , we
> should move this to trace level logging.
> {code:java}
> int getLeastLoadedTopServerForRegion (int region, int currentServer) {
> ...
> if (leastLoadedServerIndex != -1) {
> LOG.debug("Pick the least loaded server " +
> servers[leastLoadedServerIndex].getHostname()
> + " with better locality for region " + regions[region]);
> }
> ...
> }{code}
> This was fixed in branch-2.0 as part of -HBASE-14614-
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20788) Write up a doc about how to rolling upgrade from 1.x to 2.x
[ https://issues.apache.org/jira/browse/HBASE-20788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20788: -- Priority: Blocker (was: Major) > Write up a doc about how to rolling upgrade from 1.x to 2.x > --- > > Key: HBASE-20788 > URL: https://issues.apache.org/jira/browse/HBASE-20788 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Duo Zhang >Priority: Blocker > Fix For: 2.1.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20788) Write up a doc about how to rolling upgrade from 1.x to 2.x
[ https://issues.apache.org/jira/browse/HBASE-20788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20788: -- Fix Version/s: 2.1.0 > Write up a doc about how to rolling upgrade from 1.x to 2.x > --- > > Key: HBASE-20788 > URL: https://issues.apache.org/jira/browse/HBASE-20788 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Duo Zhang >Priority: Blocker > Fix For: 2.1.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20788) Write up a doc about how to rolling upgrade from 1.x to 2.x
[ https://issues.apache.org/jira/browse/HBASE-20788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20788: -- Component/s: documentation > Write up a doc about how to rolling upgrade from 1.x to 2.x > --- > > Key: HBASE-20788 > URL: https://issues.apache.org/jira/browse/HBASE-20788 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Duo Zhang >Priority: Major > Fix For: 2.1.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-20788) Write up a doc about how to rolling upgrade from 1.x to 2.x
[ https://issues.apache.org/jira/browse/HBASE-20788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HBASE-20788: - Assignee: Duo Zhang > Write up a doc about how to rolling upgrade from 1.x to 2.x > --- > > Key: HBASE-20788 > URL: https://issues.apache.org/jira/browse/HBASE-20788 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 2.1.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20785) NPE getting metrics in PE testing scans
[
https://issues.apache.org/jira/browse/HBASE-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523078#comment-16523078
]
Hadoop QA commented on HBASE-20785:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-1.4 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
46s{color} | {color:green} branch-1.4 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} branch-1.4 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} branch-1.4 passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} branch-1.4 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
18s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} branch-1.4 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} branch-1.4 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
11s{color} | {color:red} hbase-server: The patch generated 1 new + 30 unchanged
- 0 fixed = 31 total (was 30) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
12s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
7m 17s{color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.5 2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 98m 38s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}127m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:4b0a87a |
| JIRA Issue | HBASE-20785 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12929107/HBASE-20785.branch-1.4.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux f015778b0c5f 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016
[jira] [Updated] (HBASE-20631) B: Merge command enhancements
[ https://issues.apache.org/jira/browse/HBASE-20631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-20631: -- Attachment: HBASE-20631-v3.patch > B: Merge command enhancements > > > Key: HBASE-20631 > URL: https://issues.apache.org/jira/browse/HBASE-20631 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Major > Attachments: HBASE-20631-v1.patch, HBASE-20631-v2.patch, > HBASE-20631-v3.patch > > > Currently, merge supports only list of backup ids, which users must provide. > Date range merges seem more convenient for users. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20786) Table create with thousands of regions takes too long
[ https://issues.apache.org/jira/browse/HBASE-20786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522942#comment-16522942 ] stack commented on HBASE-20786: --- oh... nvm. Details have log-rolled away. Will try and run a new create... will be back. > Table create with thousands of regions takes too long > - > > Key: HBASE-20786 > URL: https://issues.apache.org/jira/browse/HBASE-20786 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Priority: Major > > Internal testing has create of a table with 33k regions taking 18 minutes. > Let me provide more info below. We have an executor with default ten threads > handling the creation of the regions in HDFS which helps distribute out the > load but its not enough. This cluster had >600 servers. Let me add detail. > Need to spend some time on speeding up create/assigns. Made this an umbrella > issue so can pick off pieces of the problem as subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20777) RpcConnection could still remain opened after we shutdown the NettyRpcServer
[
https://issues.apache.org/jira/browse/HBASE-20777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523004#comment-16523004
]
Duo Zhang commented on HBASE-20777:
---
TestAsyncTableBatch is fine now. Let me pushed to all branches which have netty
rpc server.
But there is another problem
https://builds.apache.org/job/HBASE-Flaky-Tests/33682/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.replication.multiwal.TestReplicationKillMasterRSCompressedWithMultipleAsyncWAL-output.txt/*view*/
{noformat}
2018-06-25 16:36:04,306 DEBUG [master/asf911:0.Chore.1]
client.ResultBoundedCompletionService(226): Replica 0 returns
java.net.SocketTimeoutException: callTimeout=6, callDuration=68578: Call to
asf911.gq1.ygridcore.net/67.195.81.155:55296 failed on connection exception:
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException:
syscall:getsockopt(..) failed: Connection refused:
asf911.gq1.ygridcore.net/67.195.81.155:55296 row '' on table 'hbase:meta' at
region=hbase:meta,,1.1588230740,
hostname=asf911.gq1.ygridcore.net,55296,1529944208029, seqNum=-1
java.net.SocketTimeoutException: callTimeout=6, callDuration=68578: Call to
asf911.gq1.ygridcore.net/67.195.81.155:55296 failed on connection exception:
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException:
syscall:getsockopt(..) failed: Connection refused:
asf911.gq1.ygridcore.net/67.195.81.155:55296 row '' on table 'hbase:meta' at
region=hbase:meta,,1.1588230740,
hostname=asf911.gq1.ygridcore.net,55296,1529944208029, seqNum=-1
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:158)
at
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Call to
asf911.gq1.ygridcore.net/67.195.81.155:55296 failed on connection exception:
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException:
syscall:getsockopt(..) failed: Connection refused:
asf911.gq1.ygridcore.net/67.195.81.155:55296
at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:165)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406)
at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)
at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)
at
org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)
at
org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1377)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)
at
org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)
at
org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:929)
at
org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:179)
at
org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$500(NettyRpcConnection.java:71)
at
org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:267)
at
org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:261)
at
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
at
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)
at
[jira] [Updated] (HBASE-20777) RpcConnection could still remain opened after we shutdown the NettyRpcServer
[ https://issues.apache.org/jira/browse/HBASE-20777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20777: -- Component/s: rpc > RpcConnection could still remain opened after we shutdown the NettyRpcServer > > > Key: HBASE-20777 > URL: https://issues.apache.org/jira/browse/HBASE-20777 > Project: HBase > Issue Type: Bug > Components: rpc >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.0.2, 2.2.0 > > Attachments: HBASE-20777-v1.patch, HBASE-20777.patch, > org.apache.hadoop.hbase.client.TestAsyncTableBatch-output.txt > > > The log is very strange, we keep sending request to a dead RS, and the result > is not connection refused, but rpc timeout, and later it becomes > CallQueueTooBig... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20777) RpcConnection could still remain opened after we shutdown the NettyRpcServer
[ https://issues.apache.org/jira/browse/HBASE-20777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20777: -- Resolution: Fixed Fix Version/s: 2.2.0 2.0.2 2.1.0 3.0.0 Status: Resolved (was: Patch Available) Pushed to branch-2.0+. > RpcConnection could still remain opened after we shutdown the NettyRpcServer > > > Key: HBASE-20777 > URL: https://issues.apache.org/jira/browse/HBASE-20777 > Project: HBase > Issue Type: Bug > Components: rpc >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.1.0, 2.0.2, 2.2.0 > > Attachments: HBASE-20777-v1.patch, HBASE-20777.patch, > org.apache.hadoop.hbase.client.TestAsyncTableBatch-output.txt > > > The log is very strange, we keep sending request to a dead RS, and the result > is not connection refused, but rpc timeout, and later it becomes > CallQueueTooBig... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-18366) Fix flaky test hbase.master.procedure.TestServerCrashProcedure#testRecoveryAndDoubleExecutionOnRsWithMeta
[ https://issues.apache.org/jira/browse/HBASE-18366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Umesh Agashe resolved HBASE-18366. -- Resolution: Not A Problem Not flaky anymore. Fixed by other JIRAs. > Fix flaky test > hbase.master.procedure.TestServerCrashProcedure#testRecoveryAndDoubleExecutionOnRsWithMeta > - > > Key: HBASE-18366 > URL: https://issues.apache.org/jira/browse/HBASE-18366 > Project: HBase > Issue Type: Bug >Reporter: Umesh Agashe >Assignee: Umesh Agashe >Priority: Major > Attachments: hbase-18366.fix1.patch, hbase-18366.fix2.patch > > > It worked for a few days after enabling it with HBASE-18278. But started > failing after commits: > 6786b2b > 68436c9 > 75d2eca > 50bb045 > df93c13 > It works with one commit before: c5abb6c. Need to see what changed with those > commits. > Currently it fails with TableNotFoundException. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20763) Update guava >=24.1.1
[ https://issues.apache.org/jira/browse/HBASE-20763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522674#comment-16522674 ] stack commented on HBASE-20763: --- Yeah. +1. Lets see if we can update other stuff... For 2.1.0? pb is now 3.6 (we are at 3.3.4). Some of the changes look helpful: e.g. 3.6.0 "Added a UTF-8 decoder that uses Unsafe to directly decode a byte buffer." 3.4.0 "Optimized CodedInputStream to do less copies when parsing large bytes fields." Netty is now 4.1.25 (we are 4.1.12). > Update guava >=24.1.1 > - > > Key: HBASE-20763 > URL: https://issues.apache.org/jira/browse/HBASE-20763 > Project: HBase > Issue Type: Task > Components: thirdparty >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: thirdparty-2.2.0 > > Attachments: HBASE-20763.001.patch > > > We should update Guava in hbase-thirdparty to stop shipping the code cited as > vulnerable in CVE-2018-10237. We do not invoke this code ourselves and users > would have to try pretty hard to use it themselves, but we've seen more > strange things before ;) > Let's just bump up the dependency and move on. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20785) NPE getting metrics in PE testing scans
[ https://issues.apache.org/jira/browse/HBASE-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20785: -- Attachment: HBASE-20785.branch-1.4.001.patch > NPE getting metrics in PE testing scans > --- > > Key: HBASE-20785 > URL: https://issues.apache.org/jira/browse/HBASE-20785 > Project: HBase > Issue Type: Bug > Components: Performance >Affects Versions: 1.4.4 >Reporter: stack >Assignee: stack >Priority: Major > Attachments: HBASE-20785.branch-1.4.001.patch > > > Comparing scans using PE. In branch-1 at least, I was getting an NPE when we > tried to use a null metrics instance. Seems transient around startup. > One-liner patch coming. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20785) NPE getting metrics in PE testing scans
[ https://issues.apache.org/jira/browse/HBASE-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20785: -- Status: Patch Available (was: Open) > NPE getting metrics in PE testing scans > --- > > Key: HBASE-20785 > URL: https://issues.apache.org/jira/browse/HBASE-20785 > Project: HBase > Issue Type: Bug > Components: Performance >Affects Versions: 1.4.4 >Reporter: stack >Assignee: stack >Priority: Major > Attachments: HBASE-20785.branch-1.4.001.patch > > > Comparing scans using PE. In branch-1 at least, I was getting an NPE when we > tried to use a null metrics instance. Seems transient around startup. > One-liner patch coming. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[
https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522717#comment-16522717
]
Andrew Purtell commented on HBASE-20403:
Thanks for the fix and the commit [~tlipcon]
> Prefetch sometimes doesn't work with encrypted file system
> --
>
> Key: HBASE-20403
> URL: https://issues.apache.org/jira/browse/HBASE-20403
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-beta-2
>Reporter: Umesh Agashe
>Assignee: Todd Lipcon
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.2
>
> Attachments: hbase-20403.patch, hbase-20403.patch
>
>
> Log from long running test has following stack trace a few times:
> {code}
> 2018-04-09 18:33:21,523 WARN
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch
> path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988,
> offset=36884200, end=231005989
> java.lang.IllegalArgumentException
> at java.nio.Buffer.limit(Buffer.java:275)
> at
> org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183)
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831)
> at
> org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Size on disk calculations seem to get messed up due to encryption. Possible
> fixes can be:
> * if file is encrypted with FileStatus#isEncrypted() and do not prefetch.
> * document that hbase.rs.prefetchblocksonopen cannot be true if file is
> encrypted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20781) Save recalculating families in a WALEdit batch of Cells
[ https://issues.apache.org/jira/browse/HBASE-20781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522787#comment-16522787 ] stack commented on HBASE-20781: --- bq. IIRC the extra code which is used to construct the families for FSWALEntry is introduced by per column family flush. I can remember it because I'm sure that I also wanted to eliminate it in the first place but finally gave up. Maybe the problem is in CP we can add or remove cells in WALEdit? Not sure, long time ago... Thanks [~Apache9] You thinking we should keep this extra iteration? Should be fine if CPs add edits for families already in the WALEdit. If they want to add edits for column families not in current batch, they can update the WALEdit Set of families too? Let me add a note. A note is not much by way of protection but I think better to have CPs suffer than pay this CPU on each and every edit? > Save recalculating families in a WALEdit batch of Cells > --- > > Key: HBASE-20781 > URL: https://issues.apache.org/jira/browse/HBASE-20781 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Major > Attachments: 2.0621.2.12782.cpu.svg, 2.0621.256k.51351.cpu.svg, > 2.0623.families.121250.cpu.svg, HBASE-20781.master.001.patch > > > Doing a doMiniBatchMutate, we calculate the set of families that the WALEdit > Cells belong to up front but down after the RingBuffer when we make an > FSWALEdit, we spin through all the Cells again to figure the set of families > in a particularly painful way. Just pass the calculated family set in the > WALEdit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20781) Save recalculating families in a WALEdit batch of Cells
[ https://issues.apache.org/jira/browse/HBASE-20781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522804#comment-16522804 ] stack commented on HBASE-20781: --- FSWALEntry is @InterfaceAudience.Private FSWALEntry is not passed to CPs anywhere that I can see; its an internal object. Therefore we should be find w/ this savings. Let me upload new patch w/ some extra doc and a fix for checkstyle. > Save recalculating families in a WALEdit batch of Cells > --- > > Key: HBASE-20781 > URL: https://issues.apache.org/jira/browse/HBASE-20781 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Major > Attachments: 2.0621.2.12782.cpu.svg, 2.0621.256k.51351.cpu.svg, > 2.0623.families.121250.cpu.svg, HBASE-20781.master.001.patch > > > Doing a doMiniBatchMutate, we calculate the set of families that the WALEdit > Cells belong to up front but down after the RingBuffer when we make an > FSWALEdit, we spin through all the Cells again to figure the set of families > in a particularly painful way. Just pass the calculated family set in the > WALEdit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18201) add UT and docs for DataBlockEncodingTool
[
https://issues.apache.org/jira/browse/HBASE-18201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522639#comment-16522639
]
Hadoop QA commented on HBASE-18201:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 5m
9s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
25s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 5m
1s{color} | {color:blue} patch has no errors when building the reference guide.
See footer for rendered docs, which you should manually inspect. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
30s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 54s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}184m
4s{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}257m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-18201 |
| JIRA Patch URL |
[jira] [Commented] (HBASE-20666) Unsuccessful table creation leaves entry in rsgroup meta table
[
https://issues.apache.org/jira/browse/HBASE-20666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522691#comment-16522691
]
Andrew Purtell commented on HBASE-20666:
[~nihaljain.cs] I think it's ok to proceed
> Unsuccessful table creation leaves entry in rsgroup meta table
> --
>
> Key: HBASE-20666
> URL: https://issues.apache.org/jira/browse/HBASE-20666
> Project: HBase
> Issue Type: Bug
>Reporter: Biju Nair
>Priority: Minor
>
> If a table creation fails in a cluster enabled with {{rsgroup}} feature, the
> table is still listed as part of {{default}} rsgroup.
> To recreate the scenario:
> - Create a namespace (NS) with number of region limit
> - Create table in the NS which satisfies the region limit by pre-splitting
> - Create a new table in the NS which will fail
> - {{list_rsgroup}} will show the table being part of {{default}} rsgroup and
> data can be found in {{hbase:rsgroup}} table
> Would be good to revert the entry when the table creation fails or a script
> to clean up the metadata.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522698#comment-16522698 ] Andrew Purtell commented on HBASE-20704: bq. For 1.x in the small window the region hasn't be set a closed by the RS yet the client will get an NPE when the scan tries to access the fs stream. The client will retry. My opinion is a NPE is always a bug. Ideally we do something more graceful than propagate it to the client for a retry. Even simply substituting an IOException (with suitably descriptive message) would be better. > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20728) Failure and recovery of all RSes in a RSgroup requires master restart for region assignments
[
https://issues.apache.org/jira/browse/HBASE-20728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522713#comment-16522713
]
Andrew Purtell commented on HBASE-20728:
[~jatsakthi] Please go ahead. It looks ok to work on this.
> Failure and recovery of all RSes in a RSgroup requires master restart for
> region assignments
>
>
> Key: HBASE-20728
> URL: https://issues.apache.org/jira/browse/HBASE-20728
> Project: HBase
> Issue Type: Bug
> Components: master, rsgroup
>Reporter: Biju Nair
>Assignee: Sakthi
>Priority: Minor
>
> If all the RSes in a RSgroup hosting user tables fail and recover, master
> still looks for old RSes (with old timestamp in the RS identifier) to assign
> regions. i.e. Regions are left in transition making the tables in the RSGroup
> unavailable. User need to restart {{master}} or manually assign the regions
> to make the tables available. Steps to recreate the scenario in a local
> cluster
> - Add required properties to {{site.xml}} to enable {{rsgroup}} and start
> hbase
> - Bring up multiple region servers using {{local-regionservers.sh start}}
> - Create a {{rsgroup}} and move a subset of {{regionservers}} to the group
> - Create a table, move it to the group and put some data
> - Stop the {{regionservers}} in the group and restart them
> - From the {{master UI}}, we can see that the region for the table in
> transition and the RS name in the {{RIT}} message has the old timestamp.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[
https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522725#comment-16522725
]
Todd Lipcon commented on HBASE-20403:
-
I would guess it's not affected because it has locking in the file reader path.
The locking was removed by HBASE-17917 in 2.0
> Prefetch sometimes doesn't work with encrypted file system
> --
>
> Key: HBASE-20403
> URL: https://issues.apache.org/jira/browse/HBASE-20403
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-beta-2
>Reporter: Umesh Agashe
>Assignee: Todd Lipcon
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.2
>
> Attachments: hbase-20403.patch, hbase-20403.patch
>
>
> Log from long running test has following stack trace a few times:
> {code}
> 2018-04-09 18:33:21,523 WARN
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch
> path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988,
> offset=36884200, end=231005989
> java.lang.IllegalArgumentException
> at java.nio.Buffer.limit(Buffer.java:275)
> at
> org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183)
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831)
> at
> org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Size on disk calculations seem to get messed up due to encryption. Possible
> fixes can be:
> * if file is encrypted with FileStatus#isEncrypted() and do not prefetch.
> * document that hbase.rs.prefetchblocksonopen cannot be true if file is
> encrypted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[
https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522743#comment-16522743
]
Andrew Purtell commented on HBASE-20403:
I don't believe branch-1 is affected because of locking in hfileblock.
> Prefetch sometimes doesn't work with encrypted file system
> --
>
> Key: HBASE-20403
> URL: https://issues.apache.org/jira/browse/HBASE-20403
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-beta-2
>Reporter: Umesh Agashe
>Assignee: Todd Lipcon
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.2
>
> Attachments: hbase-20403.patch, hbase-20403.patch
>
>
> Log from long running test has following stack trace a few times:
> {code}
> 2018-04-09 18:33:21,523 WARN
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch
> path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988,
> offset=36884200, end=231005989
> java.lang.IllegalArgumentException
> at java.nio.Buffer.limit(Buffer.java:275)
> at
> org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183)
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831)
> at
> org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Size on disk calculations seem to get messed up due to encryption. Possible
> fixes can be:
> * if file is encrypted with FileStatus#isEncrypted() and do not prefetch.
> * document that hbase.rs.prefetchblocksonopen cannot be true if file is
> encrypted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20786) Table create with thousands of regions takes too long
stack created HBASE-20786: - Summary: Table create with thousands of regions takes too long Key: HBASE-20786 URL: https://issues.apache.org/jira/browse/HBASE-20786 Project: HBase Issue Type: Umbrella Components: Performance Reporter: stack Internal testing has create of a table with 33k regions taking 18 minutes. Let me provide more info below. We have an executor with default ten threads handling the creation of the regions in HDFS which helps distribute out the load but its not enough. This cluster had >600 servers. Let me add detail. Need to spend some time on speeding up create/assigns. Made this an umbrella issue so can pick off pieces of the problem as subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20701) too much logging when balancer runs from BaseLoadBalancer
[
https://issues.apache.org/jira/browse/HBASE-20701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-20701:
---
Fix Version/s: 1.4.6
1.3.3
1.5.0
+1
> too much logging when balancer runs from BaseLoadBalancer
> -
>
> Key: HBASE-20701
> URL: https://issues.apache.org/jira/browse/HBASE-20701
> Project: HBase
> Issue Type: Improvement
> Components: Balancer
>Reporter: Monani Mihir
>Assignee: Monani Mihir
>Priority: Trivial
> Fix For: 1.5.0, 1.3.3, 1.4.6
>
> Attachments: HBASE-20701-branch-1.3.patch,
> HBASE-20701-branch-1.3.patch, HBASE-20701-branch-1.3.patch,
> HBASE-20701-branch-1.4.patch, HBASE-20701-branch-1.4.patch,
> HBASE-20701.branch-1.001.patch
>
>
> When balancer runs, it tries to find least loaded server with better locality
> for current region. During this, we make debug level logging for each of
> those regions. It creates too much amount of logging at debug level , we
> should move this to trace level logging.
> {code:java}
> int getLeastLoadedTopServerForRegion (int region, int currentServer) {
> ...
> if (leastLoadedServerIndex != -1) {
> LOG.debug("Pick the least loaded server " +
> servers[leastLoadedServerIndex].getHostname()
> + " with better locality for region " + regions[region]);
> }
> ...
> }{code}
> This was fixed in branch-2.0 as part of -HBASE-14614-
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20785) NPE getting metrics in PE testing scans
stack created HBASE-20785: - Summary: NPE getting metrics in PE testing scans Key: HBASE-20785 URL: https://issues.apache.org/jira/browse/HBASE-20785 Project: HBase Issue Type: Bug Components: Performance Affects Versions: 1.4.4 Reporter: stack Assignee: stack Comparing scans using PE. In branch-1 at least, I was getting an NPE when we tried to use a null metrics instance. Seems transient around startup. One-liner patch coming. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[
https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522693#comment-16522693
]
Mike Drob commented on HBASE-20403:
---
bq. Got a report back from an internal test cluster who was previously
reproducing this issue. With this patch applied the issue seems to be resolved.
Sounds good, you still have commit access, right? ;)
+1
> Prefetch sometimes doesn't work with encrypted file system
> --
>
> Key: HBASE-20403
> URL: https://issues.apache.org/jira/browse/HBASE-20403
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-beta-2
>Reporter: Umesh Agashe
>Assignee: Todd Lipcon
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: hbase-20403.patch, hbase-20403.patch
>
>
> Log from long running test has following stack trace a few times:
> {code}
> 2018-04-09 18:33:21,523 WARN
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch
> path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988,
> offset=36884200, end=231005989
> java.lang.IllegalArgumentException
> at java.nio.Buffer.limit(Buffer.java:275)
> at
> org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183)
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831)
> at
> org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Size on disk calculations seem to get messed up due to encryption. Possible
> fixes can be:
> * if file is encrypted with FileStatus#isEncrypted() and do not prefetch.
> * document that hbase.rs.prefetchblocksonopen cannot be true if file is
> encrypted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[
https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HBASE-20403:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.0.2
2.1.0
Status: Resolved (was: Patch Available)
Committed to master, branch-2, branch-2.1, branch-2.0. Appears my commit access
still works after 6 years! Thanks for the review.
> Prefetch sometimes doesn't work with encrypted file system
> --
>
> Key: HBASE-20403
> URL: https://issues.apache.org/jira/browse/HBASE-20403
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-beta-2
>Reporter: Umesh Agashe
>Assignee: Todd Lipcon
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.2
>
> Attachments: hbase-20403.patch, hbase-20403.patch
>
>
> Log from long running test has following stack trace a few times:
> {code}
> 2018-04-09 18:33:21,523 WARN
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch
> path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988,
> offset=36884200, end=231005989
> java.lang.IllegalArgumentException
> at java.nio.Buffer.limit(Buffer.java:275)
> at
> org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183)
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831)
> at
> org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Size on disk calculations seem to get messed up due to encryption. Possible
> fixes can be:
> * if file is encrypted with FileStatus#isEncrypted() and do not prefetch.
> * document that hbase.rs.prefetchblocksonopen cannot be true if file is
> encrypted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[
https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522721#comment-16522721
]
Mike Drob commented on HBASE-20403:
---
is branch-1 affected by this also? I'd imagine yes since I don't think the scan
code has been rewritten, but I also would have expected to see this before if
so.
> Prefetch sometimes doesn't work with encrypted file system
> --
>
> Key: HBASE-20403
> URL: https://issues.apache.org/jira/browse/HBASE-20403
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-beta-2
>Reporter: Umesh Agashe
>Assignee: Todd Lipcon
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.2
>
> Attachments: hbase-20403.patch, hbase-20403.patch
>
>
> Log from long running test has following stack trace a few times:
> {code}
> 2018-04-09 18:33:21,523 WARN
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch
> path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988,
> offset=36884200, end=231005989
> java.lang.IllegalArgumentException
> at java.nio.Buffer.limit(Buffer.java:275)
> at
> org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183)
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831)
> at
> org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Size on disk calculations seem to get messed up due to encryption. Possible
> fixes can be:
> * if file is encrypted with FileStatus#isEncrypted() and do not prefetch.
> * document that hbase.rs.prefetchblocksonopen cannot be true if file is
> encrypted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20786) Table create with thousands of regions takes too long
[ https://issues.apache.org/jira/browse/HBASE-20786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522820#comment-16522820 ] Mike Drob commented on HBASE-20786: --- Does all the creation happen on master? Can it be farmed out to region servers? Is this a result of all splits needing to go through master becoming a bottleneck? > Table create with thousands of regions takes too long > - > > Key: HBASE-20786 > URL: https://issues.apache.org/jira/browse/HBASE-20786 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Priority: Major > > Internal testing has create of a table with 33k regions taking 18 minutes. > Let me provide more info below. We have an executor with default ten threads > handling the creation of the regions in HDFS which helps distribute out the > load but its not enough. This cluster had >600 servers. Let me add detail. > Need to spend some time on speeding up create/assigns. Made this an umbrella > issue so can pick off pieces of the problem as subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19722) Implement a meta query statistics metrics source
[ https://issues.apache.org/jira/browse/HBASE-19722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522822#comment-16522822 ] Andrew Purtell commented on HBASE-19722: Ok, let's bring this one home. Committing shortly. > Implement a meta query statistics metrics source > > > Key: HBASE-19722 > URL: https://issues.apache.org/jira/browse/HBASE-19722 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Xu Cang >Priority: Major > Attachments: HBASE-19722.branch-1.v001.patch, > HBASE-19722.master.010.patch, HBASE-19722.master.011.patch, > HBASE-19722.master.012.patch, HBASE-19722.master.013.patch, > HBASE-19722.master.014.patch, HBASE-19722.master.015.patch, > HBASE-19722.master.016.patch > > > Implement a meta query statistics metrics source, created whenever a > regionserver starts hosting meta, removed when meta hosting moves. Provide > views on top tables by request counts, top meta rowkeys by request count, top > clients making requests by their hostname. > Can be implemented as a coprocessor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20553) Add dependency CVE checking to nightly tests
[
https://issues.apache.org/jira/browse/HBASE-20553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522665#comment-16522665
]
Sakthi commented on HBASE-20553:
Sure. Makes sense.
> Add dependency CVE checking to nightly tests
>
>
> Key: HBASE-20553
> URL: https://issues.apache.org/jira/browse/HBASE-20553
> Project: HBase
> Issue Type: Umbrella
> Components: dependencies
>Affects Versions: 3.0.0
>Reporter: Sean Busbey
>Assignee: Sakthi
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
>
> We should proactively work to flag dependencies with known CVEs so that we
> can then update them early in our development instead of near a release.
> YETUS-441 is working to add a plugin for this, we should grab a copy early to
> make sure it works for us.
> Rough outline:
> 1. [install yetus locally|http://yetus.apache.org/downloads/]
> 2. [install the dependency-check
> cli|https://www.owasp.org/index.php/OWASP_Dependency_Check] (homebrew
> instructions on right hand margin)
> 3. Get a local copy of the OWASP datafile ({{dependency-check --updateonly
> --data /some/local/path/to/dir}})
> 4. Run {{hbase_nightly_yetus.sh}} using matching environment variables from
> the “yetus general check” (currently [line #126 in our nightly
> Jenkinsfile|https://github.com/apache/hbase/blob/master/dev-support/Jenkinsfile#L126])
> 5. Grab the plugin definition and suppression file from from YETUS-441
> 6. put the plugin definition either in a directory of dev-support or into the
> hbase-personality.sh directly
> 7. Re-run {{hbase_nightly_yetus.sh}} to verify that the plugin results show
> up. (Probably this will involve adding new pointers for “where is the
> suppression file”, “where is the OWASP datafile” and pointing them somewhere
> locally.)
> Once all of that is in place we’ll get the changes needed into a branch that
> we can test out. Over in YETUS-441 I’ll need to add a jenkins job that’ll
> handle periodically updating a copy of the datafile for the OWASP dependency
> checker. Presuming I have that in place by the time we have a nightly branch
> to check this out, then we’ll also need to update our nightly Jenkinsfile to
> fetch the data file from that job.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20728) Failure and recovery of all RSes in a RSgroup requires master restart for region assignments
[
https://issues.apache.org/jira/browse/HBASE-20728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522714#comment-16522714
]
Sakthi commented on HBASE-20728:
Sure [~apurtell]
> Failure and recovery of all RSes in a RSgroup requires master restart for
> region assignments
>
>
> Key: HBASE-20728
> URL: https://issues.apache.org/jira/browse/HBASE-20728
> Project: HBase
> Issue Type: Bug
> Components: master, rsgroup
>Reporter: Biju Nair
>Assignee: Sakthi
>Priority: Minor
>
> If all the RSes in a RSgroup hosting user tables fail and recover, master
> still looks for old RSes (with old timestamp in the RS identifier) to assign
> regions. i.e. Regions are left in transition making the tables in the RSGroup
> unavailable. User need to restart {{master}} or manually assign the regions
> to make the tables available. Steps to recreate the scenario in a local
> cluster
> - Add required properties to {{site.xml}} to enable {{rsgroup}} and start
> hbase
> - Bring up multiple region servers using {{local-regionservers.sh start}}
> - Create a {{rsgroup}} and move a subset of {{regionservers}} to the group
> - Create a table, move it to the group and put some data
> - Stop the {{regionservers}} in the group and restart them
> - From the {{master UI}}, we can see that the region for the table in
> transition and the RS name in the {{RIT}} message has the old timestamp.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20785) NPE getting metrics in PE testing scans
[
https://issues.apache.org/jira/browse/HBASE-20785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522848#comment-16522848
]
Hadoop QA commented on HBASE-20785:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-1.4 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
59s{color} | {color:green} branch-1.4 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} branch-1.4 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} branch-1.4 passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} branch-1.4 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
41s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} branch-1.4 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} branch-1.4 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
15s{color} | {color:red} hbase-server: The patch generated 1 new + 30 unchanged
- 0 fixed = 31 total (was 30) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
23s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
7m 59s{color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.5 2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}101m 12s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}142m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mapreduce.TestLoadIncrementalHFiles |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:4b0a87a |
| JIRA Issue | HBASE-20785 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12929077/HBASE-20785.branch-1.4.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle
[jira] [Commented] (HBASE-19997) [rolling upgrade] 1.x => 2.x
[ https://issues.apache.org/jira/browse/HBASE-19997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522663#comment-16522663 ] stack commented on HBASE-19997: --- Wow! That is great news. The zkless assignment in hbase1 master supports hbase2 RS? Thats great (I'm surprised but in the best way -- smile). So, a hbase1 Master can assign regions to an hbase2 RS? Yeah, a write-up.. .even if it was just a few lines so can try it would be sweet. As others try it we can fill it out more. This is good news. > [rolling upgrade] 1.x => 2.x > > > Key: HBASE-19997 > URL: https://issues.apache.org/jira/browse/HBASE-19997 > Project: HBase > Issue Type: Umbrella >Reporter: stack >Priority: Blocker > Fix For: 2.1.0 > > Attachments: Screenshot from 2018-05-03 14-43-46.png > > > An umbrella issue of issues needed so folks can do a rolling upgrade from > hbase-1.x to hbase-2.x. > (Recent) Notables: > * hbase-1.x can't read hbase-2.x WALs -- hbase-1.x doesn't know the > AsyncProtobufLogWriter class used writing the WAL -- see > https://issues.apache.org/jira/browse/HBASE-19166?focusedCommentId=16362897=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16362897 > for exception. > ** Might be ok... means WAL split fails on an hbase1 RS... must wait till an > hbase-2.x RS picks up the WAL for it to be split. > * hbase-1 can't open regions from tables created by hbase-2; it can't find > the Table descriptor. See > https://issues.apache.org/jira/browse/HBASE-19116?focusedCommentId=16363276=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16363276 > ** This might be ok if the tables we are doing rolling upgrade over were > written with hbase-1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[
https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522682#comment-16522682
]
Todd Lipcon commented on HBASE-20403:
-
Got a report back from an internal test cluster who was previously reproducing
this issue. With this patch applied the issue seems to be resolved.
> Prefetch sometimes doesn't work with encrypted file system
> --
>
> Key: HBASE-20403
> URL: https://issues.apache.org/jira/browse/HBASE-20403
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-beta-2
>Reporter: Umesh Agashe
>Assignee: Todd Lipcon
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: hbase-20403.patch, hbase-20403.patch
>
>
> Log from long running test has following stack trace a few times:
> {code}
> 2018-04-09 18:33:21,523 WARN
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch
> path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988,
> offset=36884200, end=231005989
> java.lang.IllegalArgumentException
> at java.nio.Buffer.limit(Buffer.java:275)
> at
> org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183)
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831)
> at
> org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Size on disk calculations seem to get messed up due to encryption. Possible
> fixes can be:
> * if file is encrypted with FileStatus#isEncrypted() and do not prefetch.
> * document that hbase.rs.prefetchblocksonopen cannot be true if file is
> encrypted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20770) WAL cleaner logs way too much; gets clogged when lots of work to do
[
https://issues.apache.org/jira/browse/HBASE-20770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-20770:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Pushed to branch-2.0+. Thanks for review [~reidchan]
> WAL cleaner logs way too much; gets clogged when lots of work to do
> ---
>
> Key: HBASE-20770
> URL: https://issues.apache.org/jira/browse/HBASE-20770
> Project: HBase
> Issue Type: Bug
> Components: logging
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.2
>
> Attachments: HBASE-20770.branch-2.0.001.patch
>
>
> Been here before (HBASE-7214 and HBASE-19652). Testing on large cluster,
> Master log is in a continuous spew of logging output fililng disks. It is
> stuck making no progress but hard to tell because it is logging minutiae
> rather than a call on whats actually wrong.
> Log is full of this:
> {code}
> 2018-06-21 01:19:12,761 DEBUG
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/meta/bd49572de3914b66985fff5ea2ca7403
> 2018-06-21 01:19:12,761 DEBUG
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/meta/fad01294c6ca421db209e89b5b97d364
> 2018-06-21 01:19:12,823 WARN
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Wait more than 6 ms
> for deleting
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/d3f759d0495257fc1d33ae780b634455/tiny/b72bac4036444dcf9265c7b5664fd403,
> exit...
> 2018-06-21 01:19:12,823 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9
> 2018-06-21 01:19:12,824 WARN
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Wait more than 6 ms
> for deleting
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/2425053ad86823081b368e00bc471e56/tiny/6ea3cb1174434aecbc448e322e2a062c,
> exit...
> 2018-06-21 01:19:12,824 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/big
> 2018-06-21 01:19:12,824 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/tiny
> 2018-06-21 01:19:12,827 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta
> 2018-06-21 01:19:12,844 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0
> 2018-06-21 01:19:12,844 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c
> 2018-06-21 01:19:12,844 DEBUG
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta/90f21dec28d140cda48d37eeb44d37e8
> 2018-06-21 01:19:12,844 DEBUG
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta/8a4cf6410d5a4201963bc1415945f877
> 2018-06-21 01:19:12,848 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c/meta
> 2018-06-21 01:19:12,849 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0/meta
> 2018-06-21 01:19:12,927 DEBUG
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0/meta/6043fce5761e4479819b15405183f193
> 2018-06-21 01:19:12,927 DEBUG
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c/meta/69e6bf4650124859b2bc7ddf134be642
> 2018-06-21 01:19:13,011 DEBUG
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing
>
[jira] [Commented] (HBASE-20306) LoadTestTool does not print summary at end of run
[ https://issues.apache.org/jira/browse/HBASE-20306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522701#comment-16522701 ] Andrew Purtell commented on HBASE-20306: Any progress here? > LoadTestTool does not print summary at end of run > - > > Key: HBASE-20306 > URL: https://issues.apache.org/jira/browse/HBASE-20306 > Project: HBase > Issue Type: Bug > Components: tooling >Reporter: Mike Drob >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: beginner > > ltt currently prints status as it goes, but doesn't give a nice summary of > what happened so users have to infer it from the last status line printed. > Would be nice to print a real summary with statistics about what was run. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522698#comment-16522698 ] Andrew Purtell edited comment on HBASE-20704 at 6/25/18 7:13 PM: - bq. For 1.x in the small window the region hasn't be set a closed by the RS yet the client will get an NPE when the scan tries to access the fs stream. The client will retry. My opinion is a NPE is always a bug. Ideally we do something more graceful. Even simply substituting an IOException with suitably descriptive message would be better then have NullPointerException appear in a log somewhere. Guaranteed to draw attention/concern. was (Author: apurtell): bq. For 1.x in the small window the region hasn't be set a closed by the RS yet the client will get an NPE when the scan tries to access the fs stream. The client will retry. My opinion is a NPE is always a bug. Ideally we do something more graceful than propagate it to the client for a retry. Even simply substituting an IOException (with suitably descriptive message) would be better. > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19722) Meta query statistics metrics source
[ https://issues.apache.org/jira/browse/HBASE-19722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-19722: --- Summary: Meta query statistics metrics source (was: Implement a meta query statistics metrics source) > Meta query statistics metrics source > > > Key: HBASE-19722 > URL: https://issues.apache.org/jira/browse/HBASE-19722 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Xu Cang >Priority: Major > Attachments: HBASE-19722.branch-1.v001.patch, > HBASE-19722.master.010.patch, HBASE-19722.master.011.patch, > HBASE-19722.master.012.patch, HBASE-19722.master.013.patch, > HBASE-19722.master.014.patch, HBASE-19722.master.015.patch, > HBASE-19722.master.016.patch > > > Implement a meta query statistics metrics source, created whenever a > regionserver starts hosting meta, removed when meta hosting moves. Provide > views on top tables by request counts, top meta rowkeys by request count, top > clients making requests by their hostname. > Can be implemented as a coprocessor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19164) Avoid UUID.randomUUID in tests
[
https://issues.apache.org/jira/browse/HBASE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Aggarwal updated HBASE-19164:
---
Attachment: HBASE-19164.master.004.patch
> Avoid UUID.randomUUID in tests
> --
>
> Key: HBASE-19164
> URL: https://issues.apache.org/jira/browse/HBASE-19164
> Project: HBase
> Issue Type: Improvement
> Components: test
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>Priority: Major
> Labels: beginner
> Attachments: HBASE-19164.master.001.patch,
> HBASE-19164.master.002.patch, HBASE-19164.master.003.patch,
> HBASE-19164.master.004.patch
>
>
> We have a lot of places in our test code where we use {{UUID.randomUUID}} to
> generate table names or paths for uniqueness. Unfortunately, this uses up a
> good chunk of system entropy, since Sun chose that random UUID's should use
> the NativePRNGBlocking implementation.
> We don't need to block on entropy for random bits to pick a random table name
> in a test, so we can use something that doesn't strain the system too much -
> secure random can be a source of problems on some VM or containers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19164) Avoid UUID.randomUUID in tests
[
https://issues.apache.org/jira/browse/HBASE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523153#comment-16523153
]
Sahil Aggarwal commented on HBASE-19164:
Rebased the patch.
> Avoid UUID.randomUUID in tests
> --
>
> Key: HBASE-19164
> URL: https://issues.apache.org/jira/browse/HBASE-19164
> Project: HBase
> Issue Type: Improvement
> Components: test
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>Priority: Major
> Labels: beginner
> Attachments: HBASE-19164.master.001.patch,
> HBASE-19164.master.002.patch, HBASE-19164.master.003.patch,
> HBASE-19164.master.004.patch
>
>
> We have a lot of places in our test code where we use {{UUID.randomUUID}} to
> generate table names or paths for uniqueness. Unfortunately, this uses up a
> good chunk of system entropy, since Sun chose that random UUID's should use
> the NativePRNGBlocking implementation.
> We don't need to block on entropy for random bits to pick a random table name
> in a test, so we can use something that doesn't strain the system too much -
> secure random can be a source of problems on some VM or containers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[
https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523193#comment-16523193
]
Hudson commented on HBASE-20403:
Results for branch branch-2
[build #908 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/908/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/908//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/908//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/908//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Prefetch sometimes doesn't work with encrypted file system
> --
>
> Key: HBASE-20403
> URL: https://issues.apache.org/jira/browse/HBASE-20403
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-beta-2
>Reporter: Umesh Agashe
>Assignee: Todd Lipcon
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.2
>
> Attachments: hbase-20403.patch, hbase-20403.patch
>
>
> Log from long running test has following stack trace a few times:
> {code}
> 2018-04-09 18:33:21,523 WARN
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch
> path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988,
> offset=36884200, end=231005989
> java.lang.IllegalArgumentException
> at java.nio.Buffer.limit(Buffer.java:275)
> at
> org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183)
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831)
> at
> org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Size on disk calculations seem to get messed up due to encryption. Possible
> fixes can be:
> * if file is encrypted with FileStatus#isEncrypted() and do not prefetch.
> * document that hbase.rs.prefetchblocksonopen cannot be true if file is
> encrypted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20777) RpcConnection could still remain opened after we shutdown the NettyRpcServer
[
https://issues.apache.org/jira/browse/HBASE-20777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523196#comment-16523196
]
Hudson commented on HBASE-20777:
Results for branch branch-2
[build #908 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/908/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/908//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/908//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/908//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> RpcConnection could still remain opened after we shutdown the NettyRpcServer
>
>
> Key: HBASE-20777
> URL: https://issues.apache.org/jira/browse/HBASE-20777
> Project: HBase
> Issue Type: Bug
> Components: rpc
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.2, 2.2.0
>
> Attachments: HBASE-20777-v1.patch, HBASE-20777.patch,
> org.apache.hadoop.hbase.client.TestAsyncTableBatch-output.txt
>
>
> The log is very strange, we keep sending request to a dead RS, and the result
> is not connection refused, but rpc timeout, and later it becomes
> CallQueueTooBig...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20770) WAL cleaner logs way too much; gets clogged when lots of work to do
[
https://issues.apache.org/jira/browse/HBASE-20770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523194#comment-16523194
]
Hudson commented on HBASE-20770:
Results for branch branch-2
[build #908 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/908/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/908//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/908//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/908//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> WAL cleaner logs way too much; gets clogged when lots of work to do
> ---
>
> Key: HBASE-20770
> URL: https://issues.apache.org/jira/browse/HBASE-20770
> Project: HBase
> Issue Type: Bug
> Components: logging
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.2
>
> Attachments: HBASE-20770.branch-2.0.001.patch
>
>
> Been here before (HBASE-7214 and HBASE-19652). Testing on large cluster,
> Master log is in a continuous spew of logging output fililng disks. It is
> stuck making no progress but hard to tell because it is logging minutiae
> rather than a call on whats actually wrong.
> Log is full of this:
> {code}
> 2018-06-21 01:19:12,761 DEBUG
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/meta/bd49572de3914b66985fff5ea2ca7403
> 2018-06-21 01:19:12,761 DEBUG
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/meta/fad01294c6ca421db209e89b5b97d364
> 2018-06-21 01:19:12,823 WARN
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Wait more than 6 ms
> for deleting
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/d3f759d0495257fc1d33ae780b634455/tiny/b72bac4036444dcf9265c7b5664fd403,
> exit...
> 2018-06-21 01:19:12,823 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9
> 2018-06-21 01:19:12,824 WARN
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Wait more than 6 ms
> for deleting
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/2425053ad86823081b368e00bc471e56/tiny/6ea3cb1174434aecbc448e322e2a062c,
> exit...
> 2018-06-21 01:19:12,824 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/big
> 2018-06-21 01:19:12,824 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/tiny
> 2018-06-21 01:19:12,827 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta
> 2018-06-21 01:19:12,844 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0
> 2018-06-21 01:19:12,844 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c
> 2018-06-21 01:19:12,844 DEBUG
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta/90f21dec28d140cda48d37eeb44d37e8
> 2018-06-21 01:19:12,844 DEBUG
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta/8a4cf6410d5a4201963bc1415945f877
> 2018-06-21 01:19:12,848 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c/meta
> 2018-06-21 01:19:12,849 DEBUG
>
[jira] [Commented] (HBASE-20780) ServerRpcConnection logging cleanup
[
https://issues.apache.org/jira/browse/HBASE-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523195#comment-16523195
]
Hudson commented on HBASE-20780:
Results for branch branch-2
[build #908 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/908/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/908//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/908//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/908//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> ServerRpcConnection logging cleanup
> ---
>
> Key: HBASE-20780
> URL: https://issues.apache.org/jira/browse/HBASE-20780
> Project: HBase
> Issue Type: Sub-task
> Components: logging, Performance
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.2
>
> Attachments: 2.0621.2.12782.lock.svg, 2.0621.2M.46340.lock.svg,
> 2.0623.111354.lock.svg, HBASE-20780.branch-2.0.001.patch,
> HBASE-20780.branch-2.0.001.patch
>
>
> The logging we do inside in connection header parsing shows as worst offender
> in the perf locking profiles. Its odd, but easy to clean up. I'd doubt it
> makes any difference in throughput but lets get it out of the way. Let me
> load up a few samples of what it current looks like.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20781) Save recalculating families in a WALEdit batch of Cells
[ https://issues.apache.org/jira/browse/HBASE-20781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20781: -- Attachment: HBASE-20781.branch-2.0.002.patch > Save recalculating families in a WALEdit batch of Cells > --- > > Key: HBASE-20781 > URL: https://issues.apache.org/jira/browse/HBASE-20781 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Major > Attachments: 2.0621.2.12782.cpu.svg, 2.0621.256k.51351.cpu.svg, > 2.0623.families.121250.cpu.svg, HBASE-20781.branch-2.0.001.patch, > HBASE-20781.branch-2.0.002.patch, HBASE-20781.master.001.patch > > > Doing a doMiniBatchMutate, we calculate the set of families that the WALEdit > Cells belong to up front but down after the RingBuffer when we make an > FSWALEdit, we spin through all the Cells again to figure the set of families > in a particularly painful way. Just pass the calculated family set in the > WALEdit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20781) Save recalculating families in a WALEdit batch of Cells
[ https://issues.apache.org/jira/browse/HBASE-20781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523237#comment-16523237 ] stack commented on HBASE-20781: --- .0002 Fix broke unit tests. More explanation of the optimization in comments. > Save recalculating families in a WALEdit batch of Cells > --- > > Key: HBASE-20781 > URL: https://issues.apache.org/jira/browse/HBASE-20781 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Major > Attachments: 2.0621.2.12782.cpu.svg, 2.0621.256k.51351.cpu.svg, > 2.0623.families.121250.cpu.svg, HBASE-20781.branch-2.0.001.patch, > HBASE-20781.branch-2.0.002.patch, HBASE-20781.master.001.patch > > > Doing a doMiniBatchMutate, we calculate the set of families that the WALEdit > Cells belong to up front but down after the RingBuffer when we make an > FSWALEdit, we spin through all the Cells again to figure the set of families > in a particularly painful way. Just pass the calculated family set in the > WALEdit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19164) Avoid UUID.randomUUID in tests
[
https://issues.apache.org/jira/browse/HBASE-19164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Aggarwal updated HBASE-19164:
---
Attachment: HBASE-19164.master.005.patch
> Avoid UUID.randomUUID in tests
> --
>
> Key: HBASE-19164
> URL: https://issues.apache.org/jira/browse/HBASE-19164
> Project: HBase
> Issue Type: Improvement
> Components: test
>Reporter: Mike Drob
>Assignee: Sahil Aggarwal
>Priority: Major
> Labels: beginner
> Attachments: HBASE-19164.master.001.patch,
> HBASE-19164.master.002.patch, HBASE-19164.master.003.patch,
> HBASE-19164.master.004.patch, HBASE-19164.master.005.patch
>
>
> We have a lot of places in our test code where we use {{UUID.randomUUID}} to
> generate table names or paths for uniqueness. Unfortunately, this uses up a
> good chunk of system entropy, since Sun chose that random UUID's should use
> the NativePRNGBlocking implementation.
> We don't need to block on entropy for random bits to pick a random table name
> in a test, so we can use something that doesn't strain the system too much -
> secure random can be a source of problems on some VM or containers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20788) Write up a doc about how to rolling upgrade from 1.x to 2.x
[ https://issues.apache.org/jira/browse/HBASE-20788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523240#comment-16523240 ] stack commented on HBASE-20788: --- Doc is great. Let me commit in morning. I'll prefix it with the tag "Experimental", at least until it has been tried by others. bq. It is OK that during the rolling upgrading there are region server crashes. its this true? What if the 2.0 server carrying hbase:meta goes down before another hbase2 is up on the cluster? That this works is super sweet. Good on you [~Apache9] > Write up a doc about how to rolling upgrade from 1.x to 2.x > --- > > Key: HBASE-20788 > URL: https://issues.apache.org/jira/browse/HBASE-20788 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 2.1.0 > > Attachments: HBASE-20788.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20781) Save recalculating families in a WALEdit batch of Cells
[ https://issues.apache.org/jira/browse/HBASE-20781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523093#comment-16523093 ] Duo Zhang commented on HBASE-20781: --- The approach is fine. +1. But seems we broke some UTs? > Save recalculating families in a WALEdit batch of Cells > --- > > Key: HBASE-20781 > URL: https://issues.apache.org/jira/browse/HBASE-20781 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Major > Attachments: 2.0621.2.12782.cpu.svg, 2.0621.256k.51351.cpu.svg, > 2.0623.families.121250.cpu.svg, HBASE-20781.branch-2.0.001.patch, > HBASE-20781.master.001.patch > > > Doing a doMiniBatchMutate, we calculate the set of families that the WALEdit > Cells belong to up front but down after the RingBuffer when we make an > FSWALEdit, we spin through all the Cells again to figure the set of families > in a particularly painful way. Just pass the calculated family set in the > WALEdit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20788) Write up a doc about how to rolling upgrade from 1.x to 2.x
[ https://issues.apache.org/jira/browse/HBASE-20788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20788: -- Attachment: HBASE-20788.patch > Write up a doc about how to rolling upgrade from 1.x to 2.x > --- > > Key: HBASE-20788 > URL: https://issues.apache.org/jira/browse/HBASE-20788 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 2.1.0 > > Attachments: HBASE-20788.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20788) Write up a doc about how to rolling upgrade from 1.x to 2.x
[ https://issues.apache.org/jira/browse/HBASE-20788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20788: -- Status: Patch Available (was: Open) > Write up a doc about how to rolling upgrade from 1.x to 2.x > --- > > Key: HBASE-20788 > URL: https://issues.apache.org/jira/browse/HBASE-20788 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 2.1.0 > > Attachments: HBASE-20788.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20786) Table create with thousands of regions takes too long
[ https://issues.apache.org/jira/browse/HBASE-20786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523143#comment-16523143 ] Andrew Purtell commented on HBASE-20786: There are a lot of fixed size executor pools in the master for various tasks. Can they be serviced by one common pool, maybe with a relatively small core pool but with a large upper bound? Same thought applies to RSes to a lesser extent. It's annoying to have to adjust them up or down due to environment particulars, although I suppose that is not common. (I needed to do it for deploy of new test tables on slow FS (S3)) Will help here aside from other improvements. > Table create with thousands of regions takes too long > - > > Key: HBASE-20786 > URL: https://issues.apache.org/jira/browse/HBASE-20786 > Project: HBase > Issue Type: Umbrella > Components: Performance >Reporter: stack >Priority: Major > > Internal testing has create of a table with 33k regions taking 18 minutes. > Let me provide more info below. We have an executor with default ten threads > handling the creation of the regions in HDFS which helps distribute out the > load but its not enough. This cluster had >600 servers. Let me add detail. > Need to spend some time on speeding up create/assigns. Made this an umbrella > issue so can pick off pieces of the problem as subtasks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20788) Write up a doc about how to rolling upgrade from 1.x to 2.x
[
https://issues.apache.org/jira/browse/HBASE-20788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523161#comment-16523161
]
Hadoop QA commented on HBASE-20788:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
46s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 6m
6s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 4m
50s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20788 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12929129/HBASE-20788.patch |
| Optional Tests | asflicense refguide |
| uname | Linux 28fd2fe17466 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 4ba6242a62 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| refguide |
https://builds.apache.org/job/PreCommit-HBASE-Build/13388/artifact/patchprocess/branch-site/book.html
|
| refguide |
https://builds.apache.org/job/PreCommit-HBASE-Build/13388/artifact/patchprocess/patch-site/book.html
|
| Max. process+thread count | 83 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13388/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Write up a doc about how to rolling upgrade from 1.x to 2.x
> ---
>
> Key: HBASE-20788
> URL: https://issues.apache.org/jira/browse/HBASE-20788
> Project: HBase
> Issue Type: Sub-task
> Components: documentation
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Blocker
> Fix For: 2.1.0
>
> Attachments: HBASE-20788.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20777) RpcConnection could still remain opened after we shutdown the NettyRpcServer
[
https://issues.apache.org/jira/browse/HBASE-20777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523218#comment-16523218
]
Hudson commented on HBASE-20777:
Results for branch branch-2.0
[build #474 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/474/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/474//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/474//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/474//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> RpcConnection could still remain opened after we shutdown the NettyRpcServer
>
>
> Key: HBASE-20777
> URL: https://issues.apache.org/jira/browse/HBASE-20777
> Project: HBase
> Issue Type: Bug
> Components: rpc
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.2, 2.2.0
>
> Attachments: HBASE-20777-v1.patch, HBASE-20777.patch,
> org.apache.hadoop.hbase.client.TestAsyncTableBatch-output.txt
>
>
> The log is very strange, we keep sending request to a dead RS, and the result
> is not connection refused, but rpc timeout, and later it becomes
> CallQueueTooBig...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20770) WAL cleaner logs way too much; gets clogged when lots of work to do
[
https://issues.apache.org/jira/browse/HBASE-20770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523215#comment-16523215
]
Hudson commented on HBASE-20770:
Results for branch branch-2.0
[build #474 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/474/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/474//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/474//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/474//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> WAL cleaner logs way too much; gets clogged when lots of work to do
> ---
>
> Key: HBASE-20770
> URL: https://issues.apache.org/jira/browse/HBASE-20770
> Project: HBase
> Issue Type: Bug
> Components: logging
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.2
>
> Attachments: HBASE-20770.branch-2.0.001.patch
>
>
> Been here before (HBASE-7214 and HBASE-19652). Testing on large cluster,
> Master log is in a continuous spew of logging output fililng disks. It is
> stuck making no progress but hard to tell because it is logging minutiae
> rather than a call on whats actually wrong.
> Log is full of this:
> {code}
> 2018-06-21 01:19:12,761 DEBUG
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/meta/bd49572de3914b66985fff5ea2ca7403
> 2018-06-21 01:19:12,761 DEBUG
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/meta/fad01294c6ca421db209e89b5b97d364
> 2018-06-21 01:19:12,823 WARN
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Wait more than 6 ms
> for deleting
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/d3f759d0495257fc1d33ae780b634455/tiny/b72bac4036444dcf9265c7b5664fd403,
> exit...
> 2018-06-21 01:19:12,823 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9
> 2018-06-21 01:19:12,824 WARN
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Wait more than 6 ms
> for deleting
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/2425053ad86823081b368e00bc471e56/tiny/6ea3cb1174434aecbc448e322e2a062c,
> exit...
> 2018-06-21 01:19:12,824 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/big
> 2018-06-21 01:19:12,824 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/e98cdb817bb3af5fa26e2b885a0b2ec6/tiny
> 2018-06-21 01:19:12,827 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta
> 2018-06-21 01:19:12,844 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/17f85c98389104b19358f6751da577d0
> 2018-06-21 01:19:12,844 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c
> 2018-06-21 01:19:12,844 DEBUG
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta/90f21dec28d140cda48d37eeb44d37e8
> 2018-06-21 01:19:12,844 DEBUG
> org.apache.hadoop.hbase.master.cleaner.HFileCleaner: Removing
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/665bfa38c86a28d641ce08f8fea0a7f9/meta/8a4cf6410d5a4201963bc1415945f877
> 2018-06-21 01:19:12,848 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore: Cleaning under
> hdfs://ns1/hbase/archive/data/default/IntegrationTestBigLinkedList/c98e276423813aaa74d848983c47d93c/meta
> 2018-06-21 01:19:12,849 DEBUG
> org.apache.hadoop.hbase.master.cleaner.CleanerChore:
[jira] [Commented] (HBASE-20780) ServerRpcConnection logging cleanup
[
https://issues.apache.org/jira/browse/HBASE-20780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523217#comment-16523217
]
Hudson commented on HBASE-20780:
Results for branch branch-2.0
[build #474 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/474/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/474//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/474//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/474//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> ServerRpcConnection logging cleanup
> ---
>
> Key: HBASE-20780
> URL: https://issues.apache.org/jira/browse/HBASE-20780
> Project: HBase
> Issue Type: Sub-task
> Components: logging, Performance
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.2
>
> Attachments: 2.0621.2.12782.lock.svg, 2.0621.2M.46340.lock.svg,
> 2.0623.111354.lock.svg, HBASE-20780.branch-2.0.001.patch,
> HBASE-20780.branch-2.0.001.patch
>
>
> The logging we do inside in connection header parsing shows as worst offender
> in the perf locking profiles. Its odd, but easy to clean up. I'd doubt it
> makes any difference in throughput but lets get it out of the way. Let me
> load up a few samples of what it current looks like.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20403) Prefetch sometimes doesn't work with encrypted file system
[
https://issues.apache.org/jira/browse/HBASE-20403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523216#comment-16523216
]
Hudson commented on HBASE-20403:
Results for branch branch-2.0
[build #474 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/474/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/474//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/474//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/474//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Prefetch sometimes doesn't work with encrypted file system
> --
>
> Key: HBASE-20403
> URL: https://issues.apache.org/jira/browse/HBASE-20403
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-beta-2
>Reporter: Umesh Agashe
>Assignee: Todd Lipcon
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.2
>
> Attachments: hbase-20403.patch, hbase-20403.patch
>
>
> Log from long running test has following stack trace a few times:
> {code}
> 2018-04-09 18:33:21,523 WARN
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl: Prefetch
> path=hdfs://ns1/hbase/data/default/IntegrationTestBigLinkedList_20180409172704/35f1a7ef13b9d327665228abdbcdffae/meta/9089d98b2a6b4847b3fcf6aceb124988,
> offset=36884200, end=231005989
> java.lang.IllegalArgumentException
> at java.nio.Buffer.limit(Buffer.java:275)
> at
> org.apache.hadoop.hdfs.ByteBufferStrategy.readFromBlock(ReaderStrategy.java:183)
> at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:705)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:766)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:831)
> at
> org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:197)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:762)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readAtOffset(HFileBlock.java:1559)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1771)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1594)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl.readBlock(HFileReaderImpl.java:1488)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderImpl$1.run(HFileReaderImpl.java:278)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Size on disk calculations seem to get messed up due to encryption. Possible
> fixes can be:
> * if file is encrypted with FileStatus#isEncrypted() and do not prefetch.
> * document that hbase.rs.prefetchblocksonopen cannot be true if file is
> encrypted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
