[jira] [Commented] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633249#comment-16633249 ] Duo Zhang commented on HBASE-21256: --- Mind uploading the patch to review board? > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, HBASE-21256-v2.patch, > HBASE-21256-v3.patch, HBASE-21256-v4.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[ https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633250#comment-16633250 ] Duo Zhang commented on HBASE-21254: --- I think maybe we could introduce something like the force flush at RS side. There is a holdingCleanupTracker for the oldest procedure wal file, if there are still procedure alive and only have record in this file then we can not deleted it, and we can get the procedure id through the holdingCleanupTracker. So maybe we can do this, add a listener call to indicate the upper layer, that these procedures need to be persist again, then the upper layer will call ProcedureStore.update to persist these procedures to the newest procedure wal file, then we are safe to delete the oldest files. I do not think this will break any assumptions? What do you guys think? [~stack] [~allan163]. > Need to find a way to limit the number of proc wal files > > > Key: HBASE-21254 > URL: https://issues.apache.org/jira/browse/HBASE-21254 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > > For regionserver, we have a max wal file limitation, if we reach the > limitation, we will trigger a flush on specific regions so that we can delete > old wal files. But for proc wals, we do not have this mechanism, and it will > be worse after HBASE-21233, as if there is an old procedure which can not > make progress and do not persist its state, we need to keep the old proc wal > file for ever... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21207) Add client side sorting functionality in master web UI for table and region server details.
[
https://issues.apache.org/jira/browse/HBASE-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633243#comment-16633243
]
Ted Yu commented on HBASE-21207:
This is the command I used locally to verify the addendum:
{code}
mvn clean verify -fae --batch-mode -pl
hbase-shaded/hbase-shaded-check-invariants -am -Dtest=NoUnitTests
-DHBasePatchProcess -Prelease -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true
-Dfindbugs.skip=true
{code}
>From
>https://builds.apache.org/job/PreCommit-HBASE-Build/14548/artifact/patchprocess/archiver/hbase-server/target/rat.txt
> , I don't see file without license.
> Add client side sorting functionality in master web UI for table and region
> server details.
> ---
>
> Key: HBASE-21207
> URL: https://issues.apache.org/jira/browse/HBASE-21207
> Project: HBase
> Issue Type: Improvement
> Components: master, monitoring, UI, Usability
>Reporter: Archana Katiyar
>Assignee: Archana Katiyar
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8
>
> Attachments: 14926e82-b929-11e8-8bdd-4ce4621f1118.png,
> 21207.branch-1.addendum.patch, 2724afd8-b929-11e8-8171-8b5b2ba3084e.png,
> HBASE-21207-branch-1.patch, HBASE-21207-branch-1.v1.patch,
> HBASE-21207-branch-2.v1.patch, HBASE-21207.patch, HBASE-21207.patch,
> HBASE-21207.v1.patch, edc5c812-b928-11e8-87e2-ce6396629bbc.png
>
>
> In Master UI, we can see region server details like requests per seconds and
> number of regions etc. Similarly, for tables also we can see online regions ,
> offline regions.
> It will help ops people in determining hot spotting if we can provide sort
> functionality in the UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21207) Add client side sorting functionality in master web UI for table and region server details.
[
https://issues.apache.org/jira/browse/HBASE-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633242#comment-16633242
]
Hadoop QA commented on HBASE-21207:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 1m
49s{color} | {color:red} branch has 10 errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
37s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} xml {color} | {color:red} 0m 0s{color} |
{color:red} The patch has 1 ill-formed XML file(s). {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
41s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 38s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
56s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
42s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}145m
28s{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}170m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| XML | Parsing Error(s): |
| | pom.xml |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA Issue | HBASE-21207 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12941851/21207.branch-1.addendum.patch
|
| Optional Tests | dupname asflicense javac javadoc unit shadedjars
hadoopcheck xml compile |
| uname | Linux 3102ad7ffefd 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-1 / 121fad3 |
| maven | version: Apache Maven 3.0.5 |
| Default Java | 1.7.0_191 |
| Multi-JDK versions |
[jira] [Commented] (HBASE-21260) The whole balancer plans might be aborted if there are more than one plans to move a same region
[
https://issues.apache.org/jira/browse/HBASE-21260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633240#comment-16633240
]
Allan Yang commented on HBASE-21260:
Good finding, I believe this issue is introduced in HBASE-20881 by [~Apache9].
In pre branch-2, move one region to different locations at the same time won't
cause any exception(it will execute sequentially, and unassign the region from
the right location).
> The whole balancer plans might be aborted if there are more than one plans to
> move a same region
> -
>
> Key: HBASE-21260
> URL: https://issues.apache.org/jira/browse/HBASE-21260
> Project: HBase
> Issue Type: Bug
> Components: Balancer, master
>Affects Versions: 2.1.0, 2.0.0
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
>
> In SimpleLoadBalancer, plans are generated firstly by average number regions
> per server for a table. Each server will be randomly assigned either
> floor(average) or ceiling(average) regions (if the average is not an integer
> number). But afterwards, the balanceOverall method might generate new plans
> of some regions of the table to balance server loads in whole cluster scope.
> As a result, there are plans to move a same region in one call of balance.
> Currently, branch-2 is using async procedures to implement balancer plans.
> But the concurrency of moving the same regions will cause the balance method
> failed. And all the afterwards plans will not be implement when one plan
> encounters exception.
> We have encountered this problem in our practices, the logs are as follows,
> {color:#205081}2018-09-26,12:12:38,224 INFO
> [master/c4-hadoop-tst-ct15:52900.Chore.1]
> org.apache.hadoop.hbase.master.HMaster: Balancer plans size is 3757, the
> balance interval is 79 ms, and the max number regions in transition is 25
> 2018-09-26,12:12:38,224 INFO [master/c4-hadoop-tst-ct15:52900.Chore.1]
> org.apache.hadoop.hbase.master.HMaster: balance hri=1588230740,
> source=c4-hadoop-tst-st99.bj,52900,1537522783781,
> destination=c4-hadoop-tst-st28.bj,52900,1537520009497
> 2018-09-26,12:12:38,325 INFO [master/c4-hadoop-tst-ct15:52900.Chore.1]
> org.apache.hadoop.hbase.master.HMaster: balance hri=1588230740,
> source=c4-hadoop-tst-st99.bj,52900,1537522783781,
> destination=c4-hadoop-tst-st29.bj,52900,1537522784188
> 2018-09-26,12:12:38,325 INFO [PEWorker-16]
> org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler:
> pid=119197, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE;
> TransitRegionStateProcedure table=hbase:meta, region=1588230740, REOPEN/MOVE
> checking lock on 1588230740
> 2018-09-26,12:12:38,325 ERROR [master/c4-hadoop-tst-ct15:52900.Chore.1]
> org.apache.hadoop.hbase.master.balancer.BalancerChore: Failed to balance.
> org.apache.hadoop.hbase.HBaseIOException: rit=OPEN,
> location=c4-hadoop-tst-st99.bj,52900,1537522783781, table=hbase:meta,
> region=1588230740 is currently in transition
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.preTransitCheck(AssignmentManager.java:536)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createMoveRegionProcedure(AssignmentManager.java:592)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.moveAsync(AssignmentManager.java:609)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1707)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1622)
> at
> org.apache.hadoop.hbase.master.balancer.BalancerChore.chore(BalancerChore.java:49)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745){color}
> This is a serious problem because it often occurs when new RSs started or old
> RSs failover. And what's more, no effective methods can be used to make the
> balance of the cluster back to normal.
> But the solution of this problem may be simple. We
[jira] [Commented] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[
https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633238#comment-16633238
]
Hadoop QA commented on HBASE-21256:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
12s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} hbase-it: The patch generated 1 new + 40 unchanged - 0
fixed = 41 total (was 40) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
12s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 46s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
44s{color} | {color:green} hbase-it in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21256 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12941858/HBASE-21256-v4.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux d6294e3420e4 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / fdbaa4c3f0 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/14552/artifact/patchprocess/diff-checkstyle-hbase-it.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14552/testReport/ |
| Max. process+thread count | 391 (vs. ulimit of 1) |
| modules | C: hbase-it U: hbase-it |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/14552/console
[jira] [Commented] (HBASE-20952) Re-visit the WAL API
[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633237#comment-16633237
]
Hudson commented on HBASE-20952:
Results for branch HBASE-20952
[build #3 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/3/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/3//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/3//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/3//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Re-visit the WAL API
>
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[
https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633236#comment-16633236
]
Hadoop QA commented on HBASE-21256:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
49s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} hbase-it: The patch generated 2 new + 40 unchanged - 0
fixed = 42 total (was 40) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
43s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 55s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
46s{color} | {color:green} hbase-it in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21256 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12941856/HBASE-21256-v3.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux cf8bb24f6fd6 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / fdbaa4c3f0 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/14551/artifact/patchprocess/diff-checkstyle-hbase-it.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14551/testReport/ |
| Max. process+thread count | 390 (vs. ulimit of 1) |
| modules | C: hbase-it U: hbase-it |
| Console output |
[jira] [Updated] (HBASE-21245) Add exponential backoff when retrying for sync replication related procedures
[ https://issues.apache.org/jira/browse/HBASE-21245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-21245: --- Attachment: HBASE-21245.master.003.patch > Add exponential backoff when retrying for sync replication related procedures > - > > Key: HBASE-21245 > URL: https://issues.apache.org/jira/browse/HBASE-21245 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-21245.master.001.patch, > HBASE-21245.master.002.patch, HBASE-21245.master.003.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21260) The whole balancer plans might be aborted if there are more than one plans to move a same region
[
https://issues.apache.org/jira/browse/HBASE-21260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaolin Ha updated HBASE-21260:
---
Summary: The whole balancer plans might be aborted if there are more than
one plans to move a same region (was: The whole balancer plans might be
aborted if there are more than one plans to move the same region )
> The whole balancer plans might be aborted if there are more than one plans to
> move a same region
> -
>
> Key: HBASE-21260
> URL: https://issues.apache.org/jira/browse/HBASE-21260
> Project: HBase
> Issue Type: Bug
> Components: Balancer, master
>Affects Versions: 2.1.0, 2.0.0
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
>
> In SimpleLoadBalancer, plans are generated firstly by average number regions
> per server for a table. Each server will be randomly assigned either
> floor(average) or ceiling(average) regions (if the average is not an integer
> number). But afterwards, the balanceOverall method might generate new plans
> of some regions of the table to balance server loads in whole cluster scope.
> As a result, there are plans to move a same region in one call of balance.
> Currently, branch-2 is using async procedures to implement balancer plans.
> But the concurrency of moving the same regions will cause the balance method
> failed. And all the afterwards plans will not be implement when one plan
> encounters exception.
> We have encountered this problem in our practices, the logs are as follows,
> {color:#205081}2018-09-26,12:12:38,224 INFO
> [master/c4-hadoop-tst-ct15:52900.Chore.1]
> org.apache.hadoop.hbase.master.HMaster: Balancer plans size is 3757, the
> balance interval is 79 ms, and the max number regions in transition is 25
> 2018-09-26,12:12:38,224 INFO [master/c4-hadoop-tst-ct15:52900.Chore.1]
> org.apache.hadoop.hbase.master.HMaster: balance hri=1588230740,
> source=c4-hadoop-tst-st99.bj,52900,1537522783781,
> destination=c4-hadoop-tst-st28.bj,52900,1537520009497
> 2018-09-26,12:12:38,325 INFO [master/c4-hadoop-tst-ct15:52900.Chore.1]
> org.apache.hadoop.hbase.master.HMaster: balance hri=1588230740,
> source=c4-hadoop-tst-st99.bj,52900,1537522783781,
> destination=c4-hadoop-tst-st29.bj,52900,1537522784188
> 2018-09-26,12:12:38,325 INFO [PEWorker-16]
> org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler:
> pid=119197, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE;
> TransitRegionStateProcedure table=hbase:meta, region=1588230740, REOPEN/MOVE
> checking lock on 1588230740
> 2018-09-26,12:12:38,325 ERROR [master/c4-hadoop-tst-ct15:52900.Chore.1]
> org.apache.hadoop.hbase.master.balancer.BalancerChore: Failed to balance.
> org.apache.hadoop.hbase.HBaseIOException: rit=OPEN,
> location=c4-hadoop-tst-st99.bj,52900,1537522783781, table=hbase:meta,
> region=1588230740 is currently in transition
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.preTransitCheck(AssignmentManager.java:536)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createMoveRegionProcedure(AssignmentManager.java:592)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.moveAsync(AssignmentManager.java:609)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1707)
> at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1622)
> at
> org.apache.hadoop.hbase.master.balancer.BalancerChore.chore(BalancerChore.java:49)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745){color}
> This is a serious problem because it often occurs when new RSs started or old
> RSs failover. And what's more, no effective methods can be used to make the
> balance of the cluster back to normal.
> But the solution of this problem may be simple. We can cache Exceptions when
> implementing a plan, and then just skip it, avoiding
[jira] [Created] (HBASE-21260) The whole balancer plans might be aborted if there are more than one plans to move the same region
Xiaolin Ha created HBASE-21260:
--
Summary: The whole balancer plans might be aborted if there are
more than one plans to move the same region
Key: HBASE-21260
URL: https://issues.apache.org/jira/browse/HBASE-21260
Project: HBase
Issue Type: Bug
Components: Balancer, master
Affects Versions: 2.0.0, 2.1.0
Reporter: Xiaolin Ha
Assignee: Xiaolin Ha
In SimpleLoadBalancer, plans are generated firstly by average number regions
per server for a table. Each server will be randomly assigned either
floor(average) or ceiling(average) regions (if the average is not an integer
number). But afterwards, the balanceOverall method might generate new plans of
some regions of the table to balance server loads in whole cluster scope. As a
result, there are plans to move a same region in one call of balance.
Currently, branch-2 is using async procedures to implement balancer plans. But
the concurrency of moving the same regions will cause the balance method
failed. And all the afterwards plans will not be implement when one plan
encounters exception.
We have encountered this problem in our practices, the logs are as follows,
{color:#205081}2018-09-26,12:12:38,224 INFO
[master/c4-hadoop-tst-ct15:52900.Chore.1]
org.apache.hadoop.hbase.master.HMaster: Balancer plans size is 3757, the
balance interval is 79 ms, and the max number regions in transition is 25
2018-09-26,12:12:38,224 INFO [master/c4-hadoop-tst-ct15:52900.Chore.1]
org.apache.hadoop.hbase.master.HMaster: balance hri=1588230740,
source=c4-hadoop-tst-st99.bj,52900,1537522783781,
destination=c4-hadoop-tst-st28.bj,52900,1537520009497
2018-09-26,12:12:38,325 INFO [master/c4-hadoop-tst-ct15:52900.Chore.1]
org.apache.hadoop.hbase.master.HMaster: balance hri=1588230740,
source=c4-hadoop-tst-st99.bj,52900,1537522783781,
destination=c4-hadoop-tst-st29.bj,52900,1537522784188
2018-09-26,12:12:38,325 INFO [PEWorker-16]
org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler: pid=119197,
state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure
table=hbase:meta, region=1588230740, REOPEN/MOVE checking lock on 1588230740
2018-09-26,12:12:38,325 ERROR [master/c4-hadoop-tst-ct15:52900.Chore.1]
org.apache.hadoop.hbase.master.balancer.BalancerChore: Failed to balance.
org.apache.hadoop.hbase.HBaseIOException: rit=OPEN,
location=c4-hadoop-tst-st99.bj,52900,1537522783781, table=hbase:meta,
region=1588230740 is currently in transition
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.preTransitCheck(AssignmentManager.java:536)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.createMoveRegionProcedure(AssignmentManager.java:592)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.moveAsync(AssignmentManager.java:609)
at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1707)
at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1622)
at
org.apache.hadoop.hbase.master.balancer.BalancerChore.chore(BalancerChore.java:49)
at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745){color}
This is a serious problem because it often occurs when new RSs started or old
RSs failover. And what's more, no effective methods can be used to make the
balance of the cluster back to normal.
But the solution of this problem may be simple. We can cache Exceptions when
implementing a plan, and then just skip it, avoiding failed plans effect later
plans in the whole plans list. New calls of balance can fetch up the failed and
skipped plans.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[
https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633219#comment-16633219
]
Hadoop QA commented on HBASE-21256:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
19s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} hbase-it: The patch generated 2 new + 40 unchanged - 0
fixed = 42 total (was 40) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
14s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 22s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
43s{color} | {color:green} hbase-it in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21256 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12941855/HBASE-21256-v3.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux cf327b19c862 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / fdbaa4c3f0 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/14550/artifact/patchprocess/diff-checkstyle-hbase-it.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14550/testReport/ |
| Max. process+thread count | 393 (vs. ulimit of 1) |
| modules | C: hbase-it U: hbase-it |
| Console output |
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated HBASE-21256: --- Attachment: HBASE-21256-v4.patch > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, HBASE-21256-v2.patch, > HBASE-21256-v3.patch, HBASE-21256-v4.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated HBASE-21256: --- Attachment: HBASE-21256-v3.patch > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, HBASE-21256-v2.patch, > HBASE-21256-v3.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated HBASE-21256: --- Attachment: (was: HBASE-21256-v3.patch) > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, HBASE-21256-v2.patch, > HBASE-21256-v3.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[
https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633216#comment-16633216
]
Hadoop QA commented on HBASE-21256:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
17s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} hbase-it: The patch generated 2 new + 40 unchanged - 0
fixed = 42 total (was 40) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
14s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 54s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
43s{color} | {color:green} hbase-it in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21256 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12941853/HBASE-21256-v3.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 69345d48d882 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / fdbaa4c3f0 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/14549/artifact/patchprocess/diff-checkstyle-hbase-it.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14549/testReport/ |
| Max. process+thread count | 393 (vs. ulimit of 1) |
| modules | C: hbase-it U: hbase-it |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/14549/console
[jira] [Commented] (HBASE-21259) [amv2] Revived deadservers; recreated serverstatenode
[
https://issues.apache.org/jira/browse/HBASE-21259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633210#comment-16633210
]
stack commented on HBASE-21259:
---
[~allan163] It prevents our being able to finish unassigns when the server
involved has been crash processed; they end up hanging around suspended forever
(STUCK).
The way unassign works when remote server is 'crashed' being processed or
finished being processed is that we ask the ServerStateNode if log split has
been done or if it is online. If not, we suspend ourselves waiting on SCP to
tell us when it is ok to proceed. If the server becomes 'online' again
unintentionally, then we have a bunch of regions STUCK, suspended (For detail,
see UnassignProcedure#remoteCallFailed).
I've been running this patch on cluster and seems to do the job. Let me run it
longer to be sure.
> [amv2] Revived deadservers; recreated serverstatenode
> -
>
> Key: HBASE-21259
> URL: https://issues.apache.org/jira/browse/HBASE-21259
> Project: HBase
> Issue Type: Bug
> Components: amv2
>Affects Versions: 2.1.0
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.2.0, 2.1.1, 2.0.3
>
>
> On startup, I see servers being revived; i.e. their serverstatenode is
> getting marked online even though its just been processed by
> ServerCrashProcedure. It looks like this (in a patched server that reports on
> whenever a serverstatenode is created):
> {code}
> 2018-09-29 03:45:40,963 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=3982597,
> state=SUCCESS; ServerCrashProcedure
> server=vb1442.halxg.cloudera.com,22101,1536675314426, splitWal=true,
> meta=false in 1.0130sec
> ...
> 2018-09-29 03:45:43,733 INFO
> org.apache.hadoop.hbase.master.assignment.RegionStates: CREATING!
> vb1442.halxg.cloudera.com,22101,1536675314426
> java.lang.RuntimeException: WHERE AM I?
> at
> org.apache.hadoop.hbase.master.assignment.RegionStates.getOrCreateServer(RegionStates.java:1116)
> at
> org.apache.hadoop.hbase.master.assignment.RegionStates.addRegionToServer(RegionStates.java:1143)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.markRegionAsClosing(AssignmentManager.java:1464)
> at
> org.apache.hadoop.hbase.master.assignment.UnassignProcedure.updateTransition(UnassignProcedure.java:200)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:369)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:97)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:953)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1716)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1494)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:75)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2022)
> {code}
> See how we've just finished a SCP which will have removed the
> serverstatenode... but then we come across an unassign that references the
> server that was just processed. The unassign will attempt to update the
> serverstatenode and therein we create one if one not present. We shouldn't be
> creating one.
> I think I see this a lot because I am scheduling unassigns with hbck2. The
> servers crash and then come up with SCPs doing cleanup of old server and
> unassign procedures in the procedure executor queue to be processed still
> but could happen at any time on cluster should an unassign happen get
> scheduled near an SCP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated HBASE-21256: --- Attachment: HBASE-21256-v3.patch > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, HBASE-21256-v2.patch, > HBASE-21256-v3.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated HBASE-21256: --- Attachment: (was: HBASE-21256-v3.patch) > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, HBASE-21256-v2.patch, ITBLL-1.png, > ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633207#comment-16633207 ] Zephyr Guo commented on HBASE-21256: Hi, [~Apache9] I attached the patch-v3, and use 64-bit LCG. Please review, thanks. > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, HBASE-21256-v2.patch, > HBASE-21256-v3.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated HBASE-21256: --- Attachment: HBASE-21256-v3.patch > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, HBASE-21256-v2.patch, > HBASE-21256-v3.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21207) Add client side sorting functionality in master web UI for table and region server details.
[ https://issues.apache.org/jira/browse/HBASE-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21207: --- Status: Patch Available (was: Reopened) Corrected the exclusion in pom.xml for branch-1. Master branch doesn't have this problem. > Add client side sorting functionality in master web UI for table and region > server details. > --- > > Key: HBASE-21207 > URL: https://issues.apache.org/jira/browse/HBASE-21207 > Project: HBase > Issue Type: Improvement > Components: master, monitoring, UI, Usability >Reporter: Archana Katiyar >Assignee: Archana Katiyar >Priority: Minor > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8 > > Attachments: 14926e82-b929-11e8-8bdd-4ce4621f1118.png, > 21207.branch-1.addendum.patch, 2724afd8-b929-11e8-8171-8b5b2ba3084e.png, > HBASE-21207-branch-1.patch, HBASE-21207-branch-1.v1.patch, > HBASE-21207-branch-2.v1.patch, HBASE-21207.patch, HBASE-21207.patch, > HBASE-21207.v1.patch, edc5c812-b928-11e8-87e2-ce6396629bbc.png > > > In Master UI, we can see region server details like requests per seconds and > number of regions etc. Similarly, for tables also we can see online regions , > offline regions. > It will help ops people in determining hot spotting if we can provide sort > functionality in the UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21207) Add client side sorting functionality in master web UI for table and region server details.
[ https://issues.apache.org/jira/browse/HBASE-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21207: --- Attachment: 21207.branch-1.addendum.patch > Add client side sorting functionality in master web UI for table and region > server details. > --- > > Key: HBASE-21207 > URL: https://issues.apache.org/jira/browse/HBASE-21207 > Project: HBase > Issue Type: Improvement > Components: master, monitoring, UI, Usability >Reporter: Archana Katiyar >Assignee: Archana Katiyar >Priority: Minor > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8 > > Attachments: 14926e82-b929-11e8-8bdd-4ce4621f1118.png, > 21207.branch-1.addendum.patch, 2724afd8-b929-11e8-8171-8b5b2ba3084e.png, > HBASE-21207-branch-1.patch, HBASE-21207-branch-1.v1.patch, > HBASE-21207-branch-2.v1.patch, HBASE-21207.patch, HBASE-21207.patch, > HBASE-21207.v1.patch, edc5c812-b928-11e8-87e2-ce6396629bbc.png > > > In Master UI, we can see region server details like requests per seconds and > number of regions etc. Similarly, for tables also we can see online regions , > offline regions. > It will help ops people in determining hot spotting if we can provide sort > functionality in the UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (HBASE-21207) Add client side sorting functionality in master web UI for table and region server details.
[ https://issues.apache.org/jira/browse/HBASE-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reopened HBASE-21207: > Add client side sorting functionality in master web UI for table and region > server details. > --- > > Key: HBASE-21207 > URL: https://issues.apache.org/jira/browse/HBASE-21207 > Project: HBase > Issue Type: Improvement > Components: master, monitoring, UI, Usability >Reporter: Archana Katiyar >Assignee: Archana Katiyar >Priority: Minor > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8 > > Attachments: 14926e82-b929-11e8-8bdd-4ce4621f1118.png, > 21207.branch-1.addendum.patch, 2724afd8-b929-11e8-8171-8b5b2ba3084e.png, > HBASE-21207-branch-1.patch, HBASE-21207-branch-1.v1.patch, > HBASE-21207-branch-2.v1.patch, HBASE-21207.patch, HBASE-21207.patch, > HBASE-21207.v1.patch, edc5c812-b928-11e8-87e2-ce6396629bbc.png > > > In Master UI, we can see region server details like requests per seconds and > number of regions etc. Similarly, for tables also we can see online regions , > offline regions. > It will help ops people in determining hot spotting if we can provide sort > functionality in the UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21259) [amv2] Revived deadservers; recreated serverstatenode
[
https://issues.apache.org/jira/browse/HBASE-21259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633182#comment-16633182
]
Allan Yang commented on HBASE-21259:
It seems like a bug, but, adding a crashed server into ServerStateNode will
casue any trouble?
> [amv2] Revived deadservers; recreated serverstatenode
> -
>
> Key: HBASE-21259
> URL: https://issues.apache.org/jira/browse/HBASE-21259
> Project: HBase
> Issue Type: Bug
> Components: amv2
>Affects Versions: 2.1.0
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.2.0, 2.1.1, 2.0.3
>
>
> On startup, I see servers being revived; i.e. their serverstatenode is
> getting marked online even though its just been processed by
> ServerCrashProcedure. It looks like this (in a patched server that reports on
> whenever a serverstatenode is created):
> {code}
> 2018-09-29 03:45:40,963 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=3982597,
> state=SUCCESS; ServerCrashProcedure
> server=vb1442.halxg.cloudera.com,22101,1536675314426, splitWal=true,
> meta=false in 1.0130sec
> ...
> 2018-09-29 03:45:43,733 INFO
> org.apache.hadoop.hbase.master.assignment.RegionStates: CREATING!
> vb1442.halxg.cloudera.com,22101,1536675314426
> java.lang.RuntimeException: WHERE AM I?
> at
> org.apache.hadoop.hbase.master.assignment.RegionStates.getOrCreateServer(RegionStates.java:1116)
> at
> org.apache.hadoop.hbase.master.assignment.RegionStates.addRegionToServer(RegionStates.java:1143)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.markRegionAsClosing(AssignmentManager.java:1464)
> at
> org.apache.hadoop.hbase.master.assignment.UnassignProcedure.updateTransition(UnassignProcedure.java:200)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:369)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:97)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:953)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1716)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1494)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:75)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2022)
> {code}
> See how we've just finished a SCP which will have removed the
> serverstatenode... but then we come across an unassign that references the
> server that was just processed. The unassign will attempt to update the
> serverstatenode and therein we create one if one not present. We shouldn't be
> creating one.
> I think I see this a lot because I am scheduling unassigns with hbck2. The
> servers crash and then come up with SCPs doing cleanup of old server and
> unassign procedures in the procedure executor queue to be processed still
> but could happen at any time on cluster should an unassign happen get
> scheduled near an SCP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21207) Add client side sorting functionality in master web UI for table and region server details.
[ https://issues.apache.org/jira/browse/HBASE-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633180#comment-16633180 ] Andrew Purtell commented on HBASE-21207: There was a POM change added to exclude it. If that is not sufficient figure out how to exclude it. We already exclude the other minimized JS file in that directory so it's not a problem. It just needs to be done so the tool actually excludes it. > Add client side sorting functionality in master web UI for table and region > server details. > --- > > Key: HBASE-21207 > URL: https://issues.apache.org/jira/browse/HBASE-21207 > Project: HBase > Issue Type: Improvement > Components: master, monitoring, UI, Usability >Reporter: Archana Katiyar >Assignee: Archana Katiyar >Priority: Minor > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8 > > Attachments: 14926e82-b929-11e8-8bdd-4ce4621f1118.png, > 2724afd8-b929-11e8-8171-8b5b2ba3084e.png, HBASE-21207-branch-1.patch, > HBASE-21207-branch-1.v1.patch, HBASE-21207-branch-2.v1.patch, > HBASE-21207.patch, HBASE-21207.patch, HBASE-21207.v1.patch, > edc5c812-b928-11e8-87e2-ce6396629bbc.png > > > In Master UI, we can see region server details like requests per seconds and > number of regions etc. Similarly, for tables also we can see online regions , > offline regions. > It will help ops people in determining hot spotting if we can provide sort > functionality in the UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633176#comment-16633176 ] Duo Zhang commented on HBASE-21256: --- Just use the 64 bits LCG? It is very simple next = 6364136223846793005L * current + 1442695040888963407L > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, HBASE-21256-v2.patch, ITBLL-1.png, > ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21250) Refactor WALProcedureStore and add more comments for better understanding the implementation
[
https://issues.apache.org/jira/browse/HBASE-21250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633171#comment-16633171
]
Allan Yang commented on HBASE-21250:
Great job! I will try to read these nice comment and refactoring and try to
catch the implementation if I have time in the 7 days long national holiday
coming.
> Refactor WALProcedureStore and add more comments for better understanding the
> implementation
>
>
> Key: HBASE-21250
> URL: https://issues.apache.org/jira/browse/HBASE-21250
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21250.patch
>
>
> The implementation is complicated and lack of comments to say how it works.
> {code}
> /**
> * WAL implementation of the ProcedureStore.
> * @see ProcedureWALPrettyPrinter for printing content of a single WAL.
> * @see #main(String[]) to parse a directory of MasterWALProcs.
> */
> {code}
> I think at least we can move sub classes to separated files to make the class
> smaller, and add more comments to describe what is going on here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21207) Add client side sorting functionality in master web UI for table and region server details.
[
https://issues.apache.org/jira/browse/HBASE-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633172#comment-16633172
]
Ted Yu commented on HBASE-21207:
hbase-server/src/main/resources/hbase-webapps/static/js/jquery.tablesorter.min.js
is causing rat check failure.
{code}
/*! jQuery jquery.com | jquery.org/license */
{code}
What can we do about the above ?
> Add client side sorting functionality in master web UI for table and region
> server details.
> ---
>
> Key: HBASE-21207
> URL: https://issues.apache.org/jira/browse/HBASE-21207
> Project: HBase
> Issue Type: Improvement
> Components: master, monitoring, UI, Usability
>Reporter: Archana Katiyar
>Assignee: Archana Katiyar
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8
>
> Attachments: 14926e82-b929-11e8-8bdd-4ce4621f1118.png,
> 2724afd8-b929-11e8-8171-8b5b2ba3084e.png, HBASE-21207-branch-1.patch,
> HBASE-21207-branch-1.v1.patch, HBASE-21207-branch-2.v1.patch,
> HBASE-21207.patch, HBASE-21207.patch, HBASE-21207.v1.patch,
> edc5c812-b928-11e8-87e2-ce6396629bbc.png
>
>
> In Master UI, we can see region server details like requests per seconds and
> number of regions etc. Similarly, for tables also we can see online regions ,
> offline regions.
> It will help ops people in determining hot spotting if we can provide sort
> functionality in the UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633168#comment-16633168 ] Xu Cang commented on HBASE-21258: - +1 for v5. Lgtm. Thanks [~yuzhih...@gmail.com] > Add resetting of flags for RS Group pre/post hooks in TestRSGroups > -- > > Key: HBASE-21258 > URL: https://issues.apache.org/jira/browse/HBASE-21258 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21258.branch-1.04.txt, 21258.branch-1.05.txt, > 21258.v1.txt > > > Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS > Group pre/post hooks in TestRSGroups was absent. > This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[
https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633167#comment-16633167
]
Hadoop QA commented on HBASE-21258:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
2s{color} | {color:blue} The patch file was not named according to hbase's
naming conventions. Please see
https://yetus.apache.org/documentation/0.8.0/precommit-patchnames for
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
57s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
13s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 1m
57s{color} | {color:red} branch has 10 errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} hbase-rsgroup: The patch generated 0 new + 2
unchanged - 1 fixed = 2 total (was 3) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 1m
56s{color} | {color:red} patch has 10 errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 44s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
33s{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA Issue | HBASE-21258 |
| JIRA Patch URL |
[jira] [Comment Edited] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[
https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633164#comment-16633164
]
Zephyr Guo edited comment on HBASE-21256 at 9/30/18 12:44 AM:
--
[~Apache9]
After saw that page, I seem understand what you mean. If the first 32-bits is
same, then all 128-bits will be same.Could we solve it in other ways? What
about the following solution?
{code:java}
random bytes nanoTime map taskID iteration
|48bits|--64bits--|8bits---|---8bits---|{code}
It seems that we don't need to care about collision.
was (Author: gzh1992n):
[~Apache9]
After saw that page, I seem understand you means. If the first 32-bits is same,
then all 128-bits will be same.Could we solve it in other ways? What about the
following solution?
{code:java}
random bytes nanoTime map taskID iteration
|48bits|--64bits--|8bits---|---8bits---|{code}
It seems that we don't need to care about collision.
> Improve IntegrationTestBigLinkedList for testing huge data
> --
>
> Key: HBASE-21256
> URL: https://issues.apache.org/jira/browse/HBASE-21256
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Affects Versions: 3.0.0
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21256-v1.patch, HBASE-21256-v2.patch, ITBLL-1.png,
> ITBLL-2.png
>
>
> Recently, I use ITBLL to test some features in our company. I have
> encountered the following problems:
>
> 1. Generator is too slow at the generating stage, the root cause is
> SecureRandom. There is a global lock in SecureRandom( See the following
> picture). I use Random instead of SecureRandom, and it could speed up this
> stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but
> speaking of generating random bytes, in my opnion,
> it is the same with Random.
> !ITBLL-1.png!
> 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by
> create keyString variable. However, keyString may never be used if test
> result is correct.(and that's in most cases). Just delay creating keyString
> can yield huge performance boost in verifing stage.
> !ITBLL-2.png!
> 3.Arguments check is needed, because there's constraint between arguments. If
> we broken this constraint, we can not get a correct circular list.
>
> 4.Let big family value size could be configured.
>
> 5.Avoid starting RS at backup master
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[
https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633164#comment-16633164
]
Zephyr Guo edited comment on HBASE-21256 at 9/30/18 12:42 AM:
--
[~Apache9]
After saw that page, I seem understand you means. If the first 32-bits is same,
then all 128-bits will be same.Could we solve it in other ways? What about the
following solution?
{code:java}
random bytes nanoTime map taskID iteration
|48bits|--64bits--|8bits---|---8bits---|{code}
It seems that we don't need to care about collision.
was (Author: gzh1992n):
[~Apache9]
After saw that page, I seem understand you means. If the first 32-bits is same,
then all 128-bits will be same.Could we solve in other ways? What about the
following solution?
{code:java}
random bytes nanoTime map taskID iteration
|48bits|--64bits--|8bits---|---8bits---|{code}
It seems that we don't need to care about collision.
> Improve IntegrationTestBigLinkedList for testing huge data
> --
>
> Key: HBASE-21256
> URL: https://issues.apache.org/jira/browse/HBASE-21256
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Affects Versions: 3.0.0
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21256-v1.patch, HBASE-21256-v2.patch, ITBLL-1.png,
> ITBLL-2.png
>
>
> Recently, I use ITBLL to test some features in our company. I have
> encountered the following problems:
>
> 1. Generator is too slow at the generating stage, the root cause is
> SecureRandom. There is a global lock in SecureRandom( See the following
> picture). I use Random instead of SecureRandom, and it could speed up this
> stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but
> speaking of generating random bytes, in my opnion,
> it is the same with Random.
> !ITBLL-1.png!
> 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by
> create keyString variable. However, keyString may never be used if test
> result is correct.(and that's in most cases). Just delay creating keyString
> can yield huge performance boost in verifing stage.
> !ITBLL-2.png!
> 3.Arguments check is needed, because there's constraint between arguments. If
> we broken this constraint, we can not get a correct circular list.
>
> 4.Let big family value size could be configured.
>
> 5.Avoid starting RS at backup master
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[
https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633164#comment-16633164
]
Zephyr Guo commented on HBASE-21256:
[~Apache9]
After saw that page, I seem understand you means. If the first 32-bits is same,
then all 128-bits will be same.Could we solve in other ways? What about the
following solution?
{code:java}
random bytes nanoTime map taskID iteration
|48bits|--64bits--|8bits---|---8bits---|{code}
It seems that we don't need to care about collision.
> Improve IntegrationTestBigLinkedList for testing huge data
> --
>
> Key: HBASE-21256
> URL: https://issues.apache.org/jira/browse/HBASE-21256
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Affects Versions: 3.0.0
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21256-v1.patch, HBASE-21256-v2.patch, ITBLL-1.png,
> ITBLL-2.png
>
>
> Recently, I use ITBLL to test some features in our company. I have
> encountered the following problems:
>
> 1. Generator is too slow at the generating stage, the root cause is
> SecureRandom. There is a global lock in SecureRandom( See the following
> picture). I use Random instead of SecureRandom, and it could speed up this
> stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but
> speaking of generating random bytes, in my opnion,
> it is the same with Random.
> !ITBLL-1.png!
> 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by
> create keyString variable. However, keyString may never be used if test
> result is correct.(and that's in most cases). Just delay creating keyString
> can yield huge performance boost in verifing stage.
> !ITBLL-2.png!
> 3.Arguments check is needed, because there's constraint between arguments. If
> we broken this constraint, we can not get a correct circular list.
>
> 4.Let big family value size could be configured.
>
> 5.Avoid starting RS at backup master
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[
https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633161#comment-16633161
]
Ted Yu commented on HBASE-21258:
For patch v5, the following shows that only two subtests run:
{code}
grep 'server(244): calling resetFlag'
hbase-rsgroup/target/surefire-reports/org.apache.hadoop.hbase.rsgroup.TestRSGroups1-output.txt
{code}
> Add resetting of flags for RS Group pre/post hooks in TestRSGroups
> --
>
> Key: HBASE-21258
> URL: https://issues.apache.org/jira/browse/HBASE-21258
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 21258.branch-1.04.txt, 21258.branch-1.05.txt,
> 21258.v1.txt
>
>
> Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS
> Group pre/post hooks in TestRSGroups was absent.
> This issue is to add the resetting of these flags before each subtest starts.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21258: --- Attachment: 21258.branch-1.05.txt > Add resetting of flags for RS Group pre/post hooks in TestRSGroups > -- > > Key: HBASE-21258 > URL: https://issues.apache.org/jira/browse/HBASE-21258 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21258.branch-1.04.txt, 21258.branch-1.05.txt, > 21258.v1.txt > > > Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS > Group pre/post hooks in TestRSGroups was absent. > This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21117) Backport HBASE-18350 (fix RSGroups) to branch-1 (Only port the part fixing table locking issue.)
[
https://issues.apache.org/jira/browse/HBASE-21117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632861#comment-16632861
]
Xu Cang edited comment on HBASE-21117 at 9/30/18 12:02 AM:
---
thanks [~apurtell]
I put some comments to here: https://issues.apache.org/jira/browse/HBASE-20627
and suspect the tests are not working as expected. Will follow up on that Jira
and fix this afterward.
Edited:
Will resume this work after HBASE-21258 is fixed.
was (Author: xucang):
thanks [~apurtell]
I put some comments to here: https://issues.apache.org/jira/browse/HBASE-20627
and suspect the tests are not working as expected. Will follow up on that Jira
and fix this afterward.
> Backport HBASE-18350 (fix RSGroups) to branch-1 (Only port the part fixing
> table locking issue.)
> --
>
> Key: HBASE-21117
> URL: https://issues.apache.org/jira/browse/HBASE-21117
> Project: HBase
> Issue Type: Bug
> Components: backport, rsgroup, shell
>Affects Versions: 1.3.2
>Reporter: Xu Cang
>Assignee: Xu Cang
>Priority: Major
> Labels: backport
> Attachments: HBASE-21117-branch-1.001.patch,
> HBASE-21117-branch-1.002.patch
>
>
> When working on HBASE-20666, I found out HBASE-18350 did not get ported to
> branch-1, which causes procedure to hang when #moveTables called sometimes.
> After looking into the 18350 patch, seems it's important since it fixes 4
> issues. This Jira is an attempt to backport it to branch-1.
>
>
> Edited: Aug26.
> After reviewed the HBASE-18350 patch. I decided to only port part 2 of the
> patch.
> Because part1 and part3 is AMv2 related. I won't touch is since Amv2 is only
> for branch-2
>
> {quote}
> Subject: [PATCH] HBASE-18350 RSGroups are broken under AMv2
> - Table moving to RSG was buggy, because it left the table unassigned.
> Now it is fixed we immediately assign to an appropriate RS
> (MoveRegionProcedure).
> *- Table was locked while moving, but unassign operation hung, because*
> *locked table queues are not scheduled while locked. Fixed. port
> this one.*
> - ProcedureSyncWait was buggy, because it searched the procId in
> executor, but executor does not store the return values of internal
> operations (they are stored, but immediately removed by the cleaner).
> - list_rsgroups in the shell show also the assigned tables and servers.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633157#comment-16633157 ] Xu Cang commented on HBASE-21258: - Tested branch1.04 and all tests ran successfully. Though, will this separating out 2 tests to TestRSGroups1 cause more duplicated test running during unittest phase? If we have to do so, can you put some comments/explanations in the TestRSGroups1 code to avoid confusion? It may be attempting for people to move this test back to TestRSGroups in future. > Add resetting of flags for RS Group pre/post hooks in TestRSGroups > -- > > Key: HBASE-21258 > URL: https://issues.apache.org/jira/browse/HBASE-21258 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21258.branch-1.04.txt, 21258.v1.txt > > > Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS > Group pre/post hooks in TestRSGroups was absent. > This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[
https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633156#comment-16633156
]
Hadoop QA commented on HBASE-21258:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
3s{color} | {color:blue} The patch file was not named according to hbase's
naming conventions. Please see
https://yetus.apache.org/documentation/0.8.0/precommit-patchnames for
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
54s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 1m
55s{color} | {color:red} branch has 10 errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
13s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} hbase-rsgroup: The patch generated 2 new + 2 unchanged
- 1 fixed = 4 total (was 3) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 1m
50s{color} | {color:red} patch has 10 errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 36s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m
35s{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA Issue | HBASE-21258 |
| JIRA Patch URL |
[jira] [Updated] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21258: --- Attachment: 21258.branch-1.04.txt > Add resetting of flags for RS Group pre/post hooks in TestRSGroups > -- > > Key: HBASE-21258 > URL: https://issues.apache.org/jira/browse/HBASE-21258 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21258.branch-1.04.txt, 21258.v1.txt > > > Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS > Group pre/post hooks in TestRSGroups was absent. > This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[
https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633139#comment-16633139
]
Ted Yu commented on HBASE-21258:
Even for a run where testNamespaceConstraint failed, we can see that
preAddRSGroupCalled was set (additional log):
{code}
2018-09-29 22:33:39,357 DEBUG [main]
rsgroup.TestRSGroups$CPMasterObserver(288): calling resetFlags
2018-09-29 22:33:39,357 INFO [main] rsgroup.TestRSGroups(221):
testNamespaceConstraint
2018-09-29 22:33:39,358 DEBUG
[RpcServer.default.FPBQ.Fifo.handler=27,queue=0,port=39540]
rsgroup.TestRSGroups$CPMasterObserver(340): setting preAddRSGroupCalled
{code}
> Add resetting of flags for RS Group pre/post hooks in TestRSGroups
> --
>
> Key: HBASE-21258
> URL: https://issues.apache.org/jira/browse/HBASE-21258
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 21258.v1.txt
>
>
> Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS
> Group pre/post hooks in TestRSGroups was absent.
> This issue is to add the resetting of these flags before each subtest starts.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633136#comment-16633136 ] Ted Yu commented on HBASE-21258: Interesting - I couldn't reproduce test failure on branch-1 when running single subtest. Each complete run of the test takes 280 seconds, making debugging prolonged. > Add resetting of flags for RS Group pre/post hooks in TestRSGroups > -- > > Key: HBASE-21258 > URL: https://issues.apache.org/jira/browse/HBASE-21258 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21258.v1.txt > > > Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS > Group pre/post hooks in TestRSGroups was absent. > This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633125#comment-16633125 ] Ted Yu commented on HBASE-21258: I ran the 7 tests in TestRSGroups by commenting out the tests in TestRSGroupsBase. They all passed with branch-1 patch. > Add resetting of flags for RS Group pre/post hooks in TestRSGroups > -- > > Key: HBASE-21258 > URL: https://issues.apache.org/jira/browse/HBASE-21258 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21258.v1.txt > > > Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS > Group pre/post hooks in TestRSGroups was absent. > This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[
https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633123#comment-16633123
]
Hadoop QA commented on HBASE-21258:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
3s{color} | {color:blue} The patch file was not named according to hbase's
naming conventions. Please see
https://yetus.apache.org/documentation/0.8.0/precommit-patchnames for
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
0s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
12s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
12m 36s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
22s{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21258 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12941844/21258.v1.txt |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 88abcc787a1f 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / fdbaa4c3f0 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14545/testReport/ |
| Max. process+thread count | 2047 (vs. ulimit of 1) |
|
[jira] [Updated] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21258: --- Attachment: (was: 21258.v1.txt) > Add resetting of flags for RS Group pre/post hooks in TestRSGroups > -- > > Key: HBASE-21258 > URL: https://issues.apache.org/jira/browse/HBASE-21258 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21258.v1.txt > > > Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS > Group pre/post hooks in TestRSGroups was absent. > This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21258: --- Attachment: 21258.v1.txt > Add resetting of flags for RS Group pre/post hooks in TestRSGroups > -- > > Key: HBASE-21258 > URL: https://issues.apache.org/jira/browse/HBASE-21258 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21258.v1.txt > > > Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS > Group pre/post hooks in TestRSGroups was absent. > This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633111#comment-16633111 ] Xu Cang commented on HBASE-21258: - thanks, [~yuzhih...@gmail.com]! It fails on branch-1 for this test #testNamespaceConstraint for me locally. > Add resetting of flags for RS Group pre/post hooks in TestRSGroups > -- > > Key: HBASE-21258 > URL: https://issues.apache.org/jira/browse/HBASE-21258 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21258.v1.txt > > > Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS > Group pre/post hooks in TestRSGroups was absent. > This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21198) Exclude dependency on net.minidev:json-smart
[
https://issues.apache.org/jira/browse/HBASE-21198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633106#comment-16633106
]
Ted Yu commented on HBASE-21198:
bq. Where did it come in?
json-smart comes in as transitive dependency:
{code}
[INFO] | \- org.apache.hadoop:hadoop-auth:jar:3.0.0:compile
[INFO] | +- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile
[INFO] | | +- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
[INFO] | | \- net.minidev:json-smart:jar:2.3:compile (version selected
from constraint [1.3.1,2.3])
{code}
bq. Why did it come in?
For hadoop-2.0 profile in hbase-client/pom.xml , we have:
{code}
org.apache.hadoop
hadoop-auth
{code}
The transitive dependency on json-smart seems not needed.
bq. Was it a mistake?
If you meant introducing the dependency transitively was a mistake, I would
agree.
> Exclude dependency on net.minidev:json-smart
>
>
> Key: HBASE-21198
> URL: https://issues.apache.org/jira/browse/HBASE-21198
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Priority: Major
>
> From
> https://builds.apache.org/job/PreCommit-HBASE-Build/14414/artifact/patchprocess/patch-javac-3.0.0.txt
> :
> {code}
> [ERROR] Failed to execute goal on project hbase-common: Could not resolve
> dependencies for project org.apache.hbase:hbase-common:jar:3.0.0-SNAPSHOT:
> Failed to collect dependencies at org.apache.hadoop:hadoop-common:jar:3.0.0
> -> org.apache.hadoop:hadoop-auth:jar:3.0.0 ->
> com.nimbusds:nimbus-jose-jwt:jar:4.41.1 ->
> net.minidev:json-smart:jar:2.3-SNAPSHOT: Failed to read artifact descriptor
> for net.minidev:json-smart:jar:2.3-SNAPSHOT: Could not transfer artifact
> net.minidev:json-smart:pom:2.3-SNAPSHOT from/to dynamodb-local-oregon
> (https://s3-us-west-2.amazonaws.com/dynamodb-local/release): Access denied
> to:
> https://s3-us-west-2.amazonaws.com/dynamodb-local/release/net/minidev/json-smart/2.3-SNAPSHOT/json-smart-2.3-SNAPSHOT.pom
> , ReasonPhrase:Forbidden. -> [Help 1]
> {code}
> We should exclude dependency on net.minidev:json-smart
> hbase-common/bin/pom.xml has done so.
> The other pom.xml should do the same.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21213) [hbck2] bypass leaves behind state in RegionStates when assign/unassign
[
https://issues.apache.org/jira/browse/HBASE-21213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633081#comment-16633081
]
Hudson commented on HBASE-21213:
Results for branch branch-2.1
[build #395 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/395/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/395//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/395//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/395//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> [hbck2] bypass leaves behind state in RegionStates when assign/unassign
> ---
>
> Key: HBASE-21213
> URL: https://issues.apache.org/jira/browse/HBASE-21213
> Project: HBase
> Issue Type: Bug
> Components: amv2, hbck2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.1.1
>
> Attachments: HBASE-21213.branch-2.1.001.patch,
> HBASE-21213.branch-2.1.002.patch, HBASE-21213.branch-2.1.003.patch,
> HBASE-21213.branch-2.1.004.patch, HBASE-21213.branch-2.1.005.patch,
> HBASE-21213.branch-2.1.006.patch, HBASE-21213.branch-2.1.007.patch,
> HBASE-21213.branch-2.1.007.patch, HBASE-21213.branch-2.1.008.patch,
> HBASE-21213.branch-2.1.009.patch, HBASE-21213.branch-2.1.010.patch,
> HBASE-21213.branch-2.1.011.patch
>
>
> This is a follow-on from HBASE-21083 which added the 'bypass' functionality.
> On bypass, there is more state to be cleared if we are allow new Procedures
> to be scheduled.
> For example, here is a bypass:
> {code}
> 2018-09-20 05:45:43,722 INFO org.apache.hadoop.hbase.procedure2.Procedure:
> pid=100449, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true,
> bypass=LOG-REDACTED UnassignProcedure table=hbase:namespace,
> region=37cc206fe9c4bc1c0a46a34c5f523d16,
> server=ve1233.halxg.cloudera.com,22101,1537397961664 bypassed, returning null
> to finish it
> 2018-09-20 05:45:44,022 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=100449,
> state=SUCCESS, bypass=LOG-REDACTED UnassignProcedure table=hbase:namespace,
> region=37cc206fe9c4bc1c0a46a34c5f523d16,
> server=ve1233.halxg.cloudera.com,22101,1537397961664 in 2mins, 7.618sec
> {code}
> ... but then when I try to assign the bypassed region later, I get this:
> {code}
> 2018-09-20 05:46:31,435 WARN
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure: There is
> already another procedure running on this region this=pid=100450,
> state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure
> table=hbase:namespace, region=37cc206fe9c4bc1c0a46a34c5f523d16
> owner=pid=100449, state=SUCCESS, bypass=LOG-REDACTED UnassignProcedure
> table=hbase:namespace, region=37cc206fe9c4bc1c0a46a34c5f523d16,
> server=ve1233.halxg.cloudera.com,22101,1537397961664 pid=100450,
> state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure
> table=hbase:namespace, region=37cc206fe9c4bc1c0a46a34c5f523d16; rit=OPENING,
> location=ve1233.halxg.cloudera.com,22101,1537397961664
> 2018-09-20 05:46:31,510 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Rolled back pid=100450,
> state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via
> AssignProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException:
> There is already another procedure running on this region this=pid=100450,
> state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure
> table=hbase:namespace, region=37cc206fe9c4bc1c0a46a34c5f523d16
> owner=pid=100449, state=SUCCESS, bypass=LOG-REDACTED UnassignProcedure
> table=hbase:namespace, region=37cc206fe9c4bc1c0a46a34c5f523d16,
> server=ve1233.halxg.cloudera.com,22101,1537397961664; AssignProcedure
> table=hbase:namespace, region=37cc206fe9c4bc1c0a46a34c5f523d16
> exec-time=473msec
> {code}
> ... which is a long-winded way of saying the Unassign Procedure still exists
> still in RegionStateNodes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21213) [hbck2] bypass leaves behind state in RegionStates when assign/unassign
[
https://issues.apache.org/jira/browse/HBASE-21213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633038#comment-16633038
]
Hadoop QA commented on HBASE-21213:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.1 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
23s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
18s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
57s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
10s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
50s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
9s{color} | {color:green} branch-2.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 17m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
16s{color} | {color:red} hbase-server: The patch generated 1 new + 326
unchanged - 1 fixed = 327 total (was 327) {color} |
| {color:red}-1{color} | {color:red} rubocop {color} | {color:red} 0m
11s{color} | {color:red} The patch generated 2 new + 21 unchanged - 1 fixed =
23 total (was 22) {color} |
| {color:green}+1{color} | {color:green} ruby-lint {color} | {color:green} 0m
3s{color} | {color:green} There were no new ruby-lint issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
36s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 49s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
2m 24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
9s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
5s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}116m
43s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
19s{color} | {color:green}
[jira] [Updated] (HBASE-21175) Partially initialized SnapshotHFileCleaner leads to NPE during TestHFileArchiving
[
https://issues.apache.org/jira/browse/HBASE-21175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-21175:
---
Labels: snapshot (was: )
> Partially initialized SnapshotHFileCleaner leads to NPE during
> TestHFileArchiving
> -
>
> Key: HBASE-21175
> URL: https://issues.apache.org/jira/browse/HBASE-21175
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Priority: Minor
> Labels: snapshot
>
> TestHFileArchiving#testCleaningRace creates HFileCleaner instance within the
> test.
> When SnapshotHFileCleaner.init() is called, there is no master parameter
> passed in {{params}}.
> When the chore runs the cleaner during the test, NPE comes out of this line
> in getDeletableFiles():
> {code}
> return cache.getUnreferencedFiles(files, master.getSnapshotManager());
> {code}
> since master is null.
> We should either check for the null master or, pass master instance properly
> when constructing the cleaner instance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21149) TestIncrementalBackupWithBulkLoad may fail due to file copy failure
[
https://issues.apache.org/jira/browse/HBASE-21149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-21149:
---
Status: Open (was: Patch Available)
> TestIncrementalBackupWithBulkLoad may fail due to file copy failure
> ---
>
> Key: HBASE-21149
> URL: https://issues.apache.org/jira/browse/HBASE-21149
> Project: HBase
> Issue Type: Test
> Components: backuprestore
>Reporter: Ted Yu
>Assignee: Vladimir Rodionov
>Priority: Major
> Attachments: HBASE-21149-v1.patch
>
>
> From
> https://builds.apache.org/job/HBase%20Nightly/job/master/471/testReport/junit/org.apache.hadoop.hbase.backup/TestIncrementalBackupWithBulkLoad/TestIncBackupDeleteTable/
> :
> {code}
> 2018-09-03 11:54:30,526 ERROR [Time-limited test]
> impl.TableBackupClient(235): Unexpected Exception : Failed copy from
> hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/0f626c66493649daaf84057b8dd71a30_SeqId_205_,hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/ad8df6415bd9459d9b3df76c588d79df_SeqId_205_
> to hdfs://localhost:53075/backupUT/backup_1535975655488
> java.io.IOException: Failed copy from
> hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/0f626c66493649daaf84057b8dd71a30_SeqId_205_,hdfs://localhost:53075/user/jenkins/test-data/ecd40bd0-cb93-91e0-90b5-7bfd5bb2c566/data/default/test-1535975627781/773f5709b645b46bd3840f9cfb549c5a/f/ad8df6415bd9459d9b3df76c588d79df_SeqId_205_
> to hdfs://localhost:53075/backupUT/backup_1535975655488
> at
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.incrementalCopyHFiles(IncrementalTableBackupClient.java:351)
> at
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.copyBulkLoadedFiles(IncrementalTableBackupClient.java:219)
> at
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.handleBulkLoad(IncrementalTableBackupClient.java:198)
> at
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.execute(IncrementalTableBackupClient.java:320)
> at
> org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:605)
> at
> org.apache.hadoop.hbase.backup.TestIncrementalBackupWithBulkLoad.TestIncBackupDeleteTable(TestIncrementalBackupWithBulkLoad.java:104)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> {code}
> However, some part of the test output was lost:
> {code}
> 2018-09-03 11:53:36,793 DEBUG [RS:0;765c9ca5ea28:36357] regions
> ...[truncated 398396 chars]...
> 8)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21250) Refactor WALProcedureStore and add more comments for better understanding the implementation
[
https://issues.apache.org/jira/browse/HBASE-21250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633030#comment-16633030
]
Duo Zhang commented on HBASE-21250:
---
Review board link:
https://reviews.apache.org/r/68886/
And again, 'Duo Zhang got a fish trophy!' :)
> Refactor WALProcedureStore and add more comments for better understanding the
> implementation
>
>
> Key: HBASE-21250
> URL: https://issues.apache.org/jira/browse/HBASE-21250
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21250.patch
>
>
> The implementation is complicated and lack of comments to say how it works.
> {code}
> /**
> * WAL implementation of the ProcedureStore.
> * @see ProcedureWALPrettyPrinter for printing content of a single WAL.
> * @see #main(String[]) to parse a directory of MasterWALProcs.
> */
> {code}
> I think at least we can move sub classes to separated files to make the class
> smaller, and add more comments to describe what is going on here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21250) Refactor WALProcedureStore and add more comments for better understanding the implementation
[
https://issues.apache.org/jira/browse/HBASE-21250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633026#comment-16633026
]
Duo Zhang commented on HBASE-21250:
---
This is the patch by now. Not finished yet. Still need to think more on how we
build the cleanup tracker, and also the start up logic. We read the wal files
from new to old, and there is little comment about it, which is really
annoying...
> Refactor WALProcedureStore and add more comments for better understanding the
> implementation
>
>
> Key: HBASE-21250
> URL: https://issues.apache.org/jira/browse/HBASE-21250
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21250.patch
>
>
> The implementation is complicated and lack of comments to say how it works.
> {code}
> /**
> * WAL implementation of the ProcedureStore.
> * @see ProcedureWALPrettyPrinter for printing content of a single WAL.
> * @see #main(String[]) to parse a directory of MasterWALProcs.
> */
> {code}
> I think at least we can move sub classes to separated files to make the class
> smaller, and add more comments to describe what is going on here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21250) Refactor WALProcedureStore and add more comments for better understanding the implementation
[
https://issues.apache.org/jira/browse/HBASE-21250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-21250:
--
Attachment: HBASE-21250.patch
> Refactor WALProcedureStore and add more comments for better understanding the
> implementation
>
>
> Key: HBASE-21250
> URL: https://issues.apache.org/jira/browse/HBASE-21250
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21250.patch
>
>
> The implementation is complicated and lack of comments to say how it works.
> {code}
> /**
> * WAL implementation of the ProcedureStore.
> * @see ProcedureWALPrettyPrinter for printing content of a single WAL.
> * @see #main(String[]) to parse a directory of MasterWALProcs.
> */
> {code}
> I think at least we can move sub classes to separated files to make the class
> smaller, and add more comments to describe what is going on here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[
https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633023#comment-16633023
]
Hadoop QA commented on HBASE-21258:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
2s{color} | {color:blue} The patch file was not named according to hbase's
naming conventions. Please see
https://yetus.apache.org/documentation/0.8.0/precommit-patchnames for
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
39s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
11s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 13s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}123m
4s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
14s{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}171m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21258 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12941823/21258.v1.txt |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 21f9895e8d6d 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-21255) Refactor TablePermission into three classes (Global, Namespace, Table)
[
https://issues.apache.org/jira/browse/HBASE-21255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633022#comment-16633022
]
Reid Chan commented on HBASE-21255:
---
There're still branches of works... let me clean them up
> Refactor TablePermission into three classes (Global, Namespace, Table)
> --
>
> Key: HBASE-21255
> URL: https://issues.apache.org/jira/browse/HBASE-21255
> Project: HBase
> Issue Type: Improvement
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Major
> Attachments: HBASE-21225.master.001.patch
>
>
> A TODO in {{TablePermission.java}}
> {code}
> //TODO refactor this class
> //we need to refacting this into three classes (Global, Table, Namespace)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21250) Refactor WALProcedureStore and add more comments for better understanding the implementation
[
https://issues.apache.org/jira/browse/HBASE-21250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633012#comment-16633012
]
Duo Zhang commented on HBASE-21250:
---
I plan to change the 'updated' to 'modified', it is really confusing as we have
an action called update...
> Refactor WALProcedureStore and add more comments for better understanding the
> implementation
>
>
> Key: HBASE-21250
> URL: https://issues.apache.org/jira/browse/HBASE-21250
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
>
> The implementation is complicated and lack of comments to say how it works.
> {code}
> /**
> * WAL implementation of the ProcedureStore.
> * @see ProcedureWALPrettyPrinter for printing content of a single WAL.
> * @see #main(String[]) to parse a directory of MasterWALProcs.
> */
> {code}
> I think at least we can move sub classes to separated files to make the class
> smaller, and add more comments to describe what is going on here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[
https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633013#comment-16633013
]
Hadoop QA commented on HBASE-21256:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
18s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} hbase-it: The patch generated 1 new + 40 unchanged - 0
fixed = 41 total (was 40) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
11s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 42s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
49s{color} | {color:green} hbase-it in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21256 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12941826/HBASE-21256-v2.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 008a50a2d285 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / fdbaa4c3f0 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/14544/artifact/patchprocess/diff-checkstyle-hbase-it.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14544/testReport/ |
| Max. process+thread count | 388 (vs. ulimit of 1) |
| modules | C: hbase-it U: hbase-it |
| Console output |
[jira] [Commented] (HBASE-21245) Add exponential backoff when retrying for sync replication related procedures
[
https://issues.apache.org/jira/browse/HBASE-21245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633002#comment-16633002
]
Hadoop QA commented on HBASE-21245:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
8s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
33s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
13m 55s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}198m 5s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
47s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}254m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestBlockEvictionFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21245 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12941813/HBASE-21245.master.002.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux d83b7e4b09c1 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / fdbaa4c3f0 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/14539/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zephyr Guo updated HBASE-21256: --- Attachment: HBASE-21256-v2.patch Update patch v2: against hbase.client.keyvalue.maxsize > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, HBASE-21256-v2.patch, ITBLL-1.png, > ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18451) PeriodicMemstoreFlusher should inspect the queue before adding a delayed flush request
[
https://issues.apache.org/jira/browse/HBASE-18451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632986#comment-16632986
]
Hudson commented on HBASE-18451:
Results for branch master
[build #517 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/517/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> PeriodicMemstoreFlusher should inspect the queue before adding a delayed
> flush request
> --
>
> Key: HBASE-18451
> URL: https://issues.apache.org/jira/browse/HBASE-18451
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.0.0-alpha-1
>Reporter: Jean-Marc Spaggiari
>Assignee: Xu Cang
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8, 2.1.1
>
> Attachments: HBASE-18451.branch-1.001.patch,
> HBASE-18451.branch-1.002.patch, HBASE-18451.branch-1.002.patch,
> HBASE-18451.master.002.patch, HBASE-18451.master.003.patch,
> HBASE-18451.master.004.patch, HBASE-18451.master.004.patch,
> HBASE-18451.master.patch
>
>
> If you run a big job every 4 hours, impacting many tables (they have 150
> regions per server), ad the end all the regions might have some data to be
> flushed, and we want, after one hour, trigger a periodic flush. That's
> totally fine.
> Now, to avoid a flush storm, when we detect a region to be flushed, we add a
> "randomDelay" to the delayed flush, that way we spread them away.
> RANGE_OF_DELAY is 5 minutes. So we spread the flush over the next 5 minutes,
> which is very good.
> However, because we don't check if there is already a request in the queue,
> 10 seconds after, we create a new request, with a new randomDelay.
> If you generate a randomDelay every 10 seconds, at some point, you will end
> up having a small one, and the flush will be triggered almost immediatly.
> As a result, instead of spreading all the flush within the next 5 minutes,
> you end-up getting them all way more quickly. Like within the first minute.
> Which not only feed the queue to to many flush requests, but also defeats the
> purpose of the randomDelay.
> {code}
> @Override
> protected void chore() {
> final StringBuffer whyFlush = new StringBuffer();
> for (Region r : this.server.onlineRegions.values()) {
> if (r == null) continue;
> if (((HRegion)r).shouldFlush(whyFlush)) {
> FlushRequester requester = server.getFlushRequester();
> if (requester != null) {
> long randomDelay = RandomUtils.nextInt(RANGE_OF_DELAY) +
> MIN_DELAY_TIME;
> LOG.info(getName() + " requesting flush of " +
> r.getRegionInfo().getRegionNameAsString() + " because " +
> whyFlush.toString() +
> " after random delay " + randomDelay + "ms");
> //Throttle the flushes by putting a delay. If we don't throttle,
> and there
> //is a balanced write-load on the regions in a table, we might
> end up
> //overwhelming the filesystem with too many flushes at once.
> requester.requestDelayedFlush(r, randomDelay, false);
> }
> }
> }
> }
> {code}
> {code}
> 2017-07-24 18:44:33,338 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> hbasetest2.domainname.com,60020,1500916375517-MemstoreFlusherChore requesting
> flush of testflush,,1500932649126.578c27d2eb7ef0ad437bf2ff38c053ae. because f
> has an old edit so flush to free WALs after random delay 270785ms
> 2017-07-24 18:44:43,328 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> hbasetest2.domainname.com,60020,1500916375517-MemstoreFlusherChore requesting
> flush of testflush,,1500932649126.578c27d2eb7ef0ad437bf2ff38c053ae. because f
> has an old edit so flush to free WALs after random delay 200143ms
> 2017-07-24 18:44:53,954 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> hbasetest2.domainname.com,60020,1500916375517-MemstoreFlusherChore requesting
> flush of
[jira] [Commented] (HBASE-21188) Print heap and gc informations in our junit ResourceChecker
[
https://issues.apache.org/jira/browse/HBASE-21188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632983#comment-16632983
]
Hudson commented on HBASE-21188:
Results for branch master
[build #517 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/517/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Print heap and gc informations in our junit ResourceChecker
> ---
>
> Key: HBASE-21188
> URL: https://issues.apache.org/jira/browse/HBASE-21188
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21188.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20857) JMX - add Balancer status = enabled / disabled
[
https://issues.apache.org/jira/browse/HBASE-20857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632989#comment-16632989
]
Hudson commented on HBASE-20857:
Results for branch master
[build #517 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/517/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> JMX - add Balancer status = enabled / disabled
> --
>
> Key: HBASE-20857
> URL: https://issues.apache.org/jira/browse/HBASE-20857
> Project: HBase
> Issue Type: Improvement
> Components: API, master, metrics, REST, tooling, Usability
>Reporter: Hari Sekhon
>Assignee: Kiran Kumar Maturi
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8, 2.1.1
>
> Attachments: HBASE-20857.branch-1.4.001.patch,
> HBASE-20857.branch-1.4.002.patch
>
>
> Add HBase Balancer enabled/disabled status to JMX API on HMaster.
> Right now the HMaster will give a warning near the top of HMaster UI if
> balancer is disabled, but scraping this is for monitoring integration is not
> nice, it should be available in JMX API as there is already a
> Master,sub=Balancer bean with metrics for the balancer ops etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21207) Add client side sorting functionality in master web UI for table and region server details.
[
https://issues.apache.org/jira/browse/HBASE-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632988#comment-16632988
]
Hudson commented on HBASE-21207:
Results for branch master
[build #517 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/517/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add client side sorting functionality in master web UI for table and region
> server details.
> ---
>
> Key: HBASE-21207
> URL: https://issues.apache.org/jira/browse/HBASE-21207
> Project: HBase
> Issue Type: Improvement
> Components: master, monitoring, UI, Usability
>Reporter: Archana Katiyar
>Assignee: Archana Katiyar
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8
>
> Attachments: 14926e82-b929-11e8-8bdd-4ce4621f1118.png,
> 2724afd8-b929-11e8-8171-8b5b2ba3084e.png, HBASE-21207-branch-1.patch,
> HBASE-21207-branch-1.v1.patch, HBASE-21207-branch-2.v1.patch,
> HBASE-21207.patch, HBASE-21207.patch, HBASE-21207.v1.patch,
> edc5c812-b928-11e8-87e2-ce6396629bbc.png
>
>
> In Master UI, we can see region server details like requests per seconds and
> number of regions etc. Similarly, for tables also we can see online regions ,
> offline regions.
> It will help ops people in determining hot spotting if we can provide sort
> functionality in the UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21248) Implement exponential backoff when retrying for ModifyPeerProcedure
[
https://issues.apache.org/jira/browse/HBASE-21248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632982#comment-16632982
]
Hudson commented on HBASE-21248:
Results for branch master
[build #517 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/517/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Implement exponential backoff when retrying for ModifyPeerProcedure
> ---
>
> Key: HBASE-21248
> URL: https://issues.apache.org/jira/browse/HBASE-21248
> Project: HBase
> Issue Type: Bug
> Components: proc-v2, Replication
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-21248-v1.patch, HBASE-21248-v2.patch,
> HBASE-21248.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21244) Skip persistence when retrying for assignment related procedures
[
https://issues.apache.org/jira/browse/HBASE-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632984#comment-16632984
]
Hudson commented on HBASE-21244:
Results for branch master
[build #517 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/517/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Skip persistence when retrying for assignment related procedures
>
>
> Key: HBASE-21244
> URL: https://issues.apache.org/jira/browse/HBASE-21244
> Project: HBase
> Issue Type: Sub-task
> Components: amv2, Performance, proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21244.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19418) RANGE_OF_DELAY in PeriodicMemstoreFlusher should be configurable.
[
https://issues.apache.org/jira/browse/HBASE-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632987#comment-16632987
]
Hudson commented on HBASE-19418:
Results for branch master
[build #517 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/517/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> RANGE_OF_DELAY in PeriodicMemstoreFlusher should be configurable.
> -
>
> Key: HBASE-19418
> URL: https://issues.apache.org/jira/browse/HBASE-19418
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha-4
>Reporter: Jean-Marc Spaggiari
>Assignee: Ramie Raufdeen
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.8, 2.1.1
>
> Attachments: HBASE-19418.master.000.patch
>
>
> When RSs have a LOT of regions and CFs, flushing everything within 5 minutes
> is not always doable. It might be interesting to be able to increase the
> RANGE_OF_DELAY.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21249) Add jitter for ProcedureUtil.getBackoffTimeMs
[
https://issues.apache.org/jira/browse/HBASE-21249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632985#comment-16632985
]
Hudson commented on HBASE-21249:
Results for branch master
[build #517 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/517/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Add jitter for ProcedureUtil.getBackoffTimeMs
> -
>
> Key: HBASE-21249
> URL: https://issues.apache.org/jira/browse/HBASE-21249
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Yi Mei
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21249.master.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21196) HTableMultiplexer clears the meta cache after every put operation
[
https://issues.apache.org/jira/browse/HBASE-21196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632990#comment-16632990
]
Hudson commented on HBASE-21196:
Results for branch master
[build #517 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/517/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/517//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> HTableMultiplexer clears the meta cache after every put operation
> -
>
> Key: HBASE-21196
> URL: https://issues.apache.org/jira/browse/HBASE-21196
> Project: HBase
> Issue Type: Bug
> Components: Performance
>Affects Versions: 3.0.0, 1.3.3, 2.2.0
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21196.master.001.patch,
> HBASE-21196.master.001.patch, HBASE-21196.master.002.patch,
> HTableMultiplexer1000Puts.UT.txt
>
>
> *Problem:* Operations which use
> {{AsyncRequestFutureImpl.receiveMultiAction(MultiAction, ServerName,
> MultiResponse, int)}} API with tablename set to null reset the meta cache of
> the corresponding server after each call. One such operation is put operation
> of HTableMultiplexer (Might not be the only one). This may impact the
> performance of the system severely as all new ops directed to that server
> will have to go to zk first to get the meta table address and then get the
> location of the table region as it will become empty after every
> htablemultiplexer put.
> From the logs below, one can see after every other put the cached region
> locations are cleared. As a side effect of this, before every put the server
> needs to contact zk and get meta table location and read meta to get region
> locations of the table.
> {noformat}
> 2018-09-13 22:21:15,467 TRACE [htable-pool11-t1] client.MetaCache(283):
> Removed all cached region locations that map to
> root1-thinkpad-t440p,35811,1536857446588
> 2018-09-13 22:21:15,467 DEBUG [HTableFlushWorker-5]
> client.HTableMultiplexer$FlushWorker(632): Processed 1 put requests for
> root1-ThinkPad-T440p:35811 and 0 failed, latency for this send: 5
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.reader=1,bindAddress=root1-ThinkPad-T440p,port=35811]
> ipc.RpcServer$Connection(1954): RequestHeader call_id: 218 method_name: "Get"
> request_param: true priority: 0 timeout: 6 totalRequestSize: 137 bytes
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> ipc.CallRunner(105): callId: 218 service: ClientService methodName: Get size:
> 137 connection: 127.0.0.1:42338 executing as root1
> 2018-09-13 22:21:15,515 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> ipc.RpcServer(2356): callId: 218 service: ClientService methodName: Get size:
> 137 connection: 127.0.0.1:42338 param: region=
> testHTableMultiplexer_1,,1536857451720.304d914b641a738624937c7f9b4d684f.,
> row=\x00\x00\x00\xC4 connection: 127.0.0.1:42338, response result {
> associated_cell_count: 1 stale: false } queueTime: 0 processingTime: 0
> totalTime: 0
> 2018-09-13 22:21:15,516 TRACE
> [RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=35811]
> io.BoundedByteBufferPool(106): runningAverage=16384, totalCapacity=0,
> count=0, allocations=1
> 2018-09-13 22:21:15,516 TRACE [main] ipc.AbstractRpcClient(236): Call: Get,
> callTime: 2ms
> 2018-09-13 22:21:15,516 TRACE [main] client.ClientScanner(122): Scan
> table=hbase:meta,
> startRow=testHTableMultiplexer_1,\x00\x00\x00\xC5,99
> 2018-09-13 22:21:15,516 TRACE [main] client.ClientSmallReversedScanner(179):
> Advancing internal small scanner to startKey at
> 'testHTableMultiplexer_1,\x00\x00\x00\xC5,99'
> 2018-09-13 22:21:15,517 TRACE [main] client.ZooKeeperRegistry(59): Looking up
> meta region location in ZK,
> connection=org.apache.hadoop.hbase.client.ZooKeeperRegistry@599f571f
> {noformat}
> From the minicluster logs [^HTableMultiplexer1000Puts.UT.txt] one can see
> that the string "Removed all cached region locations that map" and "Looking
[jira] [Commented] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[
https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632977#comment-16632977
]
Duo Zhang commented on HBASE-21256:
---
{quote}
It is 48-bits seed algorithm (according to HBASE-13382 said) that Random
utilizes and therefore it CAN generated 32-bits numbers in a completely random
way.
{quote}
Not really. In HBASE-13161 it shows that SecureRandom can fix the problem.
Anyway, the SecureRandom is not necessary as we do not need 'secure' here.
See this page
https://en.wikipedia.org/wiki/Linear_congruential_generator
There is a table which shows different parameter for different LCGs, I think we
could make use of the one introduced by Knuth. It uses a 64 bits seed, so we
can use it to generate our 16 bytes key - just use two longs, and we can make
sure that a 10B test will not have collisions, as we need to generate 2^64
times to make it repetitive.
Thanks.
> Improve IntegrationTestBigLinkedList for testing huge data
> --
>
> Key: HBASE-21256
> URL: https://issues.apache.org/jira/browse/HBASE-21256
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Affects Versions: 3.0.0
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21256-v1.patch, ITBLL-1.png, ITBLL-2.png
>
>
> Recently, I use ITBLL to test some features in our company. I have
> encountered the following problems:
>
> 1. Generator is too slow at the generating stage, the root cause is
> SecureRandom. There is a global lock in SecureRandom( See the following
> picture). I use Random instead of SecureRandom, and it could speed up this
> stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but
> speaking of generating random bytes, in my opnion,
> it is the same with Random.
> !ITBLL-1.png!
> 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by
> create keyString variable. However, keyString may never be used if test
> result is correct.(and that's in most cases). Just delay creating keyString
> can yield huge performance boost in verifing stage.
> !ITBLL-2.png!
> 3.Arguments check is needed, because there's constraint between arguments. If
> we broken this constraint, we can not get a correct circular list.
>
> 4.Let big family value size could be configured.
>
> 5.Avoid starting RS at backup master
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21255) Refactor TablePermission into three classes (Global, Namespace, Table)
[
https://issues.apache.org/jira/browse/HBASE-21255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632972#comment-16632972
]
Hadoop QA commented on HBASE-21255:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
25s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 44s{color}
| {color:red} hbase-client generated 4 new + 99 unchanged - 4 fixed = 103 total
(was 103) {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 2m 20s{color}
| {color:red} hbase-server generated 5 new + 183 unchanged - 5 fixed = 188
total (was 188) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} hbase-client: The patch generated 16 new + 80
unchanged - 29 fixed = 96 total (was 109) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
20s{color} | {color:red} hbase-server: The patch generated 14 new + 105
unchanged - 50 fixed = 119 total (was 155) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
47s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 48s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
5s{color} | {color:red} hbase-client generated 4 new + 0 unchanged - 0 fixed =
4 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
25s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 44s{color}
| {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}124m 28s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 24s{color}
| {color:red} hbase-rsgroup in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}190m 50s{color} |
{color:black} {color} |
\\
\\
||
[jira] [Commented] (HBASE-21248) Implement exponential backoff when retrying for ModifyPeerProcedure
[
https://issues.apache.org/jira/browse/HBASE-21248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632968#comment-16632968
]
Hudson commented on HBASE-21248:
Results for branch branch-2.1
[build #393 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/393/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/393//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/393//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/393//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Implement exponential backoff when retrying for ModifyPeerProcedure
> ---
>
> Key: HBASE-21248
> URL: https://issues.apache.org/jira/browse/HBASE-21248
> Project: HBase
> Issue Type: Bug
> Components: proc-v2, Replication
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-21248-v1.patch, HBASE-21248-v2.patch,
> HBASE-21248.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[
https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632967#comment-16632967
]
Zephyr Guo commented on HBASE-21256:
{quote}On SecureRandom, what about the original argument that Random is more
likely to have a collision?
{quote}
Thanks for review. [~stack] [~Apache9] [~yuzhih...@gmail.com]
I don't agree with that. This kind of collision ONLY depends on how long the
key(128-bits) is.
It is 48-bits seed algorithm (according to HBASE-13382 said) that Random
utilizes and therefore it CAN generated 32-bits numbers in a completely random
way. Consequently, a total of (2^32)^4 possibilities can be generated by
calling nextInt() 4 times, which is equivalent to 2^128.
> Improve IntegrationTestBigLinkedList for testing huge data
> --
>
> Key: HBASE-21256
> URL: https://issues.apache.org/jira/browse/HBASE-21256
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Affects Versions: 3.0.0
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21256-v1.patch, ITBLL-1.png, ITBLL-2.png
>
>
> Recently, I use ITBLL to test some features in our company. I have
> encountered the following problems:
>
> 1. Generator is too slow at the generating stage, the root cause is
> SecureRandom. There is a global lock in SecureRandom( See the following
> picture). I use Random instead of SecureRandom, and it could speed up this
> stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but
> speaking of generating random bytes, in my opnion,
> it is the same with Random.
> !ITBLL-1.png!
> 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by
> create keyString variable. However, keyString may never be used if test
> result is correct.(and that's in most cases). Just delay creating keyString
> can yield huge performance boost in verifing stage.
> !ITBLL-2.png!
> 3.Arguments check is needed, because there's constraint between arguments. If
> we broken this constraint, we can not get a correct circular list.
>
> 4.Let big family value size could be configured.
>
> 5.Avoid starting RS at backup master
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21259) [amv2] Revived deadservers; recreated serverstatenode
stack created HBASE-21259:
-
Summary: [amv2] Revived deadservers; recreated serverstatenode
Key: HBASE-21259
URL: https://issues.apache.org/jira/browse/HBASE-21259
Project: HBase
Issue Type: Bug
Components: amv2
Affects Versions: 2.1.0
Reporter: stack
Assignee: stack
Fix For: 2.2.0, 2.1.1, 2.0.3
On startup, I see servers being revived; i.e. their serverstatenode is getting
marked online even though its just been processed by ServerCrashProcedure. It
looks like this (in a patched server that reports on whenever a serverstatenode
is created):
{code}
2018-09-29 03:45:40,963 INFO
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=3982597,
state=SUCCESS; ServerCrashProcedure
server=vb1442.halxg.cloudera.com,22101,1536675314426, splitWal=true, meta=false
in 1.0130sec
...
2018-09-29 03:45:43,733 INFO
org.apache.hadoop.hbase.master.assignment.RegionStates: CREATING!
vb1442.halxg.cloudera.com,22101,1536675314426
java.lang.RuntimeException: WHERE AM I?
at
org.apache.hadoop.hbase.master.assignment.RegionStates.getOrCreateServer(RegionStates.java:1116)
at
org.apache.hadoop.hbase.master.assignment.RegionStates.addRegionToServer(RegionStates.java:1143)
at
org.apache.hadoop.hbase.master.assignment.AssignmentManager.markRegionAsClosing(AssignmentManager.java:1464)
at
org.apache.hadoop.hbase.master.assignment.UnassignProcedure.updateTransition(UnassignProcedure.java:200)
at
org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:369)
at
org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:97)
at
org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:953)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1716)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1494)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:75)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2022)
{code}
See how we've just finished a SCP which will have removed the
serverstatenode... but then we come across an unassign that references the
server that was just processed. The unassign will attempt to update the
serverstatenode and therein we create one if one not present. We shouldn't be
creating one.
I think I see this a lot because I am scheduling unassigns with hbck2. The
servers crash and then come up with SCPs doing cleanup of old server and
unassign procedures in the procedure executor queue to be processed still
but could happen at any time on cluster should an unassign happen get scheduled
near an SCP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[
https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632958#comment-16632958
]
Ted Yu commented on HBASE-21256:
Zephyr:
{code}
450 if (multipleUnevenColumnFamilies) {
451 int n =
context.getConfiguration().getInt(BIG_FAMILY_VALUE_SIZE_KEY, 256);
{code}
Since you added precondition checks in your patch, can you add one for the
above against the value of hbase.client.keyvalue.maxsize ?
thanks
> Improve IntegrationTestBigLinkedList for testing huge data
> --
>
> Key: HBASE-21256
> URL: https://issues.apache.org/jira/browse/HBASE-21256
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Affects Versions: 3.0.0
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21256-v1.patch, ITBLL-1.png, ITBLL-2.png
>
>
> Recently, I use ITBLL to test some features in our company. I have
> encountered the following problems:
>
> 1. Generator is too slow at the generating stage, the root cause is
> SecureRandom. There is a global lock in SecureRandom( See the following
> picture). I use Random instead of SecureRandom, and it could speed up this
> stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but
> speaking of generating random bytes, in my opnion,
> it is the same with Random.
> !ITBLL-1.png!
> 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by
> create keyString variable. However, keyString may never be used if test
> result is correct.(and that's in most cases). Just delay creating keyString
> can yield huge performance boost in verifing stage.
> !ITBLL-2.png!
> 3.Arguments check is needed, because there's constraint between arguments. If
> we broken this constraint, we can not get a correct circular list.
>
> 4.Let big family value size could be configured.
>
> 5.Avoid starting RS at backup master
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (HBASE-21257) misspelled words.[occured -> occurred]
[ https://issues.apache.org/jira/browse/HBASE-21257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhanggangxue reopened HBASE-21257: -- > misspelled words.[occured -> occurred] > -- > > Key: HBASE-21257 > URL: https://issues.apache.org/jira/browse/HBASE-21257 > Project: HBase > Issue Type: Bug >Reporter: zhanggangxue >Priority: Trivial > Labels: occured, occurred, typo > Attachments: 0001-misspelled-words.-occured-occurred.patch > > > I found some spelling errors on the master Branch.Found in [misspelled > words|https://github.com/apache/hbase/pull/91] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-21257) misspelled words.[occured -> occurred]
[ https://issues.apache.org/jira/browse/HBASE-21257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhanggangxue resolved HBASE-21257. -- Resolution: Fixed > misspelled words.[occured -> occurred] > -- > > Key: HBASE-21257 > URL: https://issues.apache.org/jira/browse/HBASE-21257 > Project: HBase > Issue Type: Bug >Reporter: zhanggangxue >Priority: Trivial > Labels: occured, occurred, typo > Attachments: 0001-misspelled-words.-occured-occurred.patch > > > I found some spelling errors on the master Branch.Found in [misspelled > words|https://github.com/apache/hbase/pull/91] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20627) Relocate RS Group pre/post hooks from RSGroupAdminServer to RSGroupAdminEndpoint
[
https://issues.apache.org/jira/browse/HBASE-20627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632945#comment-16632945
]
Ted Yu edited comment on HBASE-20627 at 9/29/18 11:17 AM:
--
Thanks for the reminder - looks like I dropped some local edit in the patch
here.
Since this was resolved 4 months ago, I opened HBASE-21258 to add the
resetting. As you can see from the patch there, assertions of the respective
flag pass with resetting.
was (Author: yuzhih...@gmail.com):
Thanks for the reminder - looks like I dropped some local edit in the patch
here.
Since this has been resolved 3 months ago, I am opening a new issue to add the
resetting. As you can see from the patch there, assertions of the respective
flag pass with resetting.
> Relocate RS Group pre/post hooks from RSGroupAdminServer to
> RSGroupAdminEndpoint
>
>
> Key: HBASE-20627
> URL: https://issues.apache.org/jira/browse/HBASE-20627
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.1.0, 1.4.5
>
> Attachments: 20627.branch-1.txt, 20627.v1.txt, 20627.v2.txt,
> 20627.v3.txt
>
>
> Currently RS Group pre/post hooks are called from RSGroupAdminServer.
> e.g. RSGroupAdminServer#removeRSGroup :
> {code}
> if (master.getMasterCoprocessorHost() != null) {
> master.getMasterCoprocessorHost().preRemoveRSGroup(name);
> }
> {code}
> RSGroupAdminServer#removeRSGroup is called by RSGroupAdminEndpoint :
> {code}
> checkPermission("removeRSGroup");
> groupAdminServer.removeRSGroup(request.getRSGroupName());
> {code}
> If permission check fails, the pre hook wouldn't be called.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632954#comment-16632954 ] Duo Zhang commented on HBASE-21256: --- In HBASE-13382 we want to avoid collision on a large data set so we use SecureRandom instead of Random... > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[
https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632953#comment-16632953
]
Hadoop QA commented on HBASE-21256:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
11s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} hbase-it: The patch generated 1 new + 40 unchanged - 0
fixed = 41 total (was 40) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
15s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 31s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
51s{color} | {color:green} hbase-it in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21256 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12941820/HBASE-21256-v1.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 85e3734b95ca 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / fdbaa4c3f0 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/14541/artifact/patchprocess/diff-checkstyle-hbase-it.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14541/testReport/ |
| Max. process+thread count | 389 (vs. ulimit of 1) |
| modules | C: hbase-it U: hbase-it |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/14541/console
[jira] [Updated] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21258: --- Attachment: 21258.v1.txt > Add resetting of flags for RS Group pre/post hooks in TestRSGroups > -- > > Key: HBASE-21258 > URL: https://issues.apache.org/jira/browse/HBASE-21258 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21258.v1.txt > > > Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS > Group pre/post hooks in TestRSGroups was absent. > This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21258: --- Status: Patch Available (was: Open) > Add resetting of flags for RS Group pre/post hooks in TestRSGroups > -- > > Key: HBASE-21258 > URL: https://issues.apache.org/jira/browse/HBASE-21258 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21258.v1.txt > > > Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS > Group pre/post hooks in TestRSGroups was absent. > This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
[ https://issues.apache.org/jira/browse/HBASE-21258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632950#comment-16632950 ] Ted Yu commented on HBASE-21258: [~xucang]: I ran the test locally with patch which passed. Can you take a look at the patch ? > Add resetting of flags for RS Group pre/post hooks in TestRSGroups > -- > > Key: HBASE-21258 > URL: https://issues.apache.org/jira/browse/HBASE-21258 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Attachments: 21258.v1.txt > > > Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS > Group pre/post hooks in TestRSGroups was absent. > This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21258) Add resetting of flags for RS Group pre/post hooks in TestRSGroups
Ted Yu created HBASE-21258: -- Summary: Add resetting of flags for RS Group pre/post hooks in TestRSGroups Key: HBASE-21258 URL: https://issues.apache.org/jira/browse/HBASE-21258 Project: HBase Issue Type: Test Reporter: Ted Yu Assignee: Ted Yu Over HBASE-20627, [~xucang] reminded me that the resetting of flags for RS Group pre/post hooks in TestRSGroups was absent. This issue is to add the resetting of these flags before each subtest starts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21213) [hbck2] bypass leaves behind state in RegionStates when assign/unassign
[
https://issues.apache.org/jira/browse/HBASE-21213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632947#comment-16632947
]
stack commented on HBASE-21213:
---
.010 fixes checkstyle and javadoc.
I actually committed this patch on branch-2.1 then reverted because not ready
to port to branch-2.0 and to port forward. Still tinkering with this patch too.
> [hbck2] bypass leaves behind state in RegionStates when assign/unassign
> ---
>
> Key: HBASE-21213
> URL: https://issues.apache.org/jira/browse/HBASE-21213
> Project: HBase
> Issue Type: Bug
> Components: amv2, hbck2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.1.1
>
> Attachments: HBASE-21213.branch-2.1.001.patch,
> HBASE-21213.branch-2.1.002.patch, HBASE-21213.branch-2.1.003.patch,
> HBASE-21213.branch-2.1.004.patch, HBASE-21213.branch-2.1.005.patch,
> HBASE-21213.branch-2.1.006.patch, HBASE-21213.branch-2.1.007.patch,
> HBASE-21213.branch-2.1.007.patch, HBASE-21213.branch-2.1.008.patch,
> HBASE-21213.branch-2.1.009.patch, HBASE-21213.branch-2.1.010.patch,
> HBASE-21213.branch-2.1.011.patch
>
>
> This is a follow-on from HBASE-21083 which added the 'bypass' functionality.
> On bypass, there is more state to be cleared if we are allow new Procedures
> to be scheduled.
> For example, here is a bypass:
> {code}
> 2018-09-20 05:45:43,722 INFO org.apache.hadoop.hbase.procedure2.Procedure:
> pid=100449, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true,
> bypass=LOG-REDACTED UnassignProcedure table=hbase:namespace,
> region=37cc206fe9c4bc1c0a46a34c5f523d16,
> server=ve1233.halxg.cloudera.com,22101,1537397961664 bypassed, returning null
> to finish it
> 2018-09-20 05:45:44,022 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=100449,
> state=SUCCESS, bypass=LOG-REDACTED UnassignProcedure table=hbase:namespace,
> region=37cc206fe9c4bc1c0a46a34c5f523d16,
> server=ve1233.halxg.cloudera.com,22101,1537397961664 in 2mins, 7.618sec
> {code}
> ... but then when I try to assign the bypassed region later, I get this:
> {code}
> 2018-09-20 05:46:31,435 WARN
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure: There is
> already another procedure running on this region this=pid=100450,
> state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure
> table=hbase:namespace, region=37cc206fe9c4bc1c0a46a34c5f523d16
> owner=pid=100449, state=SUCCESS, bypass=LOG-REDACTED UnassignProcedure
> table=hbase:namespace, region=37cc206fe9c4bc1c0a46a34c5f523d16,
> server=ve1233.halxg.cloudera.com,22101,1537397961664 pid=100450,
> state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure
> table=hbase:namespace, region=37cc206fe9c4bc1c0a46a34c5f523d16; rit=OPENING,
> location=ve1233.halxg.cloudera.com,22101,1537397961664
> 2018-09-20 05:46:31,510 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Rolled back pid=100450,
> state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via
> AssignProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException:
> There is already another procedure running on this region this=pid=100450,
> state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure
> table=hbase:namespace, region=37cc206fe9c4bc1c0a46a34c5f523d16
> owner=pid=100449, state=SUCCESS, bypass=LOG-REDACTED UnassignProcedure
> table=hbase:namespace, region=37cc206fe9c4bc1c0a46a34c5f523d16,
> server=ve1233.halxg.cloudera.com,22101,1537397961664; AssignProcedure
> table=hbase:namespace, region=37cc206fe9c4bc1c0a46a34c5f523d16
> exec-time=473msec
> {code}
> ... which is a long-winded way of saying the Unassign Procedure still exists
> still in RegionStateNodes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21248) Implement exponential backoff when retrying for ModifyPeerProcedure
[
https://issues.apache.org/jira/browse/HBASE-21248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632949#comment-16632949
]
Hudson commented on HBASE-21248:
Results for branch branch-2
[build #1318 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1318/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1318//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1318//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1318//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Implement exponential backoff when retrying for ModifyPeerProcedure
> ---
>
> Key: HBASE-21248
> URL: https://issues.apache.org/jira/browse/HBASE-21248
> Project: HBase
> Issue Type: Bug
> Components: proc-v2, Replication
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-21248-v1.patch, HBASE-21248-v2.patch,
> HBASE-21248.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21213) [hbck2] bypass leaves behind state in RegionStates when assign/unassign
[
https://issues.apache.org/jira/browse/HBASE-21213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-21213:
--
Attachment: HBASE-21213.branch-2.1.011.patch
> [hbck2] bypass leaves behind state in RegionStates when assign/unassign
> ---
>
> Key: HBASE-21213
> URL: https://issues.apache.org/jira/browse/HBASE-21213
> Project: HBase
> Issue Type: Bug
> Components: amv2, hbck2
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.1.1
>
> Attachments: HBASE-21213.branch-2.1.001.patch,
> HBASE-21213.branch-2.1.002.patch, HBASE-21213.branch-2.1.003.patch,
> HBASE-21213.branch-2.1.004.patch, HBASE-21213.branch-2.1.005.patch,
> HBASE-21213.branch-2.1.006.patch, HBASE-21213.branch-2.1.007.patch,
> HBASE-21213.branch-2.1.007.patch, HBASE-21213.branch-2.1.008.patch,
> HBASE-21213.branch-2.1.009.patch, HBASE-21213.branch-2.1.010.patch,
> HBASE-21213.branch-2.1.011.patch
>
>
> This is a follow-on from HBASE-21083 which added the 'bypass' functionality.
> On bypass, there is more state to be cleared if we are allow new Procedures
> to be scheduled.
> For example, here is a bypass:
> {code}
> 2018-09-20 05:45:43,722 INFO org.apache.hadoop.hbase.procedure2.Procedure:
> pid=100449, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true,
> bypass=LOG-REDACTED UnassignProcedure table=hbase:namespace,
> region=37cc206fe9c4bc1c0a46a34c5f523d16,
> server=ve1233.halxg.cloudera.com,22101,1537397961664 bypassed, returning null
> to finish it
> 2018-09-20 05:45:44,022 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=100449,
> state=SUCCESS, bypass=LOG-REDACTED UnassignProcedure table=hbase:namespace,
> region=37cc206fe9c4bc1c0a46a34c5f523d16,
> server=ve1233.halxg.cloudera.com,22101,1537397961664 in 2mins, 7.618sec
> {code}
> ... but then when I try to assign the bypassed region later, I get this:
> {code}
> 2018-09-20 05:46:31,435 WARN
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure: There is
> already another procedure running on this region this=pid=100450,
> state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure
> table=hbase:namespace, region=37cc206fe9c4bc1c0a46a34c5f523d16
> owner=pid=100449, state=SUCCESS, bypass=LOG-REDACTED UnassignProcedure
> table=hbase:namespace, region=37cc206fe9c4bc1c0a46a34c5f523d16,
> server=ve1233.halxg.cloudera.com,22101,1537397961664 pid=100450,
> state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure
> table=hbase:namespace, region=37cc206fe9c4bc1c0a46a34c5f523d16; rit=OPENING,
> location=ve1233.halxg.cloudera.com,22101,1537397961664
> 2018-09-20 05:46:31,510 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Rolled back pid=100450,
> state=ROLLEDBACK,
> exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via
> AssignProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException:
> There is already another procedure running on this region this=pid=100450,
> state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure
> table=hbase:namespace, region=37cc206fe9c4bc1c0a46a34c5f523d16
> owner=pid=100449, state=SUCCESS, bypass=LOG-REDACTED UnassignProcedure
> table=hbase:namespace, region=37cc206fe9c4bc1c0a46a34c5f523d16,
> server=ve1233.halxg.cloudera.com,22101,1537397961664; AssignProcedure
> table=hbase:namespace, region=37cc206fe9c4bc1c0a46a34c5f523d16
> exec-time=473msec
> {code}
> ... which is a long-winded way of saying the Unassign Procedure still exists
> still in RegionStateNodes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20627) Relocate RS Group pre/post hooks from RSGroupAdminServer to RSGroupAdminEndpoint
[
https://issues.apache.org/jira/browse/HBASE-20627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632945#comment-16632945
]
Ted Yu commented on HBASE-20627:
Thanks for the reminder - looks like I dropped some local edit in the patch
here.
Since this has been resolved 3 months ago, I am opening a new issue to add the
resetting. As you can see from the patch there, assertions of the respective
flag pass with resetting.
> Relocate RS Group pre/post hooks from RSGroupAdminServer to
> RSGroupAdminEndpoint
>
>
> Key: HBASE-20627
> URL: https://issues.apache.org/jira/browse/HBASE-20627
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.1.0, 1.4.5
>
> Attachments: 20627.branch-1.txt, 20627.v1.txt, 20627.v2.txt,
> 20627.v3.txt
>
>
> Currently RS Group pre/post hooks are called from RSGroupAdminServer.
> e.g. RSGroupAdminServer#removeRSGroup :
> {code}
> if (master.getMasterCoprocessorHost() != null) {
> master.getMasterCoprocessorHost().preRemoveRSGroup(name);
> }
> {code}
> RSGroupAdminServer#removeRSGroup is called by RSGroupAdminEndpoint :
> {code}
> checkPermission("removeRSGroup");
> groupAdminServer.removeRSGroup(request.getRSGroupName());
> {code}
> If permission check fails, the pre hook wouldn't be called.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20857) JMX - add Balancer status = enabled / disabled
[
https://issues.apache.org/jira/browse/HBASE-20857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632944#comment-16632944
]
Hudson commented on HBASE-20857:
Results for branch branch-1.4
[build #484 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/484/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/484//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/484//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/484//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> JMX - add Balancer status = enabled / disabled
> --
>
> Key: HBASE-20857
> URL: https://issues.apache.org/jira/browse/HBASE-20857
> Project: HBase
> Issue Type: Improvement
> Components: API, master, metrics, REST, tooling, Usability
>Reporter: Hari Sekhon
>Assignee: Kiran Kumar Maturi
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8, 2.1.1
>
> Attachments: HBASE-20857.branch-1.4.001.patch,
> HBASE-20857.branch-1.4.002.patch
>
>
> Add HBase Balancer enabled/disabled status to JMX API on HMaster.
> Right now the HMaster will give a warning near the top of HMaster UI if
> balancer is disabled, but scraping this is for monitoring integration is not
> nice, it should be available in JMX API as there is already a
> Master,sub=Balancer bean with metrics for the balancer ops etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18451) PeriodicMemstoreFlusher should inspect the queue before adding a delayed flush request
[
https://issues.apache.org/jira/browse/HBASE-18451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632941#comment-16632941
]
Hudson commented on HBASE-18451:
Results for branch branch-1.4
[build #484 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/484/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/484//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/484//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/484//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> PeriodicMemstoreFlusher should inspect the queue before adding a delayed
> flush request
> --
>
> Key: HBASE-18451
> URL: https://issues.apache.org/jira/browse/HBASE-18451
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.0.0-alpha-1
>Reporter: Jean-Marc Spaggiari
>Assignee: Xu Cang
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8, 2.1.1
>
> Attachments: HBASE-18451.branch-1.001.patch,
> HBASE-18451.branch-1.002.patch, HBASE-18451.branch-1.002.patch,
> HBASE-18451.master.002.patch, HBASE-18451.master.003.patch,
> HBASE-18451.master.004.patch, HBASE-18451.master.004.patch,
> HBASE-18451.master.patch
>
>
> If you run a big job every 4 hours, impacting many tables (they have 150
> regions per server), ad the end all the regions might have some data to be
> flushed, and we want, after one hour, trigger a periodic flush. That's
> totally fine.
> Now, to avoid a flush storm, when we detect a region to be flushed, we add a
> "randomDelay" to the delayed flush, that way we spread them away.
> RANGE_OF_DELAY is 5 minutes. So we spread the flush over the next 5 minutes,
> which is very good.
> However, because we don't check if there is already a request in the queue,
> 10 seconds after, we create a new request, with a new randomDelay.
> If you generate a randomDelay every 10 seconds, at some point, you will end
> up having a small one, and the flush will be triggered almost immediatly.
> As a result, instead of spreading all the flush within the next 5 minutes,
> you end-up getting them all way more quickly. Like within the first minute.
> Which not only feed the queue to to many flush requests, but also defeats the
> purpose of the randomDelay.
> {code}
> @Override
> protected void chore() {
> final StringBuffer whyFlush = new StringBuffer();
> for (Region r : this.server.onlineRegions.values()) {
> if (r == null) continue;
> if (((HRegion)r).shouldFlush(whyFlush)) {
> FlushRequester requester = server.getFlushRequester();
> if (requester != null) {
> long randomDelay = RandomUtils.nextInt(RANGE_OF_DELAY) +
> MIN_DELAY_TIME;
> LOG.info(getName() + " requesting flush of " +
> r.getRegionInfo().getRegionNameAsString() + " because " +
> whyFlush.toString() +
> " after random delay " + randomDelay + "ms");
> //Throttle the flushes by putting a delay. If we don't throttle,
> and there
> //is a balanced write-load on the regions in a table, we might
> end up
> //overwhelming the filesystem with too many flushes at once.
> requester.requestDelayedFlush(r, randomDelay, false);
> }
> }
> }
> }
> {code}
> {code}
> 2017-07-24 18:44:33,338 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> hbasetest2.domainname.com,60020,1500916375517-MemstoreFlusherChore requesting
> flush of testflush,,1500932649126.578c27d2eb7ef0ad437bf2ff38c053ae. because f
> has an old edit so flush to free WALs after random delay 270785ms
> 2017-07-24 18:44:43,328 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> hbasetest2.domainname.com,60020,1500916375517-MemstoreFlusherChore requesting
> flush of testflush,,1500932649126.578c27d2eb7ef0ad437bf2ff38c053ae. because f
> has an old edit so flush to free WALs after random delay 200143ms
> 2017-07-24 18:44:53,954 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> hbasetest2.domainname.com,60020,1500916375517-MemstoreFlusherChore requesting
> flush of testflush,,1500932649126.578c27d2eb7ef0ad437bf2ff38c053ae. because f
> has an old edit so flush to free
[jira] [Commented] (HBASE-21207) Add client side sorting functionality in master web UI for table and region server details.
[
https://issues.apache.org/jira/browse/HBASE-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632943#comment-16632943
]
Hudson commented on HBASE-21207:
Results for branch branch-1.4
[build #484 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/484/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/484//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/484//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/484//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Add client side sorting functionality in master web UI for table and region
> server details.
> ---
>
> Key: HBASE-21207
> URL: https://issues.apache.org/jira/browse/HBASE-21207
> Project: HBase
> Issue Type: Improvement
> Components: master, monitoring, UI, Usability
>Reporter: Archana Katiyar
>Assignee: Archana Katiyar
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8
>
> Attachments: 14926e82-b929-11e8-8bdd-4ce4621f1118.png,
> 2724afd8-b929-11e8-8171-8b5b2ba3084e.png, HBASE-21207-branch-1.patch,
> HBASE-21207-branch-1.v1.patch, HBASE-21207-branch-2.v1.patch,
> HBASE-21207.patch, HBASE-21207.patch, HBASE-21207.v1.patch,
> edc5c812-b928-11e8-87e2-ce6396629bbc.png
>
>
> In Master UI, we can see region server details like requests per seconds and
> number of regions etc. Similarly, for tables also we can see online regions ,
> offline regions.
> It will help ops people in determining hot spotting if we can provide sort
> functionality in the UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19418) RANGE_OF_DELAY in PeriodicMemstoreFlusher should be configurable.
[
https://issues.apache.org/jira/browse/HBASE-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632942#comment-16632942
]
Hudson commented on HBASE-19418:
Results for branch branch-1.4
[build #484 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/484/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/484//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/484//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/484//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> RANGE_OF_DELAY in PeriodicMemstoreFlusher should be configurable.
> -
>
> Key: HBASE-19418
> URL: https://issues.apache.org/jira/browse/HBASE-19418
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha-4
>Reporter: Jean-Marc Spaggiari
>Assignee: Ramie Raufdeen
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.8, 2.1.1
>
> Attachments: HBASE-19418.master.000.patch
>
>
> When RSs have a LOT of regions and CFs, flushing everything within 5 minutes
> is not always doable. It might be interesting to be able to increase the
> RANGE_OF_DELAY.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21207) Add client side sorting functionality in master web UI for table and region server details.
[
https://issues.apache.org/jira/browse/HBASE-21207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632939#comment-16632939
]
Hudson commented on HBASE-21207:
Results for branch branch-1
[build #482 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/482/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/482//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/482//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/482//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Add client side sorting functionality in master web UI for table and region
> server details.
> ---
>
> Key: HBASE-21207
> URL: https://issues.apache.org/jira/browse/HBASE-21207
> Project: HBase
> Issue Type: Improvement
> Components: master, monitoring, UI, Usability
>Reporter: Archana Katiyar
>Assignee: Archana Katiyar
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8
>
> Attachments: 14926e82-b929-11e8-8bdd-4ce4621f1118.png,
> 2724afd8-b929-11e8-8171-8b5b2ba3084e.png, HBASE-21207-branch-1.patch,
> HBASE-21207-branch-1.v1.patch, HBASE-21207-branch-2.v1.patch,
> HBASE-21207.patch, HBASE-21207.patch, HBASE-21207.v1.patch,
> edc5c812-b928-11e8-87e2-ce6396629bbc.png
>
>
> In Master UI, we can see region server details like requests per seconds and
> number of regions etc. Similarly, for tables also we can see online regions ,
> offline regions.
> It will help ops people in determining hot spotting if we can provide sort
> functionality in the UI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19418) RANGE_OF_DELAY in PeriodicMemstoreFlusher should be configurable.
[
https://issues.apache.org/jira/browse/HBASE-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632938#comment-16632938
]
Hudson commented on HBASE-19418:
Results for branch branch-1
[build #482 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/482/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/482//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/482//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/482//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> RANGE_OF_DELAY in PeriodicMemstoreFlusher should be configurable.
> -
>
> Key: HBASE-19418
> URL: https://issues.apache.org/jira/browse/HBASE-19418
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha-4
>Reporter: Jean-Marc Spaggiari
>Assignee: Ramie Raufdeen
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 1.3.3, 2.2.0, 1.4.8, 2.1.1
>
> Attachments: HBASE-19418.master.000.patch
>
>
> When RSs have a LOT of regions and CFs, flushing everything within 5 minutes
> is not always doable. It might be interesting to be able to increase the
> RANGE_OF_DELAY.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20857) JMX - add Balancer status = enabled / disabled
[
https://issues.apache.org/jira/browse/HBASE-20857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632940#comment-16632940
]
Hudson commented on HBASE-20857:
Results for branch branch-1
[build #482 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/482/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/482//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/482//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/482//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> JMX - add Balancer status = enabled / disabled
> --
>
> Key: HBASE-20857
> URL: https://issues.apache.org/jira/browse/HBASE-20857
> Project: HBase
> Issue Type: Improvement
> Components: API, master, metrics, REST, tooling, Usability
>Reporter: Hari Sekhon
>Assignee: Kiran Kumar Maturi
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8, 2.1.1
>
> Attachments: HBASE-20857.branch-1.4.001.patch,
> HBASE-20857.branch-1.4.002.patch
>
>
> Add HBase Balancer enabled/disabled status to JMX API on HMaster.
> Right now the HMaster will give a warning near the top of HMaster UI if
> balancer is disabled, but scraping this is for monitoring integration is not
> nice, it should be available in JMX API as there is already a
> Master,sub=Balancer bean with metrics for the balancer ops etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18451) PeriodicMemstoreFlusher should inspect the queue before adding a delayed flush request
[
https://issues.apache.org/jira/browse/HBASE-18451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632937#comment-16632937
]
Hudson commented on HBASE-18451:
Results for branch branch-1
[build #482 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/482/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/482//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/482//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/482//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> PeriodicMemstoreFlusher should inspect the queue before adding a delayed
> flush request
> --
>
> Key: HBASE-18451
> URL: https://issues.apache.org/jira/browse/HBASE-18451
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.0.0-alpha-1
>Reporter: Jean-Marc Spaggiari
>Assignee: Xu Cang
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.8, 2.1.1
>
> Attachments: HBASE-18451.branch-1.001.patch,
> HBASE-18451.branch-1.002.patch, HBASE-18451.branch-1.002.patch,
> HBASE-18451.master.002.patch, HBASE-18451.master.003.patch,
> HBASE-18451.master.004.patch, HBASE-18451.master.004.patch,
> HBASE-18451.master.patch
>
>
> If you run a big job every 4 hours, impacting many tables (they have 150
> regions per server), ad the end all the regions might have some data to be
> flushed, and we want, after one hour, trigger a periodic flush. That's
> totally fine.
> Now, to avoid a flush storm, when we detect a region to be flushed, we add a
> "randomDelay" to the delayed flush, that way we spread them away.
> RANGE_OF_DELAY is 5 minutes. So we spread the flush over the next 5 minutes,
> which is very good.
> However, because we don't check if there is already a request in the queue,
> 10 seconds after, we create a new request, with a new randomDelay.
> If you generate a randomDelay every 10 seconds, at some point, you will end
> up having a small one, and the flush will be triggered almost immediatly.
> As a result, instead of spreading all the flush within the next 5 minutes,
> you end-up getting them all way more quickly. Like within the first minute.
> Which not only feed the queue to to many flush requests, but also defeats the
> purpose of the randomDelay.
> {code}
> @Override
> protected void chore() {
> final StringBuffer whyFlush = new StringBuffer();
> for (Region r : this.server.onlineRegions.values()) {
> if (r == null) continue;
> if (((HRegion)r).shouldFlush(whyFlush)) {
> FlushRequester requester = server.getFlushRequester();
> if (requester != null) {
> long randomDelay = RandomUtils.nextInt(RANGE_OF_DELAY) +
> MIN_DELAY_TIME;
> LOG.info(getName() + " requesting flush of " +
> r.getRegionInfo().getRegionNameAsString() + " because " +
> whyFlush.toString() +
> " after random delay " + randomDelay + "ms");
> //Throttle the flushes by putting a delay. If we don't throttle,
> and there
> //is a balanced write-load on the regions in a table, we might
> end up
> //overwhelming the filesystem with too many flushes at once.
> requester.requestDelayedFlush(r, randomDelay, false);
> }
> }
> }
> }
> {code}
> {code}
> 2017-07-24 18:44:33,338 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> hbasetest2.domainname.com,60020,1500916375517-MemstoreFlusherChore requesting
> flush of testflush,,1500932649126.578c27d2eb7ef0ad437bf2ff38c053ae. because f
> has an old edit so flush to free WALs after random delay 270785ms
> 2017-07-24 18:44:43,328 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> hbasetest2.domainname.com,60020,1500916375517-MemstoreFlusherChore requesting
> flush of testflush,,1500932649126.578c27d2eb7ef0ad437bf2ff38c053ae. because f
> has an old edit so flush to free WALs after random delay 200143ms
> 2017-07-24 18:44:53,954 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> hbasetest2.domainname.com,60020,1500916375517-MemstoreFlusherChore requesting
> flush of testflush,,1500932649126.578c27d2eb7ef0ad437bf2ff38c053ae. because f
> has an old edit so flush to free WALs after
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21256: --- Status: Patch Available (was: Open) > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21257) misspelled words.[occured -> occurred]
[ https://issues.apache.org/jira/browse/HBASE-21257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhanggangxue updated HBASE-21257: - Description: I found some spelling errors on the master Branch.Found in [misspelled words|https://github.com/apache/hbase/pull/91] (was: I found some spelling errors on the master Branch.Found in [misspelled words.[occured -> occurred|https://github.com/apache/hbase/pull/91]) > misspelled words.[occured -> occurred] > -- > > Key: HBASE-21257 > URL: https://issues.apache.org/jira/browse/HBASE-21257 > Project: HBase > Issue Type: Bug >Reporter: zhanggangxue >Priority: Trivial > Labels: occured, occurred, typo > Attachments: 0001-misspelled-words.-occured-occurred.patch > > > I found some spelling errors on the master Branch.Found in [misspelled > words|https://github.com/apache/hbase/pull/91] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21257) misspelled words.[occured -> occurred]
[ https://issues.apache.org/jira/browse/HBASE-21257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhanggangxue updated HBASE-21257: - Description: I found some spelling errors on the master Branch.Found in [misspelled words.[occured -> occurred|https://github.com/apache/hbase/pull/91] (was: I found some spelling errors on the master Branch.Found in [misspelled words.[occured -> occurred] #91|https://github.com/apache/hbase/pull/91]) > misspelled words.[occured -> occurred] > -- > > Key: HBASE-21257 > URL: https://issues.apache.org/jira/browse/HBASE-21257 > Project: HBase > Issue Type: Bug >Reporter: zhanggangxue >Priority: Trivial > Labels: occured, occurred, typo > Attachments: 0001-misspelled-words.-occured-occurred.patch > > > I found some spelling errors on the master Branch.Found in [misspelled > words.[occured -> occurred|https://github.com/apache/hbase/pull/91] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
