[jira] [Commented] (HBASE-21251) Refactor RegionMover
[ https://issues.apache.org/jira/browse/HBASE-21251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647516#comment-16647516 ] stack commented on HBASE-21251: --- The RegionMover we have in branch-2.0 works but is just resource heavy and ugly code? If so, its probably fine [~zghaobac]. Let me know if you think otherwise sir. > Refactor RegionMover > > > Key: HBASE-21251 > URL: https://issues.apache.org/jira/browse/HBASE-21251 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.1.1 > > Attachments: HBASE-21251.master.001.patch, > HBASE-21251.master.002.patch, HBASE-21251.master.003.patch, > HBASE-21251.master.004.patch > > > 1. Move connection and admin to RegionMover's member variables. No need > create connection many times. > 2. use try-with-resource to reduce code > 3. use ServerName instead of String > 4. don't use Deprecated method > 5. remove duplicate code > .. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20952) Re-visit the WAL API
[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647498#comment-16647498
]
Hudson commented on HBASE-20952:
Results for branch HBASE-20952
[build #15 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/15/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/15//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/15//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/15//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Re-visit the WAL API
>
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21275) Thrift Server (branch 1 fix) -> Disable TRACE HTTP method for thrift http server (branch 1 only)
[ https://issues.apache.org/jira/browse/HBASE-21275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21275: -- Attachment: HBASE-21275-branch-1.2.003.patch > Thrift Server (branch 1 fix) -> Disable TRACE HTTP method for thrift http > server (branch 1 only) > > > Key: HBASE-21275 > URL: https://issues.apache.org/jira/browse/HBASE-21275 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Fix For: 1.4.8, 1.2.7 > > Attachments: HBASE-21275-branch-1.2.001.patch, > HBASE-21275-branch-1.2.002.patch, HBASE-21275-branch-1.2.003.patch, > HBASE-21275-branch-1.2.003.patch > > > There's been a reasonable number of users running thrift http server on hbase > 1.x suffering with security audit tests pointing thrift server allows TRACE > requests. > After doing some search, I can see HBASE-20406 added restrictions for > TRACE/OPTIONS method when Thrift is running over http, but it relies on many > other commits applied to thrift http server. This patch was later reverted > from master. Then again later, HBASE-20004 had made TRACE/OPTIONS > configurable via "*hbase.thrift.http.allow.options.method*" property, with > both methods being disabled by default. This also seems to rely on many > changes applied to thrift http server, and a branch 1 compatible patch does > not seem feasible. > A solution for branch 1 is pretty simple though, am proposing a patch that > simply uses *WebAppContext*, instead of *Context*, as the context for the > *HttpServer* instance. *WebAppContext* will already restrict TRACE methods by > default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21288) HostingServer in UnassignProcedure is not accurate
[
https://issues.apache.org/jira/browse/HBASE-21288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647489#comment-16647489
]
stack commented on HBASE-21288:
---
bq. I leave hostingServer there since toStringClassDetails() need it to print
the detail of the procedure. toStringClassDetails
Can this not get the server from the RegionNode?
> HostingServer in UnassignProcedure is not accurate
> --
>
> Key: HBASE-21288
> URL: https://issues.apache.org/jira/browse/HBASE-21288
> Project: HBase
> Issue Type: Sub-task
> Components: amv2, Balancer
>Affects Versions: 2.1.0, 2.0.2
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Attachments: HBASE-21288.branch-2.0.001.patch,
> HBASE-21288.branch-2.0.002.patch
>
>
> We have a case that a region shows status OPEN on a already dead server in
> meta table(it is hard to trace how this happen), meaning this region is
> actually not online. But balance came and scheduled a MoveReionProcedure for
> this region, which created a mess:
> The balancer 'thought' this region was on the server which has the same
> address(but with different startcode). So it schedules a MRP from this online
> server to another, but the UnassignProcedure dispatch the unassign call to
> the dead server according to regionstate, which then found the server dead
> and schedule a SCP for the dead server. But since the UnassignProcedure's
> hostingServer is not accurate, the SCP can't interrupt it.
> So, in the end, the SCP can't finish since the UnassignProcedure has the
> region' lock, the UnassignProcedure can not finish since no one wake it, thus
> stuck.
> Here is log, notice that the server of the UnassignProcedure is
> 'hb-uf6oyi699w8h700f0-003.hbase.rds. ,16020,1539153278584' but it was
> dispatch to 'hb-uf6oyi699w8h700f0-003.hbase.rds. ,16020,1539076734964'
> {code}
> 2018-10-10 14:34:50,011 INFO [PEWorker-4]
> assignment.RegionTransitionProcedure(252): Dispatch pid=13, ppid=12,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH, hasLock=true; UnassignProcedure
> table=hbase:acl, region=267335c85766c62479fb4a5f18a1e95f,
> server=hb-uf6oyi699w8h700f0-003.hbase.rds. ,16020,1539153278584; rit=CLOSING,
> location=hb-uf6oyi699w8h700f0-003.hbase.rds. ,16020,1539076734964
> 2018-10-10 14:34:50,011 WARN [PEWorker-4]
> assignment.RegionTransitionProcedure(230): Remote call failed
> hb-uf6oyi699w8h700f0-003.hbase.rds. ,16020,1539076734964; pid=13, ppid=12,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH, hasLock=true; UnassignProcedure
> table=hbase:acl, region=267335c85766c62479fb4a5f18a1e95f,
> server=hb-uf6oyi699w8h700f0-003.hbase.rds. ,16020,1539153278584; rit=CLOSING,
> location=hb-uf6oyi699w8h700f0-003.hbase.rds. ,16020,1539076734964;
> exception=NoServerDispatchException
> org.apache.hadoop.hbase.procedure2.NoServerDispatchException:
> hb-uf6oyi699w8h700f0-003.hbase.rds. ,16020,1539076734964; pid=13, ppid=12,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH, hasLock=true; UnassignProcedure
> table=hbase:acl, region=267335c85766c62479fb4a5f18a1e95f,
> server=hb-uf6oyi699w8h700f0-003.hbase.rds. ,16020,1539153278584
> //Then a SCP was scheduled
> 2018-10-10 14:34:50,012 WARN [PEWorker-4] master.ServerManager(635):
> Expiration of hb-uf6oyi699w8h700f0-003.hbase.rds. ,16020,1539076734964 but
> server not online
> 2018-10-10 14:34:50,012 INFO [PEWorker-4] master.ServerManager(615):
> Processing expiration of hb-uf6oyi699w8h700f0-003.hbase.rds.
> ,16020,1539076734964 on hb-uf6oyi699w8h700f0-001.hbase.rds.
> ,16000,1539088156164
> 2018-10-10 14:34:50,017 DEBUG [PEWorker-4]
> procedure2.ProcedureExecutor(1089): Stored pid=14,
> state=RUNNABLE:SERVER_CRASH_START, hasLock=false; ServerCrashProcedure
> server=hb-uf6oyi699w8h700f0-003.hbase.rds. ,16020,1539076734964,
> splitWal=true, meta=false
> //The SCP did not interrupt the UnassignProcedure but schedule new
> AssignProcedure for this region
> 2018-10-10 14:34:50,043 DEBUG [PEWorker-6]
> procedure.ServerCrashProcedure(250): Done splitting WALs pid=14,
> state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, hasLock=true; ServerCrashProcedure
> server=hb-uf6oyi699w8h700f0-003.hbase.rds. ,16020,1539076734964,
> splitWal=true, meta=false
> 2018-10-10 14:34:50,054 INFO [PEWorker-8]
> procedure2.ProcedureExecutor(1691): Initialized subprocedures=[{pid=15,
> ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE, hasLock=false;
> AssignProcedure table=hbase:acl, region=267335c85766c62479fb4a5f18a1e95f},
> {pid=16, ppid=14, state=RUNNABLE:REGION_TRANSITION_QUEUE, hasLock=false;
> AssignProcedure table=hbase:req_intercept_rule,
> region=460481706415d776b3742f428a6f579b}, {pid=17, ppid=14,
> state=RUNNABLE:REGION_TRANSITION_QUEUE, hasLock=false; AssignProcedure
> table=hbase:namespace,
[jira] [Commented] (HBASE-21289) Remove the log "'hbase.regionserver.maxlogs' was deprecated." in AbstractFSWAL
[ https://issues.apache.org/jira/browse/HBASE-21289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647487#comment-16647487 ] stack commented on HBASE-21289: --- Patch LGTM. > Remove the log "'hbase.regionserver.maxlogs' was deprecated." in AbstractFSWAL > -- > > Key: HBASE-21289 > URL: https://issues.apache.org/jira/browse/HBASE-21289 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Minor > Attachments: HBASE-21289.master.001.patch > > > This log was added by HBASE-14951. And the description and release note never > said this config was deprecated. I thought HBASE-14951 only changed the > default value of maxlogs (Please correct me if I am wrong). And we still use > this config in our hbase book. So the log "'hbase.regionserver.maxlogs' was > deprecated." in AbstractFSWAL is confused. Let's remove it. FYI [~vrodionov] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21291) Add a test for bypassing stuck state-machine procedures
[
https://issues.apache.org/jira/browse/HBASE-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647484#comment-16647484
]
Hadoop QA commented on HBASE-21291:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
2s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} hbase-procedure: The patch generated 3 new + 3
unchanged - 0 fixed = 6 total (was 3) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 6m
10s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
16m 19s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
31s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21291 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12943590/HBASE-21291.master.001.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 05fdeb462b03 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 9e9a1e0f0d |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/14664/artifact/patchprocess/diff-checkstyle-hbase-procedure.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14664/testReport/ |
| Max. process+thread count | 287 (vs. ulimit of 1) |
| modules | C: hbase-procedure U: hbase-procedure |
| Console output
[jira] [Updated] (HBASE-21242) [amv2] Miscellaneous minor log and assign procedure create improvements
[
https://issues.apache.org/jira/browse/HBASE-21242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-21242:
--
Attachment: HBASE-21242.branch-2.001.patch
> [amv2] Miscellaneous minor log and assign procedure create improvements
> ---
>
> Key: HBASE-21242
> URL: https://issues.apache.org/jira/browse/HBASE-21242
> Project: HBase
> Issue Type: Bug
> Components: amv2, Operability
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21242.branch-2.0.001.patch,
> HBASE-21242.branch-2.0.001.patch, HBASE-21242.branch-2.0.002.patch,
> HBASE-21242.branch-2.001.patch, HBASE-21242.branch-2.001.patch,
> HBASE-21242.branch-2.1.001.patch, HBASE-21242.branch-2.1.001.patch,
> HBASE-21242.branch-2.1.001.patch, HBASE-21242.branch-2.1.002.patch
>
>
> Some minor fixups:
> {code}
> For RIT Duration, do better than print ms/seconds. Remove redundant UI
> column dedicated to duration when we log it in the status field too.
> Make bypass log at INFO level -- when DEBUG we can miss important
> fixup detail like why we failed.
> Make it so on complete of subprocedure, we note count of outstanding
> siblings so we have a clue how much further the parent has to go before
> it is done (Helpful when hundreds of servers doing SCP).
> Have the SCP run the AP preflight check before creating an AP; saves
> creation of hundreds of thousands of APs during fixup of this big cluster
> of mine.
> Don't log tablename three times when reporting remote call failed.
> If lock is held already, note who has it. Also log after we get lock
> or if we have to wait rather than log on entrance though we may
> later have to wait (or we may have just picked up the lock).
> {code}
> Posting patch in a sec but let me try it on cluster too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21242) [amv2] Miscellaneous minor log and assign procedure create improvements
[
https://issues.apache.org/jira/browse/HBASE-21242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-21242:
--
Status: Patch Available (was: Reopened)
> [amv2] Miscellaneous minor log and assign procedure create improvements
> ---
>
> Key: HBASE-21242
> URL: https://issues.apache.org/jira/browse/HBASE-21242
> Project: HBase
> Issue Type: Bug
> Components: amv2, Operability
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21242.branch-2.0.001.patch,
> HBASE-21242.branch-2.0.001.patch, HBASE-21242.branch-2.0.002.patch,
> HBASE-21242.branch-2.001.patch, HBASE-21242.branch-2.001.patch,
> HBASE-21242.branch-2.1.001.patch, HBASE-21242.branch-2.1.001.patch,
> HBASE-21242.branch-2.1.001.patch, HBASE-21242.branch-2.1.002.patch
>
>
> Some minor fixups:
> {code}
> For RIT Duration, do better than print ms/seconds. Remove redundant UI
> column dedicated to duration when we log it in the status field too.
> Make bypass log at INFO level -- when DEBUG we can miss important
> fixup detail like why we failed.
> Make it so on complete of subprocedure, we note count of outstanding
> siblings so we have a clue how much further the parent has to go before
> it is done (Helpful when hundreds of servers doing SCP).
> Have the SCP run the AP preflight check before creating an AP; saves
> creation of hundreds of thousands of APs during fixup of this big cluster
> of mine.
> Don't log tablename three times when reporting remote call failed.
> If lock is held already, note who has it. Also log after we get lock
> or if we have to wait rather than log on entrance though we may
> later have to wait (or we may have just picked up the lock).
> {code}
> Posting patch in a sec but let me try it on cluster too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (HBASE-21242) [amv2] Miscellaneous minor log and assign procedure create improvements
[
https://issues.apache.org/jira/browse/HBASE-21242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack reopened HBASE-21242:
---
Reopen. Only branch-2.0 and branch-2.1 done so far.
> [amv2] Miscellaneous minor log and assign procedure create improvements
> ---
>
> Key: HBASE-21242
> URL: https://issues.apache.org/jira/browse/HBASE-21242
> Project: HBase
> Issue Type: Bug
> Components: amv2, Operability
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21242.branch-2.0.001.patch,
> HBASE-21242.branch-2.0.001.patch, HBASE-21242.branch-2.0.002.patch,
> HBASE-21242.branch-2.001.patch, HBASE-21242.branch-2.1.001.patch,
> HBASE-21242.branch-2.1.001.patch, HBASE-21242.branch-2.1.001.patch,
> HBASE-21242.branch-2.1.002.patch
>
>
> Some minor fixups:
> {code}
> For RIT Duration, do better than print ms/seconds. Remove redundant UI
> column dedicated to duration when we log it in the status field too.
> Make bypass log at INFO level -- when DEBUG we can miss important
> fixup detail like why we failed.
> Make it so on complete of subprocedure, we note count of outstanding
> siblings so we have a clue how much further the parent has to go before
> it is done (Helpful when hundreds of servers doing SCP).
> Have the SCP run the AP preflight check before creating an AP; saves
> creation of hundreds of thousands of APs during fixup of this big cluster
> of mine.
> Don't log tablename three times when reporting remote call failed.
> If lock is held already, note who has it. Also log after we get lock
> or if we have to wait rather than log on entrance though we may
> later have to wait (or we may have just picked up the lock).
> {code}
> Posting patch in a sec but let me try it on cluster too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21242) [amv2] Miscellaneous minor log and assign procedure create improvements
[
https://issues.apache.org/jira/browse/HBASE-21242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-21242:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to branch-2.0. Resolving since it pushed to branch-2.0+
> [amv2] Miscellaneous minor log and assign procedure create improvements
> ---
>
> Key: HBASE-21242
> URL: https://issues.apache.org/jira/browse/HBASE-21242
> Project: HBase
> Issue Type: Bug
> Components: amv2, Operability
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21242.branch-2.0.001.patch,
> HBASE-21242.branch-2.0.001.patch, HBASE-21242.branch-2.0.002.patch,
> HBASE-21242.branch-2.001.patch, HBASE-21242.branch-2.1.001.patch,
> HBASE-21242.branch-2.1.001.patch, HBASE-21242.branch-2.1.001.patch,
> HBASE-21242.branch-2.1.002.patch
>
>
> Some minor fixups:
> {code}
> For RIT Duration, do better than print ms/seconds. Remove redundant UI
> column dedicated to duration when we log it in the status field too.
> Make bypass log at INFO level -- when DEBUG we can miss important
> fixup detail like why we failed.
> Make it so on complete of subprocedure, we note count of outstanding
> siblings so we have a clue how much further the parent has to go before
> it is done (Helpful when hundreds of servers doing SCP).
> Have the SCP run the AP preflight check before creating an AP; saves
> creation of hundreds of thousands of APs during fixup of this big cluster
> of mine.
> Don't log tablename three times when reporting remote call failed.
> If lock is held already, note who has it. Also log after we get lock
> or if we have to wait rather than log on entrance though we may
> later have to wait (or we may have just picked up the lock).
> {code}
> Posting patch in a sec but let me try it on cluster too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21291) Add a test for bypassing stuck state-machine procedures
[
https://issues.apache.org/jira/browse/HBASE-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647466#comment-16647466
]
stack commented on HBASE-21291:
---
[~tianjingyun] Great. What [~allan163] said.
How we fix this so StateMachineProcedures are bypassable (I think I've seen
this issue in the wild -- but not sure.. unable to bypass a DisableTable...).
If bypass and override skip getting lock? (I've not done the study...).
> Add a test for bypassing stuck state-machine procedures
> ---
>
> Key: HBASE-21291
> URL: https://issues.apache.org/jira/browse/HBASE-21291
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Attachments: HBASE-21291.master.001.patch
>
>
> {code}
> if (!procedure.isFailed()) {
> if (subprocs != null) {
> if (subprocs.length == 1 && subprocs[0] == procedure) {
> // Procedure returned itself. Quick-shortcut for a state
> machine-like procedure;
> // i.e. we go around this loop again rather than go back out on
> the scheduler queue.
> subprocs = null;
> reExecute = true;
> LOG.trace("Short-circuit to next step on pid={}",
> procedure.getProcId());
> } else {
> // Yield the current procedure, and make the subprocedure runnable
> // subprocs may come back 'null'.
> subprocs = initializeChildren(procStack, procedure, subprocs);
> LOG.info("Initialized subprocedures=" +
> (subprocs == null? null:
> Stream.of(subprocs).map(e -> "{" + e.toString() + "}").
> collect(Collectors.toList()).toString()));
> }
> } else if (procedure.getState() == ProcedureState.WAITING_TIMEOUT) {
> LOG.debug("Added to timeoutExecutor {}", procedure);
> timeoutExecutor.add(procedure);
> } else if (!suspended) {
> // No subtask, so we are done
> procedure.setState(ProcedureState.SUCCESS);
> }
> }
> {code}
> Currently implementation of ProcedureExecutor will set the reExcecute to true
> for state machine like procedure. Then if this procedure is stuck at one
> certain state, it will loop forever.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.getLockEntry(proc.getProcId());
> try {
> executeProcedure(proc);
> } catch (AssertionError e) {
> LOG.info("ASSERT pid=" + proc.getProcId(), e);
> throw e;
> } finally {
> procExecutionLock.releaseLockEntry(lockEntry);
> {code}
> Since procedure will get the IdLock and release it after execution done,
> state machine procedure will never release IdLock until it is finished.
> Then bypassProcedure doesn't work because is will try to grab the IdLock at
> first.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.tryLockEntry(procedure.getProcId(), lockWait);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21299) List counts of actual region states in master UI tables section
[ https://issues.apache.org/jira/browse/HBASE-21299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21299: -- Assignee: stack Status: Patch Available (was: Open) > List counts of actual region states in master UI tables section > --- > > Key: HBASE-21299 > URL: https://issues.apache.org/jira/browse/HBASE-21299 > Project: HBase > Issue Type: Improvement > Components: UI >Reporter: stack >Assignee: stack >Priority: Major > Attachments: HBASE-21299.branch-2.1.001.patch, Screen Shot 2018-10-11 > at 9.44.39 PM.png, Screen Shot 2018-10-11 at 9.44.56 PM.png > > > It the tables panel, we list Open Regions, Offline Regions, Failed Regions, > Split Region, and Other. This is not very useful. It is from a time before > region states were edited down. Better to list OPEN, CLOSED, OPENING, > CLOSING... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21299) List counts of actual region states in master UI tables section
[ https://issues.apache.org/jira/browse/HBASE-21299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647461#comment-16647461 ] stack commented on HBASE-21299: --- I use these counts to find tables that need repair. Later, should be able to click on the numbers to get a list of the actual regions to work on. Currently doing this via shell messing. .001 small patch. > List counts of actual region states in master UI tables section > --- > > Key: HBASE-21299 > URL: https://issues.apache.org/jira/browse/HBASE-21299 > Project: HBase > Issue Type: Improvement > Components: UI >Reporter: stack >Priority: Major > Attachments: HBASE-21299.branch-2.1.001.patch, Screen Shot 2018-10-11 > at 9.44.39 PM.png, Screen Shot 2018-10-11 at 9.44.56 PM.png > > > It the tables panel, we list Open Regions, Offline Regions, Failed Regions, > Split Region, and Other. This is not very useful. It is from a time before > region states were edited down. Better to list OPEN, CLOSED, OPENING, > CLOSING... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21299) List counts of actual region states in master UI tables section
[ https://issues.apache.org/jira/browse/HBASE-21299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21299: -- Attachment: HBASE-21299.branch-2.1.001.patch > List counts of actual region states in master UI tables section > --- > > Key: HBASE-21299 > URL: https://issues.apache.org/jira/browse/HBASE-21299 > Project: HBase > Issue Type: Improvement > Components: UI >Reporter: stack >Priority: Major > Attachments: HBASE-21299.branch-2.1.001.patch, Screen Shot 2018-10-11 > at 9.44.39 PM.png, Screen Shot 2018-10-11 at 9.44.56 PM.png > > > It the tables panel, we list Open Regions, Offline Regions, Failed Regions, > Split Region, and Other. This is not very useful. It is from a time before > region states were edited down. Better to list OPEN, CLOSED, OPENING, > CLOSING... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21299) List counts of actual region states in master UI tables section
[ https://issues.apache.org/jira/browse/HBASE-21299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647460#comment-16647460 ] stack commented on HBASE-21299: --- Here is how it currently looks !Screen Shot 2018-10-11 at 9.44.56 PM.png! Here is how the patch makes it look !Screen Shot 2018-10-11 at 9.44.39 PM.png! > List counts of actual region states in master UI tables section > --- > > Key: HBASE-21299 > URL: https://issues.apache.org/jira/browse/HBASE-21299 > Project: HBase > Issue Type: Improvement > Components: UI >Reporter: stack >Priority: Major > Attachments: Screen Shot 2018-10-11 at 9.44.39 PM.png, Screen Shot > 2018-10-11 at 9.44.56 PM.png > > > It the tables panel, we list Open Regions, Offline Regions, Failed Regions, > Split Region, and Other. This is not very useful. It is from a time before > region states were edited down. Better to list OPEN, CLOSED, OPENING, > CLOSING... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21299) List counts of actual region states in master UI tables section
stack created HBASE-21299: - Summary: List counts of actual region states in master UI tables section Key: HBASE-21299 URL: https://issues.apache.org/jira/browse/HBASE-21299 Project: HBase Issue Type: Improvement Components: UI Reporter: stack Attachments: Screen Shot 2018-10-11 at 9.44.39 PM.png, Screen Shot 2018-10-11 at 9.44.56 PM.png It the tables panel, we list Open Regions, Offline Regions, Failed Regions, Split Region, and Other. This is not very useful. It is from a time before region states were edited down. Better to list OPEN, CLOSED, OPENING, CLOSING... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21299) List counts of actual region states in master UI tables section
[ https://issues.apache.org/jira/browse/HBASE-21299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21299: -- Attachment: Screen Shot 2018-10-11 at 9.44.39 PM.png Screen Shot 2018-10-11 at 9.44.56 PM.png > List counts of actual region states in master UI tables section > --- > > Key: HBASE-21299 > URL: https://issues.apache.org/jira/browse/HBASE-21299 > Project: HBase > Issue Type: Improvement > Components: UI >Reporter: stack >Priority: Major > Attachments: Screen Shot 2018-10-11 at 9.44.39 PM.png, Screen Shot > 2018-10-11 at 9.44.56 PM.png > > > It the tables panel, we list Open Regions, Offline Regions, Failed Regions, > Split Region, and Other. This is not very useful. It is from a time before > region states were edited down. Better to list OPEN, CLOSED, OPENING, > CLOSING... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21291) Add a test for bypassing stuck state-machine procedures
[
https://issues.apache.org/jira/browse/HBASE-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647450#comment-16647450
]
Allan Yang commented on HBASE-21291:
{code}
+ProcedureTestingUtility.restart(procExecutor);
{code}
For this case, restart is not needed.
Some line is way too long, you can modify your patch after checkstyle results
come out.
> Add a test for bypassing stuck state-machine procedures
> ---
>
> Key: HBASE-21291
> URL: https://issues.apache.org/jira/browse/HBASE-21291
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Attachments: HBASE-21291.master.001.patch
>
>
> {code}
> if (!procedure.isFailed()) {
> if (subprocs != null) {
> if (subprocs.length == 1 && subprocs[0] == procedure) {
> // Procedure returned itself. Quick-shortcut for a state
> machine-like procedure;
> // i.e. we go around this loop again rather than go back out on
> the scheduler queue.
> subprocs = null;
> reExecute = true;
> LOG.trace("Short-circuit to next step on pid={}",
> procedure.getProcId());
> } else {
> // Yield the current procedure, and make the subprocedure runnable
> // subprocs may come back 'null'.
> subprocs = initializeChildren(procStack, procedure, subprocs);
> LOG.info("Initialized subprocedures=" +
> (subprocs == null? null:
> Stream.of(subprocs).map(e -> "{" + e.toString() + "}").
> collect(Collectors.toList()).toString()));
> }
> } else if (procedure.getState() == ProcedureState.WAITING_TIMEOUT) {
> LOG.debug("Added to timeoutExecutor {}", procedure);
> timeoutExecutor.add(procedure);
> } else if (!suspended) {
> // No subtask, so we are done
> procedure.setState(ProcedureState.SUCCESS);
> }
> }
> {code}
> Currently implementation of ProcedureExecutor will set the reExcecute to true
> for state machine like procedure. Then if this procedure is stuck at one
> certain state, it will loop forever.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.getLockEntry(proc.getProcId());
> try {
> executeProcedure(proc);
> } catch (AssertionError e) {
> LOG.info("ASSERT pid=" + proc.getProcId(), e);
> throw e;
> } finally {
> procExecutionLock.releaseLockEntry(lockEntry);
> {code}
> Since procedure will get the IdLock and release it after execution done,
> state machine procedure will never release IdLock until it is finished.
> Then bypassProcedure doesn't work because is will try to grab the IdLock at
> first.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.tryLockEntry(procedure.getProcId(), lockWait);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21242) [amv2] Miscellaneous minor log and assign procedure create improvements
[
https://issues.apache.org/jira/browse/HBASE-21242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647448#comment-16647448
]
Hadoop QA commented on HBASE-21242:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.0 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
9s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
35s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
46s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
25s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
32s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} branch-2.0 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
30s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
7m 48s{color} | {color:green} Patch does not cause any errors with Hadoop 2.6.5
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
7s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
48s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}176m
12s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
56s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}224m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:6f01af0 |
| JIRA Issue | HBASE-21242 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12943560/HBASE-21242.branch-2.0.002.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux abdd3dc321d1 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
[jira] [Commented] (HBASE-13468) hbase.zookeeper.quorum ipv6 address
[ https://issues.apache.org/jira/browse/HBASE-13468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647449#comment-16647449 ] maoling commented on HBASE-13468: - [ZOOKEEPER-3057|https://issues.apache.org/jira/browse/ZOOKEEPER-3057] and [ZOOKEEPER-3106|https://issues.apache.org/jira/browse/ZOOKEEPER-3106] ,now zookeeper server and client supports ipv6 > hbase.zookeeper.quorum ipv6 address > --- > > Key: HBASE-13468 > URL: https://issues.apache.org/jira/browse/HBASE-13468 > Project: HBase > Issue Type: Bug >Reporter: Mingtao Zhang >Priority: Major > > I put ipv6 address in hbase.zookeeper.quorum, by the time this string went to > zookeeper code, the address is messed up, i.e. only '[1234' left. > I started using pseudo mode with embedded zk = true. > I downloaded 1.0.0, not sure which affected version should be here. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21254) Need to find a way to limit the number of proc wal files
[ https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21254: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to branch-2.0+. Thanks [~stack] for reviewing. > Need to find a way to limit the number of proc wal files > > > Key: HBASE-21254 > URL: https://issues.apache.org/jira/browse/HBASE-21254 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch, > HBASE-21254-v3.patch, HBASE-21254.patch > > > For regionserver, we have a max wal file limitation, if we reach the > limitation, we will trigger a flush on specific regions so that we can delete > old wal files. But for proc wals, we do not have this mechanism, and it will > be worse after HBASE-21233, as if there is an old procedure which can not > make progress and do not persist its state, we need to keep the old proc wal > file for ever... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21254) Need to find a way to limit the number of proc wal files
[ https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647334#comment-16647334 ] stack edited comment on HBASE-21254 at 10/12/18 3:48 AM: - +1 to commit to branch-2.0+ was (Author: stack): +1 to commit to branch-2.0 > Need to find a way to limit the number of proc wal files > > > Key: HBASE-21254 > URL: https://issues.apache.org/jira/browse/HBASE-21254 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch, > HBASE-21254-v3.patch, HBASE-21254.patch > > > For regionserver, we have a max wal file limitation, if we reach the > limitation, we will trigger a flush on specific regions so that we can delete > old wal files. But for proc wals, we do not have this mechanism, and it will > be worse after HBASE-21233, as if there is an old procedure which can not > make progress and do not persist its state, we need to keep the old proc wal > file for ever... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21247) Custom WAL Provider cannot be specified by configuration whose value is outside the enums in Providers
[ https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21247: --- Attachment: (was: 21247.v6.txt) > Custom WAL Provider cannot be specified by configuration whose value is > outside the enums in Providers > -- > > Key: HBASE-21247 > URL: https://issues.apache.org/jira/browse/HBASE-21247 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: 3.0.0 > > Attachments: 21247.v1.txt, 21247.v2.txt, 21247.v3.txt, 21247.v4.tst, > 21247.v4.txt, 21247.v5.txt, 21247.v6.txt > > > Currently all the WAL Providers acceptable to hbase are specified in > Providers enum of WALFactory. > This restricts the ability for additional WAL Providers to be supplied - by > class name. > This issue fixes the bug by allowing the specification of new WAL Provider > class name using the config "hbase.wal.provider". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21247) Custom WAL Provider cannot be specified by configuration whose value is outside the enums in Providers
[ https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21247: --- Attachment: 21247.v6.txt > Custom WAL Provider cannot be specified by configuration whose value is > outside the enums in Providers > -- > > Key: HBASE-21247 > URL: https://issues.apache.org/jira/browse/HBASE-21247 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: 3.0.0 > > Attachments: 21247.v1.txt, 21247.v2.txt, 21247.v3.txt, 21247.v4.tst, > 21247.v4.txt, 21247.v5.txt, 21247.v6.txt > > > Currently all the WAL Providers acceptable to hbase are specified in > Providers enum of WALFactory. > This restricts the ability for additional WAL Providers to be supplied - by > class name. > This issue fixes the bug by allowing the specification of new WAL Provider > class name using the config "hbase.wal.provider". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21247) Custom WAL Provider cannot be specified by configuration whose value is outside the enums in Providers
[ https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647351#comment-16647351 ] Ted Yu commented on HBASE-21247: For meta WAL provider, if the user wants to use the same one for hbase.wal.provider, the class name passed via defaultValue to getProviderClass. Patch v6 fixes the test. > Custom WAL Provider cannot be specified by configuration whose value is > outside the enums in Providers > -- > > Key: HBASE-21247 > URL: https://issues.apache.org/jira/browse/HBASE-21247 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: 3.0.0 > > Attachments: 21247.v1.txt, 21247.v2.txt, 21247.v3.txt, 21247.v4.tst, > 21247.v4.txt, 21247.v5.txt, 21247.v6.txt > > > Currently all the WAL Providers acceptable to hbase are specified in > Providers enum of WALFactory. > This restricts the ability for additional WAL Providers to be supplied - by > class name. > This issue fixes the bug by allowing the specification of new WAL Provider > class name using the config "hbase.wal.provider". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21291) Add a test for bypassing stuck state-machine procedures
[
https://issues.apache.org/jira/browse/HBASE-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingyun Tian updated HBASE-21291:
-
Attachment: HBASE-21291.master.001.patch
> Add a test for bypassing stuck state-machine procedures
> ---
>
> Key: HBASE-21291
> URL: https://issues.apache.org/jira/browse/HBASE-21291
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Attachments: HBASE-21291.master.001.patch
>
>
> {code}
> if (!procedure.isFailed()) {
> if (subprocs != null) {
> if (subprocs.length == 1 && subprocs[0] == procedure) {
> // Procedure returned itself. Quick-shortcut for a state
> machine-like procedure;
> // i.e. we go around this loop again rather than go back out on
> the scheduler queue.
> subprocs = null;
> reExecute = true;
> LOG.trace("Short-circuit to next step on pid={}",
> procedure.getProcId());
> } else {
> // Yield the current procedure, and make the subprocedure runnable
> // subprocs may come back 'null'.
> subprocs = initializeChildren(procStack, procedure, subprocs);
> LOG.info("Initialized subprocedures=" +
> (subprocs == null? null:
> Stream.of(subprocs).map(e -> "{" + e.toString() + "}").
> collect(Collectors.toList()).toString()));
> }
> } else if (procedure.getState() == ProcedureState.WAITING_TIMEOUT) {
> LOG.debug("Added to timeoutExecutor {}", procedure);
> timeoutExecutor.add(procedure);
> } else if (!suspended) {
> // No subtask, so we are done
> procedure.setState(ProcedureState.SUCCESS);
> }
> }
> {code}
> Currently implementation of ProcedureExecutor will set the reExcecute to true
> for state machine like procedure. Then if this procedure is stuck at one
> certain state, it will loop forever.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.getLockEntry(proc.getProcId());
> try {
> executeProcedure(proc);
> } catch (AssertionError e) {
> LOG.info("ASSERT pid=" + proc.getProcId(), e);
> throw e;
> } finally {
> procExecutionLock.releaseLockEntry(lockEntry);
> {code}
> Since procedure will get the IdLock and release it after execution done,
> state machine procedure will never release IdLock until it is finished.
> Then bypassProcedure doesn't work because is will try to grab the IdLock at
> first.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.tryLockEntry(procedure.getProcId(), lockWait);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21291) Add a test for bypassing stuck state-machine procedures
[
https://issues.apache.org/jira/browse/HBASE-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647345#comment-16647345
]
Jingyun Tian commented on HBASE-21291:
--
[~stack] A patch contains stuck state machine procedure is uploaded. pls check
it out.
> Add a test for bypassing stuck state-machine procedures
> ---
>
> Key: HBASE-21291
> URL: https://issues.apache.org/jira/browse/HBASE-21291
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Attachments: HBASE-21291.master.001.patch
>
>
> {code}
> if (!procedure.isFailed()) {
> if (subprocs != null) {
> if (subprocs.length == 1 && subprocs[0] == procedure) {
> // Procedure returned itself. Quick-shortcut for a state
> machine-like procedure;
> // i.e. we go around this loop again rather than go back out on
> the scheduler queue.
> subprocs = null;
> reExecute = true;
> LOG.trace("Short-circuit to next step on pid={}",
> procedure.getProcId());
> } else {
> // Yield the current procedure, and make the subprocedure runnable
> // subprocs may come back 'null'.
> subprocs = initializeChildren(procStack, procedure, subprocs);
> LOG.info("Initialized subprocedures=" +
> (subprocs == null? null:
> Stream.of(subprocs).map(e -> "{" + e.toString() + "}").
> collect(Collectors.toList()).toString()));
> }
> } else if (procedure.getState() == ProcedureState.WAITING_TIMEOUT) {
> LOG.debug("Added to timeoutExecutor {}", procedure);
> timeoutExecutor.add(procedure);
> } else if (!suspended) {
> // No subtask, so we are done
> procedure.setState(ProcedureState.SUCCESS);
> }
> }
> {code}
> Currently implementation of ProcedureExecutor will set the reExcecute to true
> for state machine like procedure. Then if this procedure is stuck at one
> certain state, it will loop forever.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.getLockEntry(proc.getProcId());
> try {
> executeProcedure(proc);
> } catch (AssertionError e) {
> LOG.info("ASSERT pid=" + proc.getProcId(), e);
> throw e;
> } finally {
> procExecutionLock.releaseLockEntry(lockEntry);
> {code}
> Since procedure will get the IdLock and release it after execution done,
> state machine procedure will never release IdLock until it is finished.
> Then bypassProcedure doesn't work because is will try to grab the IdLock at
> first.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.tryLockEntry(procedure.getProcId(), lockWait);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-21256: -- Fix Version/s: 2.2.0 1.5.0 > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.2.0 > > Attachments: HBASE-21256-v1.patch, HBASE-21256-v10.patch, > HBASE-21256-v11.patch, HBASE-21256-v12.patch, HBASE-21256-v2.patch, > HBASE-21256-v3.patch, HBASE-21256-v4.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21291) Add a test for bypassing stuck state-machine procedures
[
https://issues.apache.org/jira/browse/HBASE-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingyun Tian updated HBASE-21291:
-
Status: Patch Available (was: Open)
> Add a test for bypassing stuck state-machine procedures
> ---
>
> Key: HBASE-21291
> URL: https://issues.apache.org/jira/browse/HBASE-21291
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Attachments: HBASE-21291.master.001.patch
>
>
> {code}
> if (!procedure.isFailed()) {
> if (subprocs != null) {
> if (subprocs.length == 1 && subprocs[0] == procedure) {
> // Procedure returned itself. Quick-shortcut for a state
> machine-like procedure;
> // i.e. we go around this loop again rather than go back out on
> the scheduler queue.
> subprocs = null;
> reExecute = true;
> LOG.trace("Short-circuit to next step on pid={}",
> procedure.getProcId());
> } else {
> // Yield the current procedure, and make the subprocedure runnable
> // subprocs may come back 'null'.
> subprocs = initializeChildren(procStack, procedure, subprocs);
> LOG.info("Initialized subprocedures=" +
> (subprocs == null? null:
> Stream.of(subprocs).map(e -> "{" + e.toString() + "}").
> collect(Collectors.toList()).toString()));
> }
> } else if (procedure.getState() == ProcedureState.WAITING_TIMEOUT) {
> LOG.debug("Added to timeoutExecutor {}", procedure);
> timeoutExecutor.add(procedure);
> } else if (!suspended) {
> // No subtask, so we are done
> procedure.setState(ProcedureState.SUCCESS);
> }
> }
> {code}
> Currently implementation of ProcedureExecutor will set the reExcecute to true
> for state machine like procedure. Then if this procedure is stuck at one
> certain state, it will loop forever.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.getLockEntry(proc.getProcId());
> try {
> executeProcedure(proc);
> } catch (AssertionError e) {
> LOG.info("ASSERT pid=" + proc.getProcId(), e);
> throw e;
> } finally {
> procExecutionLock.releaseLockEntry(lockEntry);
> {code}
> Since procedure will get the IdLock and release it after execution done,
> state machine procedure will never release IdLock until it is finished.
> Then bypassProcedure doesn't work because is will try to grab the IdLock at
> first.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.tryLockEntry(procedure.getProcId(), lockWait);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647347#comment-16647347 ] Duo Zhang commented on HBASE-21256: --- Thanks [~apurtell], I've added the line of code when pushing to master and branch-2. And the patch does not apply cleanly to branch-1, could you please upload your patch here? Thanks. > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.2.0 > > Attachments: HBASE-21256-v1.patch, HBASE-21256-v10.patch, > HBASE-21256-v11.patch, HBASE-21256-v12.patch, HBASE-21256-v2.patch, > HBASE-21256-v3.patch, HBASE-21256-v4.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21266) Not running balancer because processing dead regionservers, but empty dead rs list
[
https://issues.apache.org/jira/browse/HBASE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647339#comment-16647339
]
Hadoop QA commented on HBASE-21266:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
20s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
51s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
41s{color} | {color:green} hbase-server: The patch generated 0 new + 68
unchanged - 5 fixed = 68 total (was 73) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
56s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 48s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}129m 43s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}156m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestReplicasClient |
| | hadoop.hbase.client.TestAdmin2 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA Issue | HBASE-21266 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12943544/HBASE-21266-branch-1.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti
[jira] [Updated] (HBASE-21247) Custom WAL Provider cannot be specified by configuration whose value is outside the enums in Providers
[ https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21247: --- Attachment: 21247.v6.txt > Custom WAL Provider cannot be specified by configuration whose value is > outside the enums in Providers > -- > > Key: HBASE-21247 > URL: https://issues.apache.org/jira/browse/HBASE-21247 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: 3.0.0 > > Attachments: 21247.v1.txt, 21247.v2.txt, 21247.v3.txt, 21247.v4.tst, > 21247.v4.txt, 21247.v5.txt, 21247.v6.txt > > > Currently all the WAL Providers acceptable to hbase are specified in > Providers enum of WALFactory. > This restricts the ability for additional WAL Providers to be supplied - by > class name. > This issue fixes the bug by allowing the specification of new WAL Provider > class name using the config "hbase.wal.provider". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21291) Add a test for bypassing stuck state-machine procedures
[
https://issues.apache.org/jira/browse/HBASE-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingyun Tian updated HBASE-21291:
-
Summary: Add a test for bypassing stuck state-machine procedures (was:
Bypass doesn't work for state-machine procedures)
> Add a test for bypassing stuck state-machine procedures
> ---
>
> Key: HBASE-21291
> URL: https://issues.apache.org/jira/browse/HBASE-21291
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
>
> {code}
> if (!procedure.isFailed()) {
> if (subprocs != null) {
> if (subprocs.length == 1 && subprocs[0] == procedure) {
> // Procedure returned itself. Quick-shortcut for a state
> machine-like procedure;
> // i.e. we go around this loop again rather than go back out on
> the scheduler queue.
> subprocs = null;
> reExecute = true;
> LOG.trace("Short-circuit to next step on pid={}",
> procedure.getProcId());
> } else {
> // Yield the current procedure, and make the subprocedure runnable
> // subprocs may come back 'null'.
> subprocs = initializeChildren(procStack, procedure, subprocs);
> LOG.info("Initialized subprocedures=" +
> (subprocs == null? null:
> Stream.of(subprocs).map(e -> "{" + e.toString() + "}").
> collect(Collectors.toList()).toString()));
> }
> } else if (procedure.getState() == ProcedureState.WAITING_TIMEOUT) {
> LOG.debug("Added to timeoutExecutor {}", procedure);
> timeoutExecutor.add(procedure);
> } else if (!suspended) {
> // No subtask, so we are done
> procedure.setState(ProcedureState.SUCCESS);
> }
> }
> {code}
> Currently implementation of ProcedureExecutor will set the reExcecute to true
> for state machine like procedure. Then if this procedure is stuck at one
> certain state, it will loop forever.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.getLockEntry(proc.getProcId());
> try {
> executeProcedure(proc);
> } catch (AssertionError e) {
> LOG.info("ASSERT pid=" + proc.getProcId(), e);
> throw e;
> } finally {
> procExecutionLock.releaseLockEntry(lockEntry);
> {code}
> Since procedure will get the IdLock and release it after execution done,
> state machine procedure will never release IdLock until it is finished.
> Then bypassProcedure doesn't work because is will try to grab the IdLock at
> first.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.tryLockEntry(procedure.getProcId(), lockWait);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[ https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647334#comment-16647334 ] stack commented on HBASE-21254: --- +1 to commit to branch-2.0 > Need to find a way to limit the number of proc wal files > > > Key: HBASE-21254 > URL: https://issues.apache.org/jira/browse/HBASE-21254 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch, > HBASE-21254-v3.patch, HBASE-21254.patch > > > For regionserver, we have a max wal file limitation, if we reach the > limitation, we will trigger a flush on specific regions so that we can delete > old wal files. But for proc wals, we do not have this mechanism, and it will > be worse after HBASE-21233, as if there is an old procedure which can not > make progress and do not persist its state, we need to keep the old proc wal > file for ever... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21247) Custom WAL Provider cannot be specified by configuration whose value is outside the enums in Providers
[
https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647312#comment-16647312
]
Hadoop QA commented on HBASE-21247:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
2s{color} | {color:blue} The patch file was not named according to hbase's
naming conventions. Please see
https://yetus.apache.org/documentation/0.8.0/precommit-patchnames for
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
24s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
23s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 1s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}114m 22s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}157m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.wal.TestWALFactory |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21247 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12943538/21247.v5.txt |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 88292b72c4c0 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 924d183ba0 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| unit |
[jira] [Commented] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[
https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647299#comment-16647299
]
Andrew Purtell commented on HBASE-21256:
I was trying this out with a branch-1 backport and noticed this was necessary
when running on a cluster. Needed on branch-2 and up too?
In runRandomInputGenerator:
{code}
diff --git
a/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java
b/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedLis
t.java
index 1b6ded1ed0..181dfcab7d 100644
---
a/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java
+++
b/hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java
@@ -749,6 +786,7 @@ public class IntegrationTestBigLinkedList extends
IntegrationTestBase {
FileOutputFormat.setOutputPath(job, tmpOutput);
job.setOutputFormatClass(SequenceFileOutputFormat.class);
+ TableMapReduceUtil.addDependencyJarsForClasses(job.getConfiguration(),
Random64.class);
boolean success = jobCompletion(job);
{code}
because now the generator needs hbase-common jar
> Improve IntegrationTestBigLinkedList for testing huge data
> --
>
> Key: HBASE-21256
> URL: https://issues.apache.org/jira/browse/HBASE-21256
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Affects Versions: 3.0.0
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-21256-v1.patch, HBASE-21256-v10.patch,
> HBASE-21256-v11.patch, HBASE-21256-v12.patch, HBASE-21256-v2.patch,
> HBASE-21256-v3.patch, HBASE-21256-v4.patch, ITBLL-1.png, ITBLL-2.png
>
>
> Recently, I use ITBLL to test some features in our company. I have
> encountered the following problems:
>
> 1. Generator is too slow at the generating stage, the root cause is
> SecureRandom. There is a global lock in SecureRandom( See the following
> picture). I use Random instead of SecureRandom, and it could speed up this
> stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but
> speaking of generating random bytes, in my opnion,
> it is the same with Random.
> !ITBLL-1.png!
> 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by
> create keyString variable. However, keyString may never be used if test
> result is correct.(and that's in most cases). Just delay creating keyString
> can yield huge performance boost in verifing stage.
> !ITBLL-2.png!
> 3.Arguments check is needed, because there's constraint between arguments. If
> we broken this constraint, we can not get a correct circular list.
>
> 4.Let big family value size could be configured.
>
> 5.Avoid starting RS at backup master
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21185) WALPrettyPrinter: Additional useful info to be printed by wal printer tool, for debugability purposes
[ https://issues.apache.org/jira/browse/HBASE-21185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647291#comment-16647291 ] Sean Busbey commented on HBASE-21185: - thanks for taking care of things [~stack]. > WALPrettyPrinter: Additional useful info to be printed by wal printer tool, > for debugability purposes > - > > Key: HBASE-21185 > URL: https://issues.apache.org/jira/browse/HBASE-21185 > Project: HBase > Issue Type: Improvement > Components: Operability >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Trivial > Fix For: 3.0.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9, 1.2.9 > > Attachments: HBASE-21185.branch-1.4.001.patch, > HBASE-21185.master.001.patch, HBASE-21185.master.002.patch, > HBASE-21185.master.003.patch > > > *WALPrettyPrinter* is very useful for troubleshooting wal issues, such as > faulty replication sinks. An useful information one might want to track is > the size of a single WAL entry edit, as well as size for each edit cell. Am > proposing a patch that adds calculations for these two, as well an option to > seek straight to a given position on the WAL file being analysed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647287#comment-16647287 ] Duo Zhang commented on HBASE-21256: --- Sorry, will commit soon. Thanks for your patient. > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, HBASE-21256-v10.patch, > HBASE-21256-v11.patch, HBASE-21256-v12.patch, HBASE-21256-v2.patch, > HBASE-21256-v3.patch, HBASE-21256-v4.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21256) Improve IntegrationTestBigLinkedList for testing huge data
[ https://issues.apache.org/jira/browse/HBASE-21256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647284#comment-16647284 ] Zephyr Guo commented on HBASE-21256: Hello, [~Apache9] Could you resolve this issue today? Thanks. > Improve IntegrationTestBigLinkedList for testing huge data > -- > > Key: HBASE-21256 > URL: https://issues.apache.org/jira/browse/HBASE-21256 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0 >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21256-v1.patch, HBASE-21256-v10.patch, > HBASE-21256-v11.patch, HBASE-21256-v12.patch, HBASE-21256-v2.patch, > HBASE-21256-v3.patch, HBASE-21256-v4.patch, ITBLL-1.png, ITBLL-2.png > > > Recently, I use ITBLL to test some features in our company. I have > encountered the following problems: > > 1. Generator is too slow at the generating stage, the root cause is > SecureRandom. There is a global lock in SecureRandom( See the following > picture). I use Random instead of SecureRandom, and it could speed up this > stage(500% up with 20 mapper). SecureRandom was brought by HBASE-13382, but > speaking of generating random bytes, in my opnion, > it is the same with Random. > !ITBLL-1.png! > 2. VerifyReducer have a cpu cost of 14% on format method. This is cause by > create keyString variable. However, keyString may never be used if test > result is correct.(and that's in most cases). Just delay creating keyString > can yield huge performance boost in verifing stage. > !ITBLL-2.png! > 3.Arguments check is needed, because there's constraint between arguments. If > we broken this constraint, we can not get a correct circular list. > > 4.Let big family value size could be configured. > > 5.Avoid starting RS at backup master -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21278) TestMergeTableRegionsProcedure is flaky
[
https://issues.apache.org/jira/browse/HBASE-21278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647283#comment-16647283
]
Duo Zhang commented on HBASE-21278:
---
OK, I finally understand the implementation. Every time we execute a procedure
we will push it into a stack, and when rolling back, we start to pop from the
stack to revert the procedures. So maybe we could just skip reverting some
procedures in executeRollback...
> TestMergeTableRegionsProcedure is flaky
> ---
>
> Key: HBASE-21278
> URL: https://issues.apache.org/jira/browse/HBASE-21278
> Project: HBase
> Issue Type: Bug
>Reporter: Duo Zhang
>Priority: Major
> Attachments:
> org.apache.hadoop.hbase.master.assignment.TestMergeTableRegionsProcedure-output.txt
>
>
> https://builds.apache.org/job/HBase-Flaky-Tests/job/master/1235/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.master.assignment.TestMergeTableRegionsProcedure-output.txt/*view*/
> I think the problem is
> {noformat}
> 2018-10-08 03:44:30,315 INFO [PEWorker-1]
> procedure.MasterProcedureScheduler(689): pid=43, ppid=42, state=SUCCESS,
> hasLock=false; TransitRegionStateProcedure
> table=testRollbackAndDoubleExecution,
> region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN checking lock on
> 9bac7c539ac0cff6dc5706ed375a3bfb
> 2018-10-08 03:44:30,320 ERROR [PEWorker-1] helpers.MarkerIgnoringBase(159):
> CODE-BUG: Uncaught runtime exception for pid=43, ppid=42, state=SUCCESS,
> hasLock=true; TransitRegionStateProcedure
> table=testRollbackAndDoubleExecution,
> region=9bac7c539ac0cff6dc5706ed375a3bfb, UNASSIGN
> java.lang.UnsupportedOperationException
> at
> org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.rollbackState(TransitRegionStateProcedure.java:458)
> at
> org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure.rollbackState(TransitRegionStateProcedure.java:97)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.rollback(StateMachineProcedure.java:208)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doRollback(Procedure.java:957)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1605)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1567)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1446)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:76)
> {noformat}
> Typically there is no rollback for TRSP. Need to dig more.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[ https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647269#comment-16647269 ] Duo Zhang commented on HBASE-21254: --- So let's commit? > Need to find a way to limit the number of proc wal files > > > Key: HBASE-21254 > URL: https://issues.apache.org/jira/browse/HBASE-21254 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch, > HBASE-21254-v3.patch, HBASE-21254.patch > > > For regionserver, we have a max wal file limitation, if we reach the > limitation, we will trigger a flush on specific regions so that we can delete > old wal files. But for proc wals, we do not have this mechanism, and it will > be worse after HBASE-21233, as if there is an old procedure which can not > make progress and do not persist its state, we need to keep the old proc wal > file for ever... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20662) Increasing space quota on a violated table does not remove SpaceViolationPolicy.DISABLE enforcement
[
https://issues.apache.org/jira/browse/HBASE-20662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647246#comment-16647246
]
Hadoop QA commented on HBASE-20662:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
18s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} hbase-client: The patch generated 0 new + 5
unchanged - 1 fixed = 5 total (was 6) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} hbase-server: The patch generated 0 new + 162
unchanged - 4 fixed = 162 total (was 166) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
17s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 24s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
7s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}119m
19s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
47s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}172m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20662 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12943519/HBASE-20662.master.004.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 866ba54fe24a 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-21243) Correct java-doc for the method RpcServer.getRemoteAddress()
[
https://issues.apache.org/jira/browse/HBASE-21243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647233#comment-16647233
]
Hadoop QA commented on HBASE-21243:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
16s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
5s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 55s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}221m 1s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}266m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.tool.TestSecureLoadIncrementalHFiles |
| | hadoop.hbase.TestIOFencing |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21243 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12943508/HBASE-21243.master.002.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 527cdf6cd759 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 924d183ba0 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/14657/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
[jira] [Commented] (HBASE-21185) WALPrettyPrinter: Additional useful info to be printed by wal printer tool, for debugability purposes
[ https://issues.apache.org/jira/browse/HBASE-21185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647195#comment-16647195 ] Hudson commented on HBASE-21185: SUCCESS: Integrated in Jenkins build HBase-1.2-IT #1170 (See [https://builds.apache.org/job/HBase-1.2-IT/1170/]) HBASE-21185 - WALPrettyPrinter: Additional useful info to be printed by (stack: rev 6165d4b107a860421a3ffe28bce7e610359d93d6) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALPrettyPrinter.java > WALPrettyPrinter: Additional useful info to be printed by wal printer tool, > for debugability purposes > - > > Key: HBASE-21185 > URL: https://issues.apache.org/jira/browse/HBASE-21185 > Project: HBase > Issue Type: Improvement > Components: Operability >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Trivial > Fix For: 3.0.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9, 1.2.9 > > Attachments: HBASE-21185.branch-1.4.001.patch, > HBASE-21185.master.001.patch, HBASE-21185.master.002.patch, > HBASE-21185.master.003.patch > > > *WALPrettyPrinter* is very useful for troubleshooting wal issues, such as > faulty replication sinks. An useful information one might want to track is > the size of a single WAL entry edit, as well as size for each edit cell. Am > proposing a patch that adds calculations for these two, as well an option to > seek straight to a given position on the WAL file being analysed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[ https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647192#comment-16647192 ] stack commented on HBASE-21254: --- Marked this critical. All goes to hell if we have a stuck procedure holding up WAL cleanup. > Need to find a way to limit the number of proc wal files > > > Key: HBASE-21254 > URL: https://issues.apache.org/jira/browse/HBASE-21254 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch, > HBASE-21254-v3.patch, HBASE-21254.patch > > > For regionserver, we have a max wal file limitation, if we reach the > limitation, we will trigger a flush on specific regions so that we can delete > old wal files. But for proc wals, we do not have this mechanism, and it will > be worse after HBASE-21233, as if there is an old procedure which can not > make progress and do not persist its state, we need to keep the old proc wal > file for ever... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21254) Need to find a way to limit the number of proc wal files
[ https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21254: -- Priority: Critical (was: Major) > Need to find a way to limit the number of proc wal files > > > Key: HBASE-21254 > URL: https://issues.apache.org/jira/browse/HBASE-21254 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch, > HBASE-21254-v3.patch, HBASE-21254.patch > > > For regionserver, we have a max wal file limitation, if we reach the > limitation, we will trigger a flush on specific regions so that we can delete > old wal files. But for proc wals, we do not have this mechanism, and it will > be worse after HBASE-21233, as if there is an old procedure which can not > make progress and do not persist its state, we need to keep the old proc wal > file for ever... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21254) Need to find a way to limit the number of proc wal files
[ https://issues.apache.org/jira/browse/HBASE-21254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647189#comment-16647189 ] stack commented on HBASE-21254: --- I tried this on my test cluster and it seems to be working... 2018-10-11 16:29:35,912 WARN org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: procedure WALs count=12 above the warning threshold 10. check running procedures to see if something is stuck. 2018-10-11 16:29:35,912 INFO org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore: Rolled new Procedure Store WAL, id=12 > Need to find a way to limit the number of proc wal files > > > Key: HBASE-21254 > URL: https://issues.apache.org/jira/browse/HBASE-21254 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21254-v1.patch, HBASE-21254-v2.patch, > HBASE-21254-v3.patch, HBASE-21254.patch > > > For regionserver, we have a max wal file limitation, if we reach the > limitation, we will trigger a flush on specific regions so that we can delete > old wal files. But for proc wals, we do not have this mechanism, and it will > be worse after HBASE-21233, as if there is an old procedure which can not > make progress and do not persist its state, we need to keep the old proc wal > file for ever... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21242) [amv2] Miscellaneous minor log and assign procedure create improvements
[
https://issues.apache.org/jira/browse/HBASE-21242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-21242:
--
Attachment: HBASE-21242.branch-2.0.002.patch
> [amv2] Miscellaneous minor log and assign procedure create improvements
> ---
>
> Key: HBASE-21242
> URL: https://issues.apache.org/jira/browse/HBASE-21242
> Project: HBase
> Issue Type: Bug
> Components: amv2, Operability
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21242.branch-2.0.001.patch,
> HBASE-21242.branch-2.0.001.patch, HBASE-21242.branch-2.0.002.patch,
> HBASE-21242.branch-2.001.patch, HBASE-21242.branch-2.1.001.patch,
> HBASE-21242.branch-2.1.001.patch, HBASE-21242.branch-2.1.001.patch,
> HBASE-21242.branch-2.1.002.patch
>
>
> Some minor fixups:
> {code}
> For RIT Duration, do better than print ms/seconds. Remove redundant UI
> column dedicated to duration when we log it in the status field too.
> Make bypass log at INFO level -- when DEBUG we can miss important
> fixup detail like why we failed.
> Make it so on complete of subprocedure, we note count of outstanding
> siblings so we have a clue how much further the parent has to go before
> it is done (Helpful when hundreds of servers doing SCP).
> Have the SCP run the AP preflight check before creating an AP; saves
> creation of hundreds of thousands of APs during fixup of this big cluster
> of mine.
> Don't log tablename three times when reporting remote call failed.
> If lock is held already, note who has it. Also log after we get lock
> or if we have to wait rather than log on entrance though we may
> later have to wait (or we may have just picked up the lock).
> {code}
> Posting patch in a sec but let me try it on cluster too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21185) WALPrettyPrinter: Additional useful info to be printed by wal printer tool, for debugability purposes
[ https://issues.apache.org/jira/browse/HBASE-21185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647177#comment-16647177 ] Hudson commented on HBASE-21185: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #490 (See [https://builds.apache.org/job/HBase-1.3-IT/490/]) HBASE-21185 - WALPrettyPrinter: Additional useful info to be printed by (stack: rev 0adef4e61b443d50ddced37078c5972ad29dc8d1) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALPrettyPrinter.java > WALPrettyPrinter: Additional useful info to be printed by wal printer tool, > for debugability purposes > - > > Key: HBASE-21185 > URL: https://issues.apache.org/jira/browse/HBASE-21185 > Project: HBase > Issue Type: Improvement > Components: Operability >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Trivial > Fix For: 3.0.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9, 1.2.9 > > Attachments: HBASE-21185.branch-1.4.001.patch, > HBASE-21185.master.001.patch, HBASE-21185.master.002.patch, > HBASE-21185.master.003.patch > > > *WALPrettyPrinter* is very useful for troubleshooting wal issues, such as > faulty replication sinks. An useful information one might want to track is > the size of a single WAL entry edit, as well as size for each edit cell. Am > proposing a patch that adds calculations for these two, as well an option to > seek straight to a given position on the WAL file being analysed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21073) "Maintenance mode" master
[ https://issues.apache.org/jira/browse/HBASE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647147#comment-16647147 ] stack commented on HBASE-21073: --- bq. All of this work is completely disjoint from the earlier maintenance mode, which is why I gave it a different name. I didn't look too deeply into what old maintenance mode did. Would be good to purge this hbck1 assumption of the label 'maintenance mode'. bq. Maybe? Limiting the access is more work, and I'm not sure what it gains us. We'd avoid confused clients thinking the hbase is 'up'. Can I connect to the Master when in maintenance mode with the shelll? That would be useful I'd say. bq. Going to hack around this by skipping the wait entirely. Ok. But maybe leave that to a follow-on. I was in here a while back convinced that the hang at this spot an error but after spending some time, put it off as a 'later'... to be dealt with when we make the startup sequence more sane such that a Master really can be a RS rather than this fake that we have now. Thanks [~mdrob] > "Maintenance mode" master > - > > Key: HBASE-21073 > URL: https://issues.apache.org/jira/browse/HBASE-21073 > Project: HBase > Issue Type: Sub-task > Components: amv2, hbck2, master >Reporter: stack >Assignee: Mike Drob >Priority: Major > Attachments: HBASE-21073.master.001.patch, > HBASE-21073.master.002.patch, HBASE-21073.master.003.patch > > > Make it so we can bring up a Master in "maintenance mode". This is parse of > master wal procs but not taking on regionservers. It would be in a state > where "repair" Procedures could run; e.g. a Procedure that could recover meta > by looking for meta WALs, splitting them, dropping recovered.edits, and even > making it so meta is readable. See parent issue for why needed (disaster > recovery). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21266) Not running balancer because processing dead regionservers, but empty dead rs list
[ https://issues.apache.org/jira/browse/HBASE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647112#comment-16647112 ] Andrew Purtell commented on HBASE-21266: In case it's not clear I'm treating TestZKLessSplitOnCluster and TestEndToEndSplitTransaction like generic flaky tests at this point. I can include the test changes in the patch here or break them out to a subtask. Will do the former unless someone would like it done as the latter. > Not running balancer because processing dead regionservers, but empty dead rs > list > -- > > Key: HBASE-21266 > URL: https://issues.apache.org/jira/browse/HBASE-21266 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0, 1.4.9 > > Attachments: HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch > > > Found during ITBLL testing. AM in master gets into a state where manual > attempts from the shell to run the balancer always return false and this is > printed in the master log: > 2018-10-03 19:17:14,892 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=21,queue=0,port=8100] master.HMaster: > Not running balancer because processing dead regionserver(s): > Note the empty list. > This errant state did not recover without intervention by way of master > restart, but the test environment was chaotic so needs investigation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21266) Not running balancer because processing dead regionservers, but empty dead rs list
[ https://issues.apache.org/jira/browse/HBASE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21266: --- Attachment: HBASE-21266-branch-1.patch > Not running balancer because processing dead regionservers, but empty dead rs > list > -- > > Key: HBASE-21266 > URL: https://issues.apache.org/jira/browse/HBASE-21266 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0, 1.4.9 > > Attachments: HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch > > > Found during ITBLL testing. AM in master gets into a state where manual > attempts from the shell to run the balancer always return false and this is > printed in the master log: > 2018-10-03 19:17:14,892 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=21,queue=0,port=8100] master.HMaster: > Not running balancer because processing dead regionserver(s): > Note the empty list. > This errant state did not recover without intervention by way of master > restart, but the test environment was chaotic so needs investigation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21266) Not running balancer because processing dead regionservers, but empty dead rs list
[ https://issues.apache.org/jira/browse/HBASE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647107#comment-16647107 ] Andrew Purtell edited comment on HBASE-21266 at 10/11/18 10:22 PM: --- TestZKLessSplitOnCluster.testSSHCleanupDaugtherRegionsOfAbortedSplit does hand-rolled waits with 10 ms sleeps. Rewrote those to use Waiter#waitFor with the same timeout and period values of other uses of Waiter#waitFor in this unit. TestEndToEndSplitTransaction.testFromClientSideWhileSplitting utilizes a chore named RegionChecker also with a 10 ms interval, increasing this to 100. This isn't necessary beyond the fact that sleep(10) is obnoxious. Might as well just be a yield() or a spin-wait. There are three instances of sleep(10) in this unit, changed to sleep(100) which IMHO is the smallest reasonable value you want to use if doing short waits. was (Author: apurtell): TestZKLessSplitOnCluster.testSSHCleanupDaugtherRegionsOfAbortedSplit does hand-rolled waits with 10 ms sleeps. Rewrote those to use Waiter#waitFor with the same timeout and period values of other uses of Waiter#waitFor in this unit. TestEndToEndSplitTransaction.testFromClientSideWhileSplitting utilizes a chore named RegionChecker also with a 10 ms interval, increasing this to 100. This isn't necessary beyond the fact that sleep(10) is obnoxious. Might as well just be a yield() or a spin-wait. > Not running balancer because processing dead regionservers, but empty dead rs > list > -- > > Key: HBASE-21266 > URL: https://issues.apache.org/jira/browse/HBASE-21266 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0, 1.4.9 > > Attachments: HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch > > > Found during ITBLL testing. AM in master gets into a state where manual > attempts from the shell to run the balancer always return false and this is > printed in the master log: > 2018-10-03 19:17:14,892 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=21,queue=0,port=8100] master.HMaster: > Not running balancer because processing dead regionserver(s): > Note the empty list. > This errant state did not recover without intervention by way of master > restart, but the test environment was chaotic so needs investigation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21266) Not running balancer because processing dead regionservers, but empty dead rs list
[ https://issues.apache.org/jira/browse/HBASE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647107#comment-16647107 ] Andrew Purtell commented on HBASE-21266: TestZKLessSplitOnCluster.testSSHCleanupDaugtherRegionsOfAbortedSplit does hand-rolled waits with 10 ms sleeps. Rewrote those to use Waiter#waitFor with the same timeout and period values of other uses of Waiter#waitFor in this unit. TestEndToEndSplitTransaction.testFromClientSideWhileSplitting utilizes a chore named RegionChecker also with a 10 ms interval, increasing this to 100. This isn't necessary beyond the fact that sleep(10) is obnoxious. Might as well just be a yield() or a spin-wait. > Not running balancer because processing dead regionservers, but empty dead rs > list > -- > > Key: HBASE-21266 > URL: https://issues.apache.org/jira/browse/HBASE-21266 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0, 1.4.9 > > Attachments: HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch > > > Found during ITBLL testing. AM in master gets into a state where manual > attempts from the shell to run the balancer always return false and this is > printed in the master log: > 2018-10-03 19:17:14,892 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=21,queue=0,port=8100] master.HMaster: > Not running balancer because processing dead regionserver(s): > Note the empty list. > This errant state did not recover without intervention by way of master > restart, but the test environment was chaotic so needs investigation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21103) nightly test cache of yetus install needs to be more thorough in verification
[
https://issues.apache.org/jira/browse/HBASE-21103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647104#comment-16647104
]
Hudson commented on HBASE-21103:
Results for branch branch-2.1
[build #451 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/451/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/451//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/451//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/451//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> nightly test cache of yetus install needs to be more thorough in verification
> -
>
> Key: HBASE-21103
> URL: https://issues.apache.org/jira/browse/HBASE-21103
> Project: HBase
> Issue Type: Improvement
> Components: test
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Labels: beginner
> Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 2.1.1, 2.0.3, 1.4.9
>
> Attachments: HBASE-21103-branch-1.v0.patch, HBASE-21103.0.patch
>
>
> branch-1.2 nightly failed because it couldn't find yetus:
> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/443/
> walking through steps, we checked and thought we had an existing install:
> https://builds.apache.org/blue/organizations/jenkins/HBase%20Nightly/detail/branch-1.2/443/pipeline/12
> {code}
> [HBase_Nightly_branch-1.2-AEAO7LJNYKPIH3O4GB2IR4TEIHRS6EXOJCJPGWYXWO6BFPUEZT3Q]
> Running shell script
> Ensure we have a copy of Apache Yetus.
> Checking for Yetus 0.6.0 in
> '/home/jenkins/jenkins-slave/workspace/HBase_Nightly_branch-1.2-AEAO7LJNYKPIH3O4GB2IR4TEIHRS6EXOJCJPGWYXWO6BFPUEZT3Q/yetus-0.6.0'
> Reusing cached download of Apache Yetus version 0.6.0.
> {code}
> So we stashed and then tried to use it.
> Examining the workspace present before the stash shows the directory we look
> for was present, but the contents were garbage:
> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/443/execution/node/3/ws/
> {code}
> $ unzip -l yetus-0.6.0.zip
> Archive: yetus-0.6.0.zip
> Length DateTimeName
> - -- -
> 0 00-00-1980 04:08 yetus-0.6.0/lib/precommit/qbt.sh
> - ---
> 0 1 file
> {code}
> we should probably check for an executable that will successfully give us a
> version or something like that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21298) Improved scheduling ("Muzzled HBase")
stack created HBASE-21298:
-
Summary: Improved scheduling ("Muzzled HBase")
Key: HBASE-21298
URL: https://issues.apache.org/jira/browse/HBASE-21298
Project: HBase
Issue Type: Improvement
Reporter: stack
Interesting paper on Thread scheduling with hbase for illustration. After study
using interesting Thread analysis tooling, authors were able to improve HBase
throughputs ("Muzzled-HBase").
https://www.usenix.org/system/files/osdi18-yang.pdf
(Via [~ebort...@oath.com])
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21282) Upgrade Jetty dependencies to latest in major-line
[
https://issues.apache.org/jira/browse/HBASE-21282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647086#comment-16647086
]
Hudson commented on HBASE-21282:
Results for branch master
[build #540 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/540/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/540//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/540//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/540//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Upgrade Jetty dependencies to latest in major-line
> --
>
> Key: HBASE-21282
> URL: https://issues.apache.org/jira/browse/HBASE-21282
> Project: HBase
> Issue Type: Task
> Components: dependencies
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21282.001.branch-2.0.patch
>
>
> Looks like we have dependencies on both jetty 9.2 and 9.3, but we're lagging
> pretty far behind in both. We can upgrade both of these to the latest (august
> 2018).
>
> I'll also have to take a look at why we're using two separate versions (maybe
> we didn't want to switch from jetty-jsp to apache-jsp on 9.2->9.3?). Not sure
> if there's a good reason for this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21103) nightly test cache of yetus install needs to be more thorough in verification
[
https://issues.apache.org/jira/browse/HBASE-21103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647087#comment-16647087
]
Hudson commented on HBASE-21103:
Results for branch master
[build #540 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/540/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/540//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/540//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/540//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> nightly test cache of yetus install needs to be more thorough in verification
> -
>
> Key: HBASE-21103
> URL: https://issues.apache.org/jira/browse/HBASE-21103
> Project: HBase
> Issue Type: Improvement
> Components: test
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Labels: beginner
> Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 2.1.1, 2.0.3, 1.4.9
>
> Attachments: HBASE-21103-branch-1.v0.patch, HBASE-21103.0.patch
>
>
> branch-1.2 nightly failed because it couldn't find yetus:
> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/443/
> walking through steps, we checked and thought we had an existing install:
> https://builds.apache.org/blue/organizations/jenkins/HBase%20Nightly/detail/branch-1.2/443/pipeline/12
> {code}
> [HBase_Nightly_branch-1.2-AEAO7LJNYKPIH3O4GB2IR4TEIHRS6EXOJCJPGWYXWO6BFPUEZT3Q]
> Running shell script
> Ensure we have a copy of Apache Yetus.
> Checking for Yetus 0.6.0 in
> '/home/jenkins/jenkins-slave/workspace/HBase_Nightly_branch-1.2-AEAO7LJNYKPIH3O4GB2IR4TEIHRS6EXOJCJPGWYXWO6BFPUEZT3Q/yetus-0.6.0'
> Reusing cached download of Apache Yetus version 0.6.0.
> {code}
> So we stashed and then tried to use it.
> Examining the workspace present before the stash shows the directory we look
> for was present, but the contents were garbage:
> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/443/execution/node/3/ws/
> {code}
> $ unzip -l yetus-0.6.0.zip
> Archive: yetus-0.6.0.zip
> Length DateTimeName
> - -- -
> 0 00-00-1980 04:08 yetus-0.6.0/lib/precommit/qbt.sh
> - ---
> 0 1 file
> {code}
> we should probably check for an executable that will successfully give us a
> version or something like that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21185) WALPrettyPrinter: Additional useful info to be printed by wal printer tool, for debugability purposes
[ https://issues.apache.org/jira/browse/HBASE-21185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21185: -- Resolution: Fixed Fix Version/s: 1.2.9 1.4.9 1.3.3 Status: Resolved (was: Patch Available) Re-resolving after pushing to branch-1.2+. > WALPrettyPrinter: Additional useful info to be printed by wal printer tool, > for debugability purposes > - > > Key: HBASE-21185 > URL: https://issues.apache.org/jira/browse/HBASE-21185 > Project: HBase > Issue Type: Improvement > Components: Operability >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Trivial > Fix For: 3.0.0, 1.3.3, 2.2.0, 2.1.1, 2.0.3, 1.4.9, 1.2.9 > > Attachments: HBASE-21185.branch-1.4.001.patch, > HBASE-21185.master.001.patch, HBASE-21185.master.002.patch, > HBASE-21185.master.003.patch > > > *WALPrettyPrinter* is very useful for troubleshooting wal issues, such as > faulty replication sinks. An useful information one might want to track is > the size of a single WAL entry edit, as well as size for each edit cell. Am > proposing a patch that adds calculations for these two, as well an option to > seek straight to a given position on the WAL file being analysed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21247) Custom WAL Provider cannot be specified by configuration whose value is outside the enums in Providers
[ https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21247: --- Status: Patch Available (was: Reopened) > Custom WAL Provider cannot be specified by configuration whose value is > outside the enums in Providers > -- > > Key: HBASE-21247 > URL: https://issues.apache.org/jira/browse/HBASE-21247 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: 3.0.0 > > Attachments: 21247.v1.txt, 21247.v2.txt, 21247.v3.txt, 21247.v4.tst, > 21247.v4.txt, 21247.v5.txt > > > Currently all the WAL Providers acceptable to hbase are specified in > Providers enum of WALFactory. > This restricts the ability for additional WAL Providers to be supplied - by > class name. > This issue fixes the bug by allowing the specification of new WAL Provider > class name using the config "hbase.wal.provider". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21247) Custom WAL Provider cannot be specified by configuration whose value is outside the enums in Providers
[ https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21247: --- Attachment: 21247.v5.txt > Custom WAL Provider cannot be specified by configuration whose value is > outside the enums in Providers > -- > > Key: HBASE-21247 > URL: https://issues.apache.org/jira/browse/HBASE-21247 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: 3.0.0 > > Attachments: 21247.v1.txt, 21247.v2.txt, 21247.v3.txt, 21247.v4.tst, > 21247.v4.txt, 21247.v5.txt > > > Currently all the WAL Providers acceptable to hbase are specified in > Providers enum of WALFactory. > This restricts the ability for additional WAL Providers to be supplied - by > class name. > This issue fixes the bug by allowing the specification of new WAL Provider > class name using the config "hbase.wal.provider". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21073) "Maintenance mode" master
[ https://issues.apache.org/jira/browse/HBASE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647061#comment-16647061 ] Mike Drob commented on HBASE-21073: --- bq. What does this do (smile)? Right now, if there is a boolean configured in the hbase-site.xml then when master comes up it will: # host regions for system tables (on master) # not assign any user regions # skip setting up a few of the chores All of this work is completely disjoint from the earlier maintenance mode, which is why I gave it a different name. I didn't look too deeply into what old maintenance mode did. bq. Should maintenance mode bring up the Master but only have it listening for Connection from localhost? Maybe? Limiting the access is more work, and I'm not sure what it gains us. bq. Is that it? This bit of code here is strange. In usual case – no regions on master – we actually seem to hang here... which doesn't seem right. It hangs... Yea, there's some race here when we need meta on master since we won't mark the master as active until meta is hosted, but the region server half of the same server won't come up until there's an active master. Normally a region server would be able to pick up meta without waiting for an active master to register with. Going to hack around this by skipping the wait entirely. A little nervous about it because I don't fully understand it, but I think it will be ok. bq. its setting a running master into a 'state' so hbck1 could run and then restoring old state after. Gotcha. Will look at this for the next revision. > "Maintenance mode" master > - > > Key: HBASE-21073 > URL: https://issues.apache.org/jira/browse/HBASE-21073 > Project: HBase > Issue Type: Sub-task > Components: amv2, hbck2, master >Reporter: stack >Assignee: Mike Drob >Priority: Major > Attachments: HBASE-21073.master.001.patch, > HBASE-21073.master.002.patch, HBASE-21073.master.003.patch > > > Make it so we can bring up a Master in "maintenance mode". This is parse of > master wal procs but not taking on regionservers. It would be in a state > where "repair" Procedures could run; e.g. a Procedure that could recover meta > by looking for meta WALs, splitting them, dropping recovered.edits, and even > making it so meta is readable. See parent issue for why needed (disaster > recovery). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21073) "Maintenance mode" master
[ https://issues.apache.org/jira/browse/HBASE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-21073: -- Attachment: HBASE-21073.master.003.patch > "Maintenance mode" master > - > > Key: HBASE-21073 > URL: https://issues.apache.org/jira/browse/HBASE-21073 > Project: HBase > Issue Type: Sub-task > Components: amv2, hbck2, master >Reporter: stack >Assignee: Mike Drob >Priority: Major > Attachments: HBASE-21073.master.001.patch, > HBASE-21073.master.002.patch, HBASE-21073.master.003.patch > > > Make it so we can bring up a Master in "maintenance mode". This is parse of > master wal procs but not taking on regionservers. It would be in a state > where "repair" Procedures could run; e.g. a Procedure that could recover meta > by looking for meta WALs, splitting them, dropping recovered.edits, and even > making it so meta is readable. See parent issue for why needed (disaster > recovery). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21268) Backport to branch-2.0 " HBASE-21213 [hbck2] bypass leaves behind state in RegionStates when assign/unassign"
[ https://issues.apache.org/jira/browse/HBASE-21268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21268: -- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to branch-2.0 after fixing one of the checkstyles and the rubycop complaint. > Backport to branch-2.0 " HBASE-21213 [hbck2] bypass leaves behind state in > RegionStates when assign/unassign" > - > > Key: HBASE-21268 > URL: https://issues.apache.org/jira/browse/HBASE-21268 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Affects Versions: 2.0.2 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.3 > > Attachments: HBASE-21268.branch-2.0.001.patch, > HBASE-21268.branch-2.0.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21103) nightly test cache of yetus install needs to be more thorough in verification
[
https://issues.apache.org/jira/browse/HBASE-21103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647054#comment-16647054
]
Hudson commented on HBASE-21103:
Results for branch branch-2
[build #1375 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1375/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1375//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1375//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1375//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> nightly test cache of yetus install needs to be more thorough in verification
> -
>
> Key: HBASE-21103
> URL: https://issues.apache.org/jira/browse/HBASE-21103
> Project: HBase
> Issue Type: Improvement
> Components: test
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Labels: beginner
> Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 2.1.1, 2.0.3, 1.4.9
>
> Attachments: HBASE-21103-branch-1.v0.patch, HBASE-21103.0.patch
>
>
> branch-1.2 nightly failed because it couldn't find yetus:
> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/443/
> walking through steps, we checked and thought we had an existing install:
> https://builds.apache.org/blue/organizations/jenkins/HBase%20Nightly/detail/branch-1.2/443/pipeline/12
> {code}
> [HBase_Nightly_branch-1.2-AEAO7LJNYKPIH3O4GB2IR4TEIHRS6EXOJCJPGWYXWO6BFPUEZT3Q]
> Running shell script
> Ensure we have a copy of Apache Yetus.
> Checking for Yetus 0.6.0 in
> '/home/jenkins/jenkins-slave/workspace/HBase_Nightly_branch-1.2-AEAO7LJNYKPIH3O4GB2IR4TEIHRS6EXOJCJPGWYXWO6BFPUEZT3Q/yetus-0.6.0'
> Reusing cached download of Apache Yetus version 0.6.0.
> {code}
> So we stashed and then tried to use it.
> Examining the workspace present before the stash shows the directory we look
> for was present, but the contents were garbage:
> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/443/execution/node/3/ws/
> {code}
> $ unzip -l yetus-0.6.0.zip
> Archive: yetus-0.6.0.zip
> Length DateTimeName
> - -- -
> 0 00-00-1980 04:08 yetus-0.6.0/lib/precommit/qbt.sh
> - ---
> 0 1 file
> {code}
> we should probably check for an executable that will successfully give us a
> version or something like that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21282) Upgrade Jetty dependencies to latest in major-line
[
https://issues.apache.org/jira/browse/HBASE-21282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647053#comment-16647053
]
Hudson commented on HBASE-21282:
Results for branch branch-2
[build #1375 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1375/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1375//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1375//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1375//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Upgrade Jetty dependencies to latest in major-line
> --
>
> Key: HBASE-21282
> URL: https://issues.apache.org/jira/browse/HBASE-21282
> Project: HBase
> Issue Type: Task
> Components: dependencies
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21282.001.branch-2.0.patch
>
>
> Looks like we have dependencies on both jetty 9.2 and 9.3, but we're lagging
> pretty far behind in both. We can upgrade both of these to the latest (august
> 2018).
>
> I'll also have to take a look at why we're using two separate versions (maybe
> we didn't want to switch from jetty-jsp to apache-jsp on 9.2->9.3?). Not sure
> if there's a good reason for this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21266) Not running balancer because processing dead regionservers, but empty dead rs list
[ https://issues.apache.org/jira/browse/HBASE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647025#comment-16647025 ] Andrew Purtell commented on HBASE-21266: TestEndToEndSplitTransaction.testFromClientSideWhileSplitting doesn't involve dead server processing directly. TestZKLessSplitOnCluster.testSSHCleanupDaugtherRegionsOfAbortedSplit waits on ServerManager#areDeadServersInProgress so this change seems to have affected timing in this test with the result of making it unstable. Will look into it. > Not running balancer because processing dead regionservers, but empty dead rs > list > -- > > Key: HBASE-21266 > URL: https://issues.apache.org/jira/browse/HBASE-21266 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0, 1.4.9 > > Attachments: HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch > > > Found during ITBLL testing. AM in master gets into a state where manual > attempts from the shell to run the balancer always return false and this is > printed in the master log: > 2018-10-03 19:17:14,892 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=21,queue=0,port=8100] master.HMaster: > Not running balancer because processing dead regionserver(s): > Note the empty list. > This errant state did not recover without intervention by way of master > restart, but the test environment was chaotic so needs investigation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21287) JVMClusterUtil Master initialization wait time not configurable
[
https://issues.apache.org/jira/browse/HBASE-21287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647023#comment-16647023
]
Hudson commented on HBASE-21287:
Results for branch branch-2.1
[build #450 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/450/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/450//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/450//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/450//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> JVMClusterUtil Master initialization wait time not configurable
> ---
>
> Key: HBASE-21287
> URL: https://issues.apache.org/jira/browse/HBASE-21287
> Project: HBase
> Issue Type: Task
> Components: test
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21287.patch
>
>
> We can configure how long the local cluster threads will wait for master to
> come up and become active, but not how long we allow initialization to take.
> Being able to tune this would improve my test loop on some experiment I am
> running.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21073) "Maintenance mode" master
[ https://issues.apache.org/jira/browse/HBASE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647019#comment-16647019 ] stack commented on HBASE-21073: --- On patch, call it maintenanceMode rather than repairMode? > "Maintenance mode" master > - > > Key: HBASE-21073 > URL: https://issues.apache.org/jira/browse/HBASE-21073 > Project: HBase > Issue Type: Sub-task > Components: amv2, hbck2, master >Reporter: stack >Assignee: Mike Drob >Priority: Major > Attachments: HBASE-21073.master.001.patch, > HBASE-21073.master.002.patch > > > Make it so we can bring up a Master in "maintenance mode". This is parse of > master wal procs but not taking on regionservers. It would be in a state > where "repair" Procedures could run; e.g. a Procedure that could recover meta > by looking for meta WALs, splitting them, dropping recovered.edits, and even > making it so meta is readable. See parent issue for why needed (disaster > recovery). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21287) JVMClusterUtil Master initialization wait time not configurable
[
https://issues.apache.org/jira/browse/HBASE-21287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647016#comment-16647016
]
Hudson commented on HBASE-21287:
Results for branch master
[build #539 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/539/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/539//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/539//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/539//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> JVMClusterUtil Master initialization wait time not configurable
> ---
>
> Key: HBASE-21287
> URL: https://issues.apache.org/jira/browse/HBASE-21287
> Project: HBase
> Issue Type: Task
> Components: test
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21287.patch
>
>
> We can configure how long the local cluster threads will wait for master to
> come up and become active, but not how long we allow initialization to take.
> Being able to tune this would improve my test loop on some experiment I am
> running.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21281) Update bouncycastle dependency.
[
https://issues.apache.org/jira/browse/HBASE-21281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647017#comment-16647017
]
Hudson commented on HBASE-21281:
Results for branch master
[build #539 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/539/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/539//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/539//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/539//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Update bouncycastle dependency.
> ---
>
> Key: HBASE-21281
> URL: https://issues.apache.org/jira/browse/HBASE-21281
> Project: HBase
> Issue Type: Task
> Components: dependencies, test
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21281.001.branch-2.0.patch
>
>
> Looks like we still depend on bcprov-jdk16 for some x509 certificate
> generation in our tests. Bouncycastle has moved beyond this in 1.47, changing
> the artifact names.
> [http://www.bouncycastle.org/wiki/display/JA1/Porting+from+earlier+BC+releases+to+1.47+and+later]
> There are some API changes too, but it looks like we don't use any of these.
> It seems like we also have vestiges in the POMs from when we were depending
> on a specific BC version that came in from Hadoop. We now have a
> KeyStoreTestUtil class in HBase, which makes me think we can also clean up
> some dependencies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21073) "Maintenance mode" master
[ https://issues.apache.org/jira/browse/HBASE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647013#comment-16647013 ] stack commented on HBASE-21073: --- What does this do (smile)? There is already a log message (?) about master being in maintenance mode... You saw that (Perhaps related to HBASE-16008 ?)? Has that been appropriated by this patch? bq. I have no idea what happens if we try to scan one of the user space tables. A client would hang, right, then fail because it would try to go to server hosting the user-space region. Should maintenance mode bring up the Master but only have it listening for Connection from localhost? bq. Why do I need to explicitly set TABLES_ON_MASTER to true in the test? I haven't yet found where Master reads that value to know that it should report as an RS. hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java: boolean tablesOnMaster = LoadBalancer.isTablesOnMaster(conf); Is that it? This bit of code here is strange. In usual case -- no regions on master -- we actually seem to hang here... which doesn't seem right. Rather than fold this into HBASE-16008, HBASE-16008 should be purged or folded in here. HBASE-16008's notion of 'maintenance mode' is strange... its setting a running master into a 'state' so hbck1 could run and then restoring old state after. > "Maintenance mode" master > - > > Key: HBASE-21073 > URL: https://issues.apache.org/jira/browse/HBASE-21073 > Project: HBase > Issue Type: Sub-task > Components: amv2, hbck2, master >Reporter: stack >Assignee: Mike Drob >Priority: Major > Attachments: HBASE-21073.master.001.patch, > HBASE-21073.master.002.patch > > > Make it so we can bring up a Master in "maintenance mode". This is parse of > master wal procs but not taking on regionservers. It would be in a state > where "repair" Procedures could run; e.g. a Procedure that could recover meta > by looking for meta WALs, splitting them, dropping recovered.edits, and even > making it so meta is readable. See parent issue for why needed (disaster > recovery). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21295) Update compatibility matrices
[
https://issues.apache.org/jira/browse/HBASE-21295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647011#comment-16647011
]
Hadoop QA commented on HBASE-21295:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 5m
25s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 5m
16s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21295 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12943493/HBASE-21295-v2.patch |
| Optional Tests | dupname asflicense refguide |
| uname | Linux d0a17a0f85b4 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 924d183ba0 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| refguide |
https://builds.apache.org/job/PreCommit-HBASE-Build/14658/artifact/patchprocess/branch-site/book.html
|
| refguide |
https://builds.apache.org/job/PreCommit-HBASE-Build/14658/artifact/patchprocess/patch-site/book.html
|
| Max. process+thread count | 87 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/14658/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Update compatibility matrices
> -
>
> Key: HBASE-21295
> URL: https://issues.apache.org/jira/browse/HBASE-21295
> Project: HBase
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 3.0.0
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Minor
> Attachments: HBASE-21295-v1.patch, HBASE-21295-v2.patch
>
>
> JDK and Hadoop compatibility matrices does not cover all of the current
> releases and available JDKs. These should be added to both tables.
> [https://hbase.apache.org/book.html#hadoop]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21266) Not running balancer because processing dead regionservers, but empty dead rs list
[
https://issues.apache.org/jira/browse/HBASE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647007#comment-16647007
]
Andrew Purtell edited comment on HBASE-21266 at 10/11/18 8:44 PM:
--
Those test failures in precommit might be flakes, let me see if I can reproduce
them.
I ran split, merge, assignment, and balancer tests, including the tests in
question, and am not seeing any issues.
{noformat}
[INFO] Running org.apache.hadoop.hbase.util.TestMergeTable
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.85 s
- in org.apache.hadoop.hbase.util.TestMergeTable
[INFO] Running org.apache.hadoop.hbase.util.TestMergeTool
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.995 s
- in org.apache.hadoop.hbase.util.TestMergeTool
[INFO] Running org.apache.hadoop.hbase.util.TestRegionSplitter
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.962 s
- in org.apache.hadoop.hbase.util.TestRegionSplitter
[INFO] Running org.apache.hadoop.hbase.util.TestRegionSplitCalculator
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.968 s
- in org.apache.hadoop.hbase.util.TestRegionSplitCalculator
[INFO] Running org.apache.hadoop.hbase.wal.TestWALSplit
[INFO] Tests run: 33, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 46.067
s - in org.apache.hadoop.hbase.wal.TestWALSplit
[INFO] Running org.apache.hadoop.hbase.wal.TestWALSplitBoundedLogWriterCreation
[WARNING] Tests run: 33, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
43.419 s - in org.apache.hadoop.hbase.wal.TestWALSplitBoundedLogWriterCreation
[INFO] Running org.apache.hadoop.hbase.wal.TestWALSplitCompressed
[INFO] Tests run: 33, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 46.075
s - in org.apache.hadoop.hbase.wal.TestWALSplitCompressed
[INFO] Running org.apache.hadoop.hbase.mapred.TestSplitTable
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.774 s
- in org.apache.hadoop.hbase.mapred.TestSplitTable
[INFO] Running org.apache.hadoop.hbase.regionserver.TestRegionSplitPolicy
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.836 s
- in org.apache.hadoop.hbase.regionserver.TestRegionSplitPolicy
[INFO] Running org.apache.hadoop.hbase.regionserver.TestCompactSplitThread
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.692 s
- in org.apache.hadoop.hbase.regionserver.TestCompactSplitThread
[INFO] Running
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
[INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 87.588
s - in org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
[INFO] Running org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.796 s
- in org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss
[INFO] Running org.apache.hadoop.hbase.regionserver.TestSplitTransaction
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.14 s -
in org.apache.hadoop.hbase.regionserver.TestSplitTransaction
[INFO] Running org.apache.hadoop.hbase.regionserver.TestRegionMergeTransaction
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.715 s
- in org.apache.hadoop.hbase.regionserver.TestRegionMergeTransaction
[INFO] Running org.apache.hadoop.hbase.regionserver.TestZKLessSplitOnCluster
[INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 71.721
s - in org.apache.hadoop.hbase.regionserver.TestZKLessSplitOnCluster
[INFO] Running org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 59.082 s
- in org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction
[INFO] Running org.apache.hadoop.hbase.regionserver.TestSplitLogWorker
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.689 s
- in org.apache.hadoop.hbase.regionserver.TestSplitLogWorker
[INFO] Running org.apache.hadoop.hbase.regionserver.TestZKLessMergeOnCluster
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.068 s
- in org.apache.hadoop.hbase.regionserver.TestZKLessMergeOnCluster
[INFO] Running
org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.386 s
- in org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster
[INFO] Running org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster
[INFO] Tests run: 23, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 185.692
s - in org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster
[INFO] Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting
[WARNING] Tests run: 18, Failures: 0, Errors: 0, Skipped: 15, Time elapsed:
90.971 s - in
[jira] [Commented] (HBASE-21266) Not running balancer because processing dead regionservers, but empty dead rs list
[
https://issues.apache.org/jira/browse/HBASE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647007#comment-16647007
]
Andrew Purtell commented on HBASE-21266:
Those test failures in precommit might be flakes, let me see if I can reproduce
them.
I ran split, merge, assignment, and balancer tests, including the tests in
question, and am not seeing any issues.
{noformat}
[INFO] Running org.apache.hadoop.hbase.util.TestMergeTable
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.85 s
- in org.apache.hadoop.hbase.util.TestMergeTable
[INFO] Running org.apache.hadoop.hbase.util.TestMergeTool
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.995 s
- in org.apache.hadoop.hbase.util.TestMergeTool
[INFO] Running org.apache.hadoop.hbase.util.TestRegionSplitter
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.962 s
- in org.apache.hadoop.hbase.util.TestRegionSplitter
[INFO] Running org.apache.hadoop.hbase.util.TestRegionSplitCalculator
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.968 s
- in org.apache.hadoop.hbase.util.TestRegionSplitCalculator
[INFO] Running org.apache.hadoop.hbase.wal.TestWALSplit
[INFO] Tests run: 33, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 46.067
s - in org.apache.hadoop.hbase.wal.TestWALSplit
[INFO] Running org.apache.hadoop.hbase.wal.TestWALSplitBoundedLogWriterCreation
[WARNING] Tests run: 33, Failures: 0, Errors: 0, Skipped: 1, Time elapsed:
43.419 s - in org.apache.hadoop.hbase.wal.TestWALSplitBoundedLogWriterCreation
[INFO] Running org.apache.hadoop.hbase.wal.TestWALSplitCompressed
[INFO] Tests run: 33, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 46.075
s - in org.apache.hadoop.hbase.wal.TestWALSplitCompressed
[INFO] Running org.apache.hadoop.hbase.mapred.TestSplitTable
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.774 s
- in org.apache.hadoop.hbase.mapred.TestSplitTable
[INFO] Running org.apache.hadoop.hbase.regionserver.TestRegionSplitPolicy
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.836 s
- in org.apache.hadoop.hbase.regionserver.TestRegionSplitPolicy
[INFO] Running org.apache.hadoop.hbase.regionserver.TestCompactSplitThread
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 30.692 s
- in org.apache.hadoop.hbase.regionserver.TestCompactSplitThread
[INFO] Running
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
[INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 87.588
s - in org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
[INFO] Running org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.796 s
- in org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss
[INFO] Running org.apache.hadoop.hbase.regionserver.TestSplitTransaction
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.14 s -
in org.apache.hadoop.hbase.regionserver.TestSplitTransaction
[INFO] Running org.apache.hadoop.hbase.regionserver.TestRegionMergeTransaction
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.715 s
- in org.apache.hadoop.hbase.regionserver.TestRegionMergeTransaction
[INFO] Running org.apache.hadoop.hbase.regionserver.TestZKLessSplitOnCluster
[INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 71.721
s - in org.apache.hadoop.hbase.regionserver.TestZKLessSplitOnCluster
[INFO] Running org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 59.082 s
- in org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction
[INFO] Running org.apache.hadoop.hbase.regionserver.TestSplitLogWorker
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.689 s
- in org.apache.hadoop.hbase.regionserver.TestSplitLogWorker
[INFO] Running org.apache.hadoop.hbase.regionserver.TestZKLessMergeOnCluster
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.068 s
- in org.apache.hadoop.hbase.regionserver.TestZKLessMergeOnCluster
[INFO] Running
org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.386 s
- in org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster
[INFO] Running org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster
[INFO] Tests run: 23, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 185.692
s - in org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster
[INFO] Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting
[WARNING] Tests run: 18, Failures: 0, Errors: 0, Skipped: 15, Time elapsed:
90.971 s - in org.apache.hadoop.hbase.master.TestDistributedLogSplitting
[INFO]
[jira] [Created] (HBASE-21297) ModifyTableProcedure can throw TNDE instead of IOE in case of REGION_REPLICATION change
Nihal Jain created HBASE-21297:
--
Summary: ModifyTableProcedure can throw TNDE instead of IOE in
case of REGION_REPLICATION change
Key: HBASE-21297
URL: https://issues.apache.org/jira/browse/HBASE-21297
Project: HBase
Issue Type: Improvement
Reporter: Nihal Jain
Assignee: Nihal Jain
Currently {{ModifyTableProcedure}} throws an {{IOException}} (See
[ModifyTableProcedure.java#L252|https://github.com/apache/hbase/blob/924d183ba0e67b975e998f6006c993f457e03c20/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ModifyTableProcedure.java#L252])
when a user tries to modify REGION_REPLICATION for an enabled table. Instead,
it can throw a more specific {{TableNotDisabledException}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20662) Increasing space quota on a violated table does not remove SpaceViolationPolicy.DISABLE enforcement
[
https://issues.apache.org/jira/browse/HBASE-20662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646996#comment-16646996
]
Nihal Jain commented on HBASE-20662:
bq. I'd suggest using the SLF4J APIs that let you do bracket substitution, e.g.
{{LOG.error("Failed .. {}", Bytes.toString(result.getRow()), e)}}
I think it's better to skip this here as we will any ways need to form the same
message in the exception.
bq. Can you leave a comment in here, as well as
DisableTableViolationPolicyEnforcement, to explain why this VPE is different
than all of the rest, please?
Added a comment in DisableTableViolationPolicyEnforcement doc. Does this look
fine?
{noformat}
A SpaceViolationPolicyEnforcement which disables the table. The enforcement
counterpart to SpaceViolationPolicy.DISABLE. This violation policy is different
from others as it doesn't take action (i.e. enable/disable table) local to the
RegionServer, like the other ViolationPolicies do. In case of violation, the
appropriate action is initiated by the master.
{noformat}
bq. IMO, the trace logging here is excessive.
Right, we can skip this. Removed the trace logging.
bq. What kind of exception do you see here? Is this something else that we
should be fixing that you will get a clear exception?
The following exception may randomly pop-up due to race between table disable
and put in the tests. Hence, added
{{msg.contains(policyToViolate.name())}} check to avoid such random failures of
UT.
{noformat}
java.lang.AssertionError: Exceptions
was:org.apache.hadoop.hbase.quotas.SpaceLimitingException: DISABLE This table
is disabled due to violating a space quota.
at
org.apache.hadoop.hbase.quotas.policies.DisableTableViolationPolicyEnforcement.check(DisableTableViolationPolicyEnforcement.java:49)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2908)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42000)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.hbase.quotas.TestSpaceQuotas.verifyViolation(TestSpaceQuotas.java:626)
at
org.apache.hadoop.hbase.quotas.TestSpaceQuotas.writeUntilViolationAndVerifyViolation(TestSpaceQuotas.java:598)
at
org.apache.hadoop.hbase.quotas.TestSpaceQuotas.setQuotaNextDisableThenIncreaseFinallyEnable(TestSpaceQuotas.java:515)
at
org.apache.hadoop.hbase.quotas.TestSpaceQuotas.testSetQuotaAndThenDisableIncrEnableWithDisable(TestSpaceQuotas.java:415)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at
[jira] [Commented] (HBASE-21103) nightly test cache of yetus install needs to be more thorough in verification
[
https://issues.apache.org/jira/browse/HBASE-21103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646993#comment-16646993
]
Hudson commented on HBASE-21103:
Results for branch branch-2.0
[build #936 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/936/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/936//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/936//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/936//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> nightly test cache of yetus install needs to be more thorough in verification
> -
>
> Key: HBASE-21103
> URL: https://issues.apache.org/jira/browse/HBASE-21103
> Project: HBase
> Issue Type: Improvement
> Components: test
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Labels: beginner
> Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 2.1.1, 2.0.3, 1.4.9
>
> Attachments: HBASE-21103-branch-1.v0.patch, HBASE-21103.0.patch
>
>
> branch-1.2 nightly failed because it couldn't find yetus:
> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/443/
> walking through steps, we checked and thought we had an existing install:
> https://builds.apache.org/blue/organizations/jenkins/HBase%20Nightly/detail/branch-1.2/443/pipeline/12
> {code}
> [HBase_Nightly_branch-1.2-AEAO7LJNYKPIH3O4GB2IR4TEIHRS6EXOJCJPGWYXWO6BFPUEZT3Q]
> Running shell script
> Ensure we have a copy of Apache Yetus.
> Checking for Yetus 0.6.0 in
> '/home/jenkins/jenkins-slave/workspace/HBase_Nightly_branch-1.2-AEAO7LJNYKPIH3O4GB2IR4TEIHRS6EXOJCJPGWYXWO6BFPUEZT3Q/yetus-0.6.0'
> Reusing cached download of Apache Yetus version 0.6.0.
> {code}
> So we stashed and then tried to use it.
> Examining the workspace present before the stash shows the directory we look
> for was present, but the contents were garbage:
> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/443/execution/node/3/ws/
> {code}
> $ unzip -l yetus-0.6.0.zip
> Archive: yetus-0.6.0.zip
> Length DateTimeName
> - -- -
> 0 00-00-1980 04:08 yetus-0.6.0/lib/precommit/qbt.sh
> - ---
> 0 1 file
> {code}
> we should probably check for an executable that will successfully give us a
> version or something like that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21286) Parallelize computeHDFSBlocksDistribution when getting splits of a HBaseSnapshot
[ https://issues.apache.org/jira/browse/HBASE-21286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646994#comment-16646994 ] stack commented on HBASE-21286: --- Related to https://issues.apache.org/jira/browse/HBASE-20786?focusedCommentId=16603944=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16603944 ? Could we parallelize for all cases, not just for snapshots? Thanks. > Parallelize computeHDFSBlocksDistribution when getting splits of a > HBaseSnapshot > > > Key: HBASE-21286 > URL: https://issues.apache.org/jira/browse/HBASE-21286 > Project: HBase > Issue Type: Improvement > Components: snapshots >Affects Versions: 1.4.0 >Reporter: Lavinia-Stefania Sirbu >Priority: Minor > Attachments: HBASE-21286.branch-1.4.001.patch, > HBASE-21286.branch-1.4.002.patch > > > Even if this step is called computeHDFSBlocksDistribution, this is executed > no matter the file system of the snapshot. For example, we have observed an > important slowness when we have a snapshot in s3 (~26k regions, 5column > families, 2 files per column family) the getsplits time is ~40min due to the > calls in s3 for listing the files to get the best locations. > Parallelizing this operation can reduce the overall setup time. The thread > pool should be configurable and a good choice could be > "hbase.snapshot.thread.pool.max" that is also used in RestoreSnapshotHelper. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20662) Increasing space quota on a violated table does not remove SpaceViolationPolicy.DISABLE enforcement
[
https://issues.apache.org/jira/browse/HBASE-20662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain updated HBASE-20662:
---
Attachment: HBASE-20662.master.004.patch
> Increasing space quota on a violated table does not remove
> SpaceViolationPolicy.DISABLE enforcement
> ---
>
> Key: HBASE-20662
> URL: https://issues.apache.org/jira/browse/HBASE-20662
> Project: HBase
> Issue Type: Bug
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20662.master.001.patch,
> HBASE-20662.master.002.patch, HBASE-20662.master.003.patch,
> HBASE-20662.master.004.patch
>
>
> *Steps to reproduce*
> * Create a table and set quota with {{SpaceViolationPolicy.DISABLE}} having
> limit say 2MB
> * Now put rows until space quota is violated and table gets disabled
> * Next, increase space quota with limit say 4MB on the table
> * Now try putting a row into the table
> {code:java}
> private void testSetQuotaThenViolateAndFinallyIncreaseQuota() throws
> Exception {
> SpaceViolationPolicy policy = SpaceViolationPolicy.DISABLE;
> Put put = new Put(Bytes.toBytes("to_reject"));
> put.addColumn(Bytes.toBytes(SpaceQuotaHelperForTests.F1),
> Bytes.toBytes("to"),
> Bytes.toBytes("reject"));
> // Do puts until we violate space policy
> final TableName tn = writeUntilViolationAndVerifyViolation(policy, put);
> // Now, increase limit
> setQuotaLimit(tn, policy, 4L);
> // Put some row now: should not violate as quota limit increased
> verifyNoViolation(policy, tn, put);
> }
> {code}
> *Expected*
> We should be able to put data as long as newly set quota limit is not reached
> *Actual*
> We fail to put any new row even after increasing limit
> *Root cause*
> Increasing quota on a violated table triggers the table to be enabled, but
> since the table is already in violation, the system does not allow it to be
> enabled (may be thinking that a user is trying to enable it)
> *Relevant exception trace*
> {noformat}
> 2018-05-31 00:34:27,563 INFO [regionserver/root1-ThinkPad-T440p:0.Chore.1]
> client.HBaseAdmin$14(844): Started enable of
> testSetQuotaAndThenIncreaseQuotaWithDisable0
> 2018-05-31 00:34:27,571 DEBUG
> [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=42525]
> ipc.CallRunner(142): callId: 11 service: MasterService methodName:
> EnableTable size: 104 connection: 127.0.0.1:38030 deadline: 1527707127568,
> exception=org.apache.hadoop.hbase.security.AccessDeniedException: Enabling
> the table 'testSetQuotaAndThenIncreaseQuotaWithDisable0' is disallowed due to
> a violated space quota.
> 2018-05-31 00:34:27,571 ERROR [regionserver/root1-ThinkPad-T440p:0.Chore.1]
> quotas.RegionServerSpaceQuotaManager(210): Failed to disable space violation
> policy for testSetQuotaAndThenIncreaseQuotaWithDisable0. This table will
> remain in violation.
> org.apache.hadoop.hbase.security.AccessDeniedException:
> org.apache.hadoop.hbase.security.AccessDeniedException: Enabling the table
> 'testSetQuotaAndThenIncreaseQuotaWithDisable0' is disallowed due to a
> violated space quota.
> at org.apache.hadoop.hbase.master.HMaster$6.run(HMaster.java:2275)
> at
> org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:131)
> at org.apache.hadoop.hbase.master.HMaster.enableTable(HMaster.java:2258)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.enableTable(MasterRpcServices.java:725)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:100)
> at
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:90)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:360)
> at
>
[jira] [Commented] (HBASE-21287) JVMClusterUtil Master initialization wait time not configurable
[
https://issues.apache.org/jira/browse/HBASE-21287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646988#comment-16646988
]
Hudson commented on HBASE-21287:
Results for branch branch-2.0
[build #935 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/935/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/935//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/935//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/935//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> JVMClusterUtil Master initialization wait time not configurable
> ---
>
> Key: HBASE-21287
> URL: https://issues.apache.org/jira/browse/HBASE-21287
> Project: HBase
> Issue Type: Task
> Components: test
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21287.patch
>
>
> We can configure how long the local cluster threads will wait for master to
> come up and become active, but not how long we allow initialization to take.
> Being able to tune this would improve my test loop on some experiment I am
> running.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21281) Update bouncycastle dependency.
[
https://issues.apache.org/jira/browse/HBASE-21281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646989#comment-16646989
]
Hudson commented on HBASE-21281:
Results for branch branch-2.0
[build #935 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/935/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/935//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/935//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/935//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Update bouncycastle dependency.
> ---
>
> Key: HBASE-21281
> URL: https://issues.apache.org/jira/browse/HBASE-21281
> Project: HBase
> Issue Type: Task
> Components: dependencies, test
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21281.001.branch-2.0.patch
>
>
> Looks like we still depend on bcprov-jdk16 for some x509 certificate
> generation in our tests. Bouncycastle has moved beyond this in 1.47, changing
> the artifact names.
> [http://www.bouncycastle.org/wiki/display/JA1/Porting+from+earlier+BC+releases+to+1.47+and+later]
> There are some API changes too, but it looks like we don't use any of these.
> It seems like we also have vestiges in the POMs from when we were depending
> on a specific BC version that came in from Hadoop. We now have a
> KeyStoreTestUtil class in HBase, which makes me think we can also clean up
> some dependencies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21295) Update compatibility matrices
[ https://issues.apache.org/jira/browse/HBASE-21295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646981#comment-16646981 ] Sean Busbey commented on HBASE-21295: - let's keep the symbols thing distinct from this change. I have not yet figured out the font thing. > Update compatibility matrices > - > > Key: HBASE-21295 > URL: https://issues.apache.org/jira/browse/HBASE-21295 > Project: HBase > Issue Type: Improvement > Components: documentation >Affects Versions: 3.0.0 >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Minor > Attachments: HBASE-21295-v1.patch, HBASE-21295-v2.patch > > > JDK and Hadoop compatibility matrices does not cover all of the current > releases and available JDKs. These should be added to both tables. > [https://hbase.apache.org/book.html#hadoop] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21073) "Maintenance mode" master
[ https://issues.apache.org/jira/browse/HBASE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-21073: -- Attachment: HBASE-21073.master.002.patch > "Maintenance mode" master > - > > Key: HBASE-21073 > URL: https://issues.apache.org/jira/browse/HBASE-21073 > Project: HBase > Issue Type: Sub-task > Components: amv2, hbck2, master >Reporter: stack >Assignee: Mike Drob >Priority: Major > Attachments: HBASE-21073.master.001.patch, > HBASE-21073.master.002.patch > > > Make it so we can bring up a Master in "maintenance mode". This is parse of > master wal procs but not taking on regionservers. It would be in a state > where "repair" Procedures could run; e.g. a Procedure that could recover meta > by looking for meta WALs, splitting them, dropping recovered.edits, and even > making it so meta is readable. See parent issue for why needed (disaster > recovery). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20140) HRegion FileSystem should be instantiated from hbase rootDir not default
[
https://issues.apache.org/jira/browse/HBASE-20140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646960#comment-16646960
]
Hadoop QA commented on HBASE-20140:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
42s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
29s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
12m 55s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}263m 22s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
0s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}317m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.quotas.TestSpaceQuotas |
| | hadoop.hbase.regionserver.TestRegionReplicaFailover |
| | hadoop.hbase.replication.TestMasterReplication |
| | hadoop.hbase.tool.TestSecureLoadIncrementalHFiles |
| | hadoop.hbase.replication.TestReplicationSmallTestsSync |
| | hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20140 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12943442/HBASE-20140.v01.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 47d1e7fe2346 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 72552301ab |
| maven | version: Apache Maven 3.5.4
[jira] [Commented] (HBASE-21073) "Maintenance mode" master
[ https://issues.apache.org/jira/browse/HBASE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646955#comment-16646955 ] Mike Drob commented on HBASE-21073: --- bq. I have no idea what happens if we try to scan one of the user space tables. We read the table region locations out of meta, try to connect to the previously hosting region server, and then fail with connection refused. The ergonomics of this are not great, but I think maybe we can leave it for follow on work. bq. RSRpcServices can try to reject requests from non-replication peers... bq. I have no idea what happens if we try to come up in a repair mode when replication is enabled. Replication doesn't apply to system tables, so that's why this never gets set up, I think. It's ok to ignore that I think. Another open question though - should we try to fold this in to the maintenance mode features of HBASE-16008, where it is tracked via ZK? > "Maintenance mode" master > - > > Key: HBASE-21073 > URL: https://issues.apache.org/jira/browse/HBASE-21073 > Project: HBase > Issue Type: Sub-task > Components: amv2, hbck2, master >Reporter: stack >Assignee: Mike Drob >Priority: Major > Attachments: HBASE-21073.master.001.patch > > > Make it so we can bring up a Master in "maintenance mode". This is parse of > master wal procs but not taking on regionservers. It would be in a state > where "repair" Procedures could run; e.g. a Procedure that could recover meta > by looking for meta WALs, splitting them, dropping recovered.edits, and even > making it so meta is readable. See parent issue for why needed (disaster > recovery). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21103) nightly test cache of yetus install needs to be more thorough in verification
[
https://issues.apache.org/jira/browse/HBASE-21103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646947#comment-16646947
]
Hudson commented on HBASE-21103:
Results for branch branch-1.4
[build #502 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/502/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/502//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/502//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/502//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> nightly test cache of yetus install needs to be more thorough in verification
> -
>
> Key: HBASE-21103
> URL: https://issues.apache.org/jira/browse/HBASE-21103
> Project: HBase
> Issue Type: Improvement
> Components: test
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Labels: beginner
> Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 2.1.1, 2.0.3, 1.4.9
>
> Attachments: HBASE-21103-branch-1.v0.patch, HBASE-21103.0.patch
>
>
> branch-1.2 nightly failed because it couldn't find yetus:
> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/443/
> walking through steps, we checked and thought we had an existing install:
> https://builds.apache.org/blue/organizations/jenkins/HBase%20Nightly/detail/branch-1.2/443/pipeline/12
> {code}
> [HBase_Nightly_branch-1.2-AEAO7LJNYKPIH3O4GB2IR4TEIHRS6EXOJCJPGWYXWO6BFPUEZT3Q]
> Running shell script
> Ensure we have a copy of Apache Yetus.
> Checking for Yetus 0.6.0 in
> '/home/jenkins/jenkins-slave/workspace/HBase_Nightly_branch-1.2-AEAO7LJNYKPIH3O4GB2IR4TEIHRS6EXOJCJPGWYXWO6BFPUEZT3Q/yetus-0.6.0'
> Reusing cached download of Apache Yetus version 0.6.0.
> {code}
> So we stashed and then tried to use it.
> Examining the workspace present before the stash shows the directory we look
> for was present, but the contents were garbage:
> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/443/execution/node/3/ws/
> {code}
> $ unzip -l yetus-0.6.0.zip
> Archive: yetus-0.6.0.zip
> Length DateTimeName
> - -- -
> 0 00-00-1980 04:08 yetus-0.6.0/lib/precommit/qbt.sh
> - ---
> 0 1 file
> {code}
> we should probably check for an executable that will successfully give us a
> version or something like that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21103) nightly test cache of yetus install needs to be more thorough in verification
[
https://issues.apache.org/jira/browse/HBASE-21103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646946#comment-16646946
]
Hudson commented on HBASE-21103:
Results for branch branch-1.3
[build #498 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/498/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/498//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/498//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/498//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> nightly test cache of yetus install needs to be more thorough in verification
> -
>
> Key: HBASE-21103
> URL: https://issues.apache.org/jira/browse/HBASE-21103
> Project: HBase
> Issue Type: Improvement
> Components: test
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Labels: beginner
> Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 2.1.1, 2.0.3, 1.4.9
>
> Attachments: HBASE-21103-branch-1.v0.patch, HBASE-21103.0.patch
>
>
> branch-1.2 nightly failed because it couldn't find yetus:
> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/443/
> walking through steps, we checked and thought we had an existing install:
> https://builds.apache.org/blue/organizations/jenkins/HBase%20Nightly/detail/branch-1.2/443/pipeline/12
> {code}
> [HBase_Nightly_branch-1.2-AEAO7LJNYKPIH3O4GB2IR4TEIHRS6EXOJCJPGWYXWO6BFPUEZT3Q]
> Running shell script
> Ensure we have a copy of Apache Yetus.
> Checking for Yetus 0.6.0 in
> '/home/jenkins/jenkins-slave/workspace/HBase_Nightly_branch-1.2-AEAO7LJNYKPIH3O4GB2IR4TEIHRS6EXOJCJPGWYXWO6BFPUEZT3Q/yetus-0.6.0'
> Reusing cached download of Apache Yetus version 0.6.0.
> {code}
> So we stashed and then tried to use it.
> Examining the workspace present before the stash shows the directory we look
> for was present, but the contents were garbage:
> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/443/execution/node/3/ws/
> {code}
> $ unzip -l yetus-0.6.0.zip
> Archive: yetus-0.6.0.zip
> Length DateTimeName
> - -- -
> 0 00-00-1980 04:08 yetus-0.6.0/lib/precommit/qbt.sh
> - ---
> 0 1 file
> {code}
> we should probably check for an executable that will successfully give us a
> version or something like that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21247) Custom WAL Provider cannot be specified by configuration whose value is outside the enums in Providers
[ https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21247: --- Description: Currently all the WAL Providers acceptable to hbase are specified in Providers enum of WALFactory. This restricts the ability for additional WAL Providers to be supplied - by class name. This issue fixes the bug by allowing the specification of new WAL Provider class name using the config "hbase.wal.provider". was: Currently all the WAL Providers acceptable to hbase are specified in Providers enum of WALFactory. This restricts the ability for additional WAL Providers to be supplied - by class name. This issue introduces additional config which allows the specification of new WAL Provider through class name. > Custom WAL Provider cannot be specified by configuration whose value is > outside the enums in Providers > -- > > Key: HBASE-21247 > URL: https://issues.apache.org/jira/browse/HBASE-21247 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: 3.0.0 > > Attachments: 21247.v1.txt, 21247.v2.txt, 21247.v3.txt, 21247.v4.tst, > 21247.v4.txt > > > Currently all the WAL Providers acceptable to hbase are specified in > Providers enum of WALFactory. > This restricts the ability for additional WAL Providers to be supplied - by > class name. > This issue fixes the bug by allowing the specification of new WAL Provider > class name using the config "hbase.wal.provider". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21246) Introduce WALIdentity interface
[ https://issues.apache.org/jira/browse/HBASE-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646943#comment-16646943 ] stack commented on HBASE-21246: --- bq. You have a suggestion? I was commenting on the fact that we have a String (one 'key') to lookup another 'key' (WALIdentity), not on the class naming; i.e. a key to a key to an artifact seems like a tier too many. I can create a WALIdentity and it has no associated WAL instance? So we'd build handling for case where we are passed a WALIdentity with no associated WAL? So lots of NoSuchWALIDExceptions on the API? > Introduce WALIdentity interface > --- > > Key: HBASE-21246 > URL: https://issues.apache.org/jira/browse/HBASE-21246 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: HBASE-20952 > > Attachments: 21246.003.patch, 21246.HBASE-20952.001.patch, > 21246.HBASE-20952.002.patch, 21246.HBASE-20952.004.patch, > 21246.HBASE-20952.005.patch > > > We are introducing WALIdentity interface so that the WAL representation can > be decoupled from distributed filesystem. > The interface provides getName method whose return value can represent > filename in distributed filesystem environment or, the name of the stream > when the WAL is backed by log stream. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21247) Custom WAL Provider cannot be specified by configuration whose value is outside the enums in Providers
[ https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21247: --- Summary: Custom WAL Provider cannot be specified by configuration whose value is outside the enums in Providers (was: Allow WAL Provider to be specified by configuration without explicit enum in Providers) > Custom WAL Provider cannot be specified by configuration whose value is > outside the enums in Providers > -- > > Key: HBASE-21247 > URL: https://issues.apache.org/jira/browse/HBASE-21247 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: 3.0.0 > > Attachments: 21247.v1.txt, 21247.v2.txt, 21247.v3.txt, 21247.v4.tst, > 21247.v4.txt > > > Currently all the WAL Providers acceptable to hbase are specified in > Providers enum of WALFactory. > This restricts the ability for additional WAL Providers to be supplied - by > class name. > This issue introduces additional config which allows the specification of new > WAL Provider through class name. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21247) Custom WAL Provider cannot be specified by configuration whose value is outside the enums in Providers
[ https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21247: --- Release Note: (was: Two config parameters, hbase.wal.provider.class and hbase.wal.meta_provider.class are introduced. hbase.wal.provider.class, when specified, configures the WAL provider class through its class name. If not specified, we fall back to the WAL provider enum specification. hbase.wal.meta_provider.class, when specified, configures the WAL provider class for hbase:meta through its class name. If not specified, we fall back to using the value for hbase.wal.provider.class . These new configs, when specified, override the enum WAL provider config.) Issue Type: Bug (was: Improvement) > Custom WAL Provider cannot be specified by configuration whose value is > outside the enums in Providers > -- > > Key: HBASE-21247 > URL: https://issues.apache.org/jira/browse/HBASE-21247 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: 3.0.0 > > Attachments: 21247.v1.txt, 21247.v2.txt, 21247.v3.txt, 21247.v4.tst, > 21247.v4.txt > > > Currently all the WAL Providers acceptable to hbase are specified in > Providers enum of WALFactory. > This restricts the ability for additional WAL Providers to be supplied - by > class name. > This issue introduces additional config which allows the specification of new > WAL Provider through class name. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21281) Update bouncycastle dependency.
[
https://issues.apache.org/jira/browse/HBASE-21281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646931#comment-16646931
]
Hudson commented on HBASE-21281:
Results for branch branch-2
[build #1374 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1374/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1374//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1374//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1374//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Update bouncycastle dependency.
> ---
>
> Key: HBASE-21281
> URL: https://issues.apache.org/jira/browse/HBASE-21281
> Project: HBase
> Issue Type: Task
> Components: dependencies, test
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21281.001.branch-2.0.patch
>
>
> Looks like we still depend on bcprov-jdk16 for some x509 certificate
> generation in our tests. Bouncycastle has moved beyond this in 1.47, changing
> the artifact names.
> [http://www.bouncycastle.org/wiki/display/JA1/Porting+from+earlier+BC+releases+to+1.47+and+later]
> There are some API changes too, but it looks like we don't use any of these.
> It seems like we also have vestiges in the POMs from when we were depending
> on a specific BC version that came in from Hadoop. We now have a
> KeyStoreTestUtil class in HBase, which makes me think we can also clean up
> some dependencies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21287) JVMClusterUtil Master initialization wait time not configurable
[
https://issues.apache.org/jira/browse/HBASE-21287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646930#comment-16646930
]
Hudson commented on HBASE-21287:
Results for branch branch-2
[build #1374 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1374/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1374//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1374//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1374//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> JVMClusterUtil Master initialization wait time not configurable
> ---
>
> Key: HBASE-21287
> URL: https://issues.apache.org/jira/browse/HBASE-21287
> Project: HBase
> Issue Type: Task
> Components: test
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21287.patch
>
>
> We can configure how long the local cluster threads will wait for master to
> come up and become active, but not how long we allow initialization to take.
> Being able to tune this would improve my test loop on some experiment I am
> running.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21103) nightly test cache of yetus install needs to be more thorough in verification
[
https://issues.apache.org/jira/browse/HBASE-21103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646907#comment-16646907
]
Hudson commented on HBASE-21103:
Results for branch branch-1
[build #503 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/503/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/503//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/503//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/503//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> nightly test cache of yetus install needs to be more thorough in verification
> -
>
> Key: HBASE-21103
> URL: https://issues.apache.org/jira/browse/HBASE-21103
> Project: HBase
> Issue Type: Improvement
> Components: test
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Labels: beginner
> Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 2.1.1, 2.0.3, 1.4.9
>
> Attachments: HBASE-21103-branch-1.v0.patch, HBASE-21103.0.patch
>
>
> branch-1.2 nightly failed because it couldn't find yetus:
> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/443/
> walking through steps, we checked and thought we had an existing install:
> https://builds.apache.org/blue/organizations/jenkins/HBase%20Nightly/detail/branch-1.2/443/pipeline/12
> {code}
> [HBase_Nightly_branch-1.2-AEAO7LJNYKPIH3O4GB2IR4TEIHRS6EXOJCJPGWYXWO6BFPUEZT3Q]
> Running shell script
> Ensure we have a copy of Apache Yetus.
> Checking for Yetus 0.6.0 in
> '/home/jenkins/jenkins-slave/workspace/HBase_Nightly_branch-1.2-AEAO7LJNYKPIH3O4GB2IR4TEIHRS6EXOJCJPGWYXWO6BFPUEZT3Q/yetus-0.6.0'
> Reusing cached download of Apache Yetus version 0.6.0.
> {code}
> So we stashed and then tried to use it.
> Examining the workspace present before the stash shows the directory we look
> for was present, but the contents were garbage:
> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/443/execution/node/3/ws/
> {code}
> $ unzip -l yetus-0.6.0.zip
> Archive: yetus-0.6.0.zip
> Length DateTimeName
> - -- -
> 0 00-00-1980 04:08 yetus-0.6.0/lib/precommit/qbt.sh
> - ---
> 0 1 file
> {code}
> we should probably check for an executable that will successfully give us a
> version or something like that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21291) Bypass doesn't work for state-machine procedures
[
https://issues.apache.org/jira/browse/HBASE-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646891#comment-16646891
]
stack commented on HBASE-21291:
---
Thanks [~tianjingyun]. Can you manufacture the state in a test? Thank you.
> Bypass doesn't work for state-machine procedures
>
>
> Key: HBASE-21291
> URL: https://issues.apache.org/jira/browse/HBASE-21291
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
>
> {code}
> if (!procedure.isFailed()) {
> if (subprocs != null) {
> if (subprocs.length == 1 && subprocs[0] == procedure) {
> // Procedure returned itself. Quick-shortcut for a state
> machine-like procedure;
> // i.e. we go around this loop again rather than go back out on
> the scheduler queue.
> subprocs = null;
> reExecute = true;
> LOG.trace("Short-circuit to next step on pid={}",
> procedure.getProcId());
> } else {
> // Yield the current procedure, and make the subprocedure runnable
> // subprocs may come back 'null'.
> subprocs = initializeChildren(procStack, procedure, subprocs);
> LOG.info("Initialized subprocedures=" +
> (subprocs == null? null:
> Stream.of(subprocs).map(e -> "{" + e.toString() + "}").
> collect(Collectors.toList()).toString()));
> }
> } else if (procedure.getState() == ProcedureState.WAITING_TIMEOUT) {
> LOG.debug("Added to timeoutExecutor {}", procedure);
> timeoutExecutor.add(procedure);
> } else if (!suspended) {
> // No subtask, so we are done
> procedure.setState(ProcedureState.SUCCESS);
> }
> }
> {code}
> Currently implementation of ProcedureExecutor will set the reExcecute to true
> for state machine like procedure. Then if this procedure is stuck at one
> certain state, it will loop forever.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.getLockEntry(proc.getProcId());
> try {
> executeProcedure(proc);
> } catch (AssertionError e) {
> LOG.info("ASSERT pid=" + proc.getProcId(), e);
> throw e;
> } finally {
> procExecutionLock.releaseLockEntry(lockEntry);
> {code}
> Since procedure will get the IdLock and release it after execution done,
> state machine procedure will never release IdLock until it is finished.
> Then bypassProcedure doesn't work because is will try to grab the IdLock at
> first.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.tryLockEntry(procedure.getProcId(), lockWait);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21103) nightly test cache of yetus install needs to be more thorough in verification
[
https://issues.apache.org/jira/browse/HBASE-21103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646875#comment-16646875
]
Hudson commented on HBASE-21103:
Results for branch branch-1.2
[build #504 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/504/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/504//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/504//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/504//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> nightly test cache of yetus install needs to be more thorough in verification
> -
>
> Key: HBASE-21103
> URL: https://issues.apache.org/jira/browse/HBASE-21103
> Project: HBase
> Issue Type: Improvement
> Components: test
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Labels: beginner
> Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 2.1.1, 2.0.3, 1.4.9
>
> Attachments: HBASE-21103-branch-1.v0.patch, HBASE-21103.0.patch
>
>
> branch-1.2 nightly failed because it couldn't find yetus:
> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/443/
> walking through steps, we checked and thought we had an existing install:
> https://builds.apache.org/blue/organizations/jenkins/HBase%20Nightly/detail/branch-1.2/443/pipeline/12
> {code}
> [HBase_Nightly_branch-1.2-AEAO7LJNYKPIH3O4GB2IR4TEIHRS6EXOJCJPGWYXWO6BFPUEZT3Q]
> Running shell script
> Ensure we have a copy of Apache Yetus.
> Checking for Yetus 0.6.0 in
> '/home/jenkins/jenkins-slave/workspace/HBase_Nightly_branch-1.2-AEAO7LJNYKPIH3O4GB2IR4TEIHRS6EXOJCJPGWYXWO6BFPUEZT3Q/yetus-0.6.0'
> Reusing cached download of Apache Yetus version 0.6.0.
> {code}
> So we stashed and then tried to use it.
> Examining the workspace present before the stash shows the directory we look
> for was present, but the contents were garbage:
> https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/443/execution/node/3/ws/
> {code}
> $ unzip -l yetus-0.6.0.zip
> Archive: yetus-0.6.0.zip
> Length DateTimeName
> - -- -
> 0 00-00-1980 04:08 yetus-0.6.0/lib/precommit/qbt.sh
> - ---
> 0 1 file
> {code}
> we should probably check for an executable that will successfully give us a
> version or something like that.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21243) Correct java-doc for the method RpcServer.getRemoteAddress()
[
https://issues.apache.org/jira/browse/HBASE-21243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain updated HBASE-21243:
---
Attachment: HBASE-21243.master.002.patch
> Correct java-doc for the method RpcServer.getRemoteAddress()
>
>
> Key: HBASE-21243
> URL: https://issues.apache.org/jira/browse/HBASE-21243
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0, 2.0.0
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Trivial
> Labels: beginner, beginners, documentaion
> Fix For: 3.0.0
>
> Attachments: HBASE-21243.master.001.patch,
> HBASE-21243.master.002.patch
>
>
> Correct the java-doc for the method {{RpcServer.getRemoteAddress()}}.
> Currently it look like as below:
> {code:java}
> /**
>* @return Address of remote client if a request is ongoing, else null
>*/
> public static Optional getRemoteAddress() {
> return getCurrentCall().map(RpcCall::getRemoteAddress);
> }
> {code}
> Contrary to the doc the method will never return null. Rather it may return
> an empty Optional.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21292) IdLock.getLockEntry() may hang if interrupted
[ https://issues.apache.org/jira/browse/HBASE-21292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646866#comment-16646866 ] stack commented on HBASE-21292: --- +1 Its for sure exotic. > IdLock.getLockEntry() may hang if interrupted > - > > Key: HBASE-21292 > URL: https://issues.apache.org/jira/browse/HBASE-21292 > Project: HBase > Issue Type: Bug >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 2.1.0, 2.0.2, 1.4.9 > > Attachments: HBASE-21292.branch-2.0.001.patch > > > This is a rare case found by my colleague which really happened on our > production env. > Thread may hang(or enter a infinite loop ) when try to call > IdLock.getLockEntry(). Here is the case: > 1. Thread1 owned the IdLock, while Thread2(the only one waiting) was waiting > for it. > 2. Thread1 called releaseLockEntry, it will set IdLock.locked = false, but > since Thread2 was waiting, it won't call map.remove(entry.id) > 3. While Thread1 was calling releaseLockEntry, Thread2 was interrupted. So no > one will remove this IdLock from the map. > 4. If another thread try to call getLockEntry on this IdLock， it will end up > in a infinite loop. Since existing = map.putIfAbsent(entry.id, entry)) != > null and existing.locked=false > It is hard to write a UT since it is a very rare race condition. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21286) Parallelize computeHDFSBlocksDistribution when getting splits of a HBaseSnapshot
[
https://issues.apache.org/jira/browse/HBASE-21286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646851#comment-16646851
]
Hadoop QA commented on HBASE-21286:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
49s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-1.4 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
27s{color} | {color:green} branch-1.4 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} branch-1.4 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} branch-1.4 passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s{color} | {color:green} branch-1.4 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
53s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} branch-1.4 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} branch-1.4 passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
21s{color} | {color:red} hbase-server: The patch generated 4 new + 8 unchanged
- 0 fixed = 12 total (was 8) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
42s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 43s{color} | {color:green} Patch does not cause any errors with Hadoop
2.4.1 2.5.2 2.6.5 2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}123m 28s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}160m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.security.visibility.TestVisibilityLabelsWithACL |
| | hadoop.hbase.client.TestReplicasClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:2cf636a |
| JIRA Issue | HBASE-21286 |
| JIRA Patch URL |
