[jira] [Commented] (HBASE-21275) Thrift Server (branch 1 fix) -> Disable TRACE HTTP method for thrift http server (branch 1 only)
[ https://issues.apache.org/jira/browse/HBASE-21275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648759#comment-16648759 ] stack commented on HBASE-21275: --- Where you want it [~wchevreuil]? It goes on branch-1.2. Andrew gives ok for branch-1.4 but the patch does not apply to branch-1. If you put up a branch-1 patch, I'll put it everywhere. Thanks. > Thrift Server (branch 1 fix) -> Disable TRACE HTTP method for thrift http > server (branch 1 only) > > > Key: HBASE-21275 > URL: https://issues.apache.org/jira/browse/HBASE-21275 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Fix For: 1.4.8, 1.2.7 > > Attachments: HBASE-21275-branch-1.2.001.patch, > HBASE-21275-branch-1.2.002.patch, HBASE-21275-branch-1.2.003.patch, > HBASE-21275-branch-1.2.003.patch > > > There's been a reasonable number of users running thrift http server on hbase > 1.x suffering with security audit tests pointing thrift server allows TRACE > requests. > After doing some search, I can see HBASE-20406 added restrictions for > TRACE/OPTIONS method when Thrift is running over http, but it relies on many > other commits applied to thrift http server. This patch was later reverted > from master. Then again later, HBASE-20004 had made TRACE/OPTIONS > configurable via "*hbase.thrift.http.allow.options.method*" property, with > both methods being disabled by default. This also seems to rely on many > changes applied to thrift http server, and a branch 1 compatible patch does > not seem feasible. > A solution for branch 1 is pretty simple though, am proposing a patch that > simply uses *WebAppContext*, instead of *Context*, as the context for the > *HttpServer* instance. *WebAppContext* will already restrict TRACE methods by > default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21259) [amv2] Revived deadservers; recreated serverstatenode
[
https://issues.apache.org/jira/browse/HBASE-21259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648756#comment-16648756
]
Hudson commented on HBASE-21259:
Results for branch branch-2.0
[build #943 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/943/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/943//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/943//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/943//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> [amv2] Revived deadservers; recreated serverstatenode
> -
>
> Key: HBASE-21259
> URL: https://issues.apache.org/jira/browse/HBASE-21259
> Project: HBase
> Issue Type: Bug
> Components: amv2
>Affects Versions: 2.1.0
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.1.1, 2.0.3
>
> Attachments: HBASE-21259.branch-2.1.001.patch,
> HBASE-21259.branch-2.1.002.patch, HBASE-21259.branch-2.1.003.patch,
> HBASE-21259.branch-2.1.004.patch, HBASE-21259.branch-2.1.005.patch,
> HBASE-21259.branch-2.1.006.patch
>
>
> On startup, I see servers being revived; i.e. their serverstatenode is
> getting marked online even though its just been processed by
> ServerCrashProcedure. It looks like this (in a patched server that reports on
> whenever a serverstatenode is created):
> {code}
> 2018-09-29 03:45:40,963 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=3982597,
> state=SUCCESS; ServerCrashProcedure
> server=vb1442.halxg.cloudera.com,22101,1536675314426, splitWal=true,
> meta=false in 1.0130sec
> ...
> 2018-09-29 03:45:43,733 INFO
> org.apache.hadoop.hbase.master.assignment.RegionStates: CREATING!
> vb1442.halxg.cloudera.com,22101,1536675314426
> java.lang.RuntimeException: WHERE AM I?
> at
> org.apache.hadoop.hbase.master.assignment.RegionStates.getOrCreateServer(RegionStates.java:1116)
> at
> org.apache.hadoop.hbase.master.assignment.RegionStates.addRegionToServer(RegionStates.java:1143)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.markRegionAsClosing(AssignmentManager.java:1464)
> at
> org.apache.hadoop.hbase.master.assignment.UnassignProcedure.updateTransition(UnassignProcedure.java:200)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:369)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:97)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:953)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1716)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1494)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:75)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2022)
> {code}
> See how we've just finished a SCP which will have removed the
> serverstatenode... but then we come across an unassign that references the
> server that was just processed. The unassign will attempt to update the
> serverstatenode and therein we create one if one not present. We shouldn't be
> creating one.
> I think I see this a lot because I am scheduling unassigns with hbck2. The
> servers crash and then come up with SCPs doing cleanup of old server and
> unassign procedures in the procedure executor queue to be processed still
> but could happen at any time on cluster should an unassign happen get
> scheduled near an SCP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21251) Refactor RegionMover
[
https://issues.apache.org/jira/browse/HBASE-21251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648757#comment-16648757
]
Hudson commented on HBASE-21251:
Results for branch branch-2.0
[build #943 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/943/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/943//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/943//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/943//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Refactor RegionMover
>
>
> Key: HBASE-21251
> URL: https://issues.apache.org/jira/browse/HBASE-21251
> Project: HBase
> Issue Type: Improvement
> Components: Operability
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21251.master.001.patch,
> HBASE-21251.master.002.patch, HBASE-21251.master.003.patch,
> HBASE-21251.master.004.patch
>
>
> 1. Move connection and admin to RegionMover's member variables. No need
> create connection many times.
> 2. use try-with-resource to reduce code
> 3. use ServerName instead of String
> 4. don't use Deprecated method
> 5. remove duplicate code
> ..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21291) Add a test for bypassing stuck state-machine procedures
[
https://issues.apache.org/jira/browse/HBASE-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648753#comment-16648753
]
stack commented on HBASE-21291:
---
Ok. Let me try testing it on cluster. Thanks.
> Add a test for bypassing stuck state-machine procedures
> ---
>
> Key: HBASE-21291
> URL: https://issues.apache.org/jira/browse/HBASE-21291
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Attachments: HBASE-21291.master.001.patch,
> HBASE-21291.master.002.patch, HBASE-21291.master.003.patch,
> HBASE-21291.master.004.patch, HBASE-21291.master.005.patch
>
>
> {code}
> if (!procedure.isFailed()) {
> if (subprocs != null) {
> if (subprocs.length == 1 && subprocs[0] == procedure) {
> // Procedure returned itself. Quick-shortcut for a state
> machine-like procedure;
> // i.e. we go around this loop again rather than go back out on
> the scheduler queue.
> subprocs = null;
> reExecute = true;
> LOG.trace("Short-circuit to next step on pid={}",
> procedure.getProcId());
> } else {
> // Yield the current procedure, and make the subprocedure runnable
> // subprocs may come back 'null'.
> subprocs = initializeChildren(procStack, procedure, subprocs);
> LOG.info("Initialized subprocedures=" +
> (subprocs == null? null:
> Stream.of(subprocs).map(e -> "{" + e.toString() + "}").
> collect(Collectors.toList()).toString()));
> }
> } else if (procedure.getState() == ProcedureState.WAITING_TIMEOUT) {
> LOG.debug("Added to timeoutExecutor {}", procedure);
> timeoutExecutor.add(procedure);
> } else if (!suspended) {
> // No subtask, so we are done
> procedure.setState(ProcedureState.SUCCESS);
> }
> }
> {code}
> Currently implementation of ProcedureExecutor will set the reExcecute to true
> for state machine like procedure. Then if this procedure is stuck at one
> certain state, it will loop forever.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.getLockEntry(proc.getProcId());
> try {
> executeProcedure(proc);
> } catch (AssertionError e) {
> LOG.info("ASSERT pid=" + proc.getProcId(), e);
> throw e;
> } finally {
> procExecutionLock.releaseLockEntry(lockEntry);
> {code}
> Since procedure will get the IdLock and release it after execution done,
> state machine procedure will never release IdLock until it is finished.
> Then bypassProcedure doesn't work because is will try to grab the IdLock at
> first.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.tryLockEntry(procedure.getProcId(), lockWait);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21291) Add a test for bypassing stuck state-machine procedures
[
https://issues.apache.org/jira/browse/HBASE-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingyun Tian updated HBASE-21291:
-
Attachment: HBASE-21291.master.005.patch
> Add a test for bypassing stuck state-machine procedures
> ---
>
> Key: HBASE-21291
> URL: https://issues.apache.org/jira/browse/HBASE-21291
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Attachments: HBASE-21291.master.001.patch,
> HBASE-21291.master.002.patch, HBASE-21291.master.003.patch,
> HBASE-21291.master.004.patch, HBASE-21291.master.005.patch
>
>
> {code}
> if (!procedure.isFailed()) {
> if (subprocs != null) {
> if (subprocs.length == 1 && subprocs[0] == procedure) {
> // Procedure returned itself. Quick-shortcut for a state
> machine-like procedure;
> // i.e. we go around this loop again rather than go back out on
> the scheduler queue.
> subprocs = null;
> reExecute = true;
> LOG.trace("Short-circuit to next step on pid={}",
> procedure.getProcId());
> } else {
> // Yield the current procedure, and make the subprocedure runnable
> // subprocs may come back 'null'.
> subprocs = initializeChildren(procStack, procedure, subprocs);
> LOG.info("Initialized subprocedures=" +
> (subprocs == null? null:
> Stream.of(subprocs).map(e -> "{" + e.toString() + "}").
> collect(Collectors.toList()).toString()));
> }
> } else if (procedure.getState() == ProcedureState.WAITING_TIMEOUT) {
> LOG.debug("Added to timeoutExecutor {}", procedure);
> timeoutExecutor.add(procedure);
> } else if (!suspended) {
> // No subtask, so we are done
> procedure.setState(ProcedureState.SUCCESS);
> }
> }
> {code}
> Currently implementation of ProcedureExecutor will set the reExcecute to true
> for state machine like procedure. Then if this procedure is stuck at one
> certain state, it will loop forever.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.getLockEntry(proc.getProcId());
> try {
> executeProcedure(proc);
> } catch (AssertionError e) {
> LOG.info("ASSERT pid=" + proc.getProcId(), e);
> throw e;
> } finally {
> procExecutionLock.releaseLockEntry(lockEntry);
> {code}
> Since procedure will get the IdLock and release it after execution done,
> state machine procedure will never release IdLock until it is finished.
> Then bypassProcedure doesn't work because is will try to grab the IdLock at
> first.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.tryLockEntry(procedure.getProcId(), lockWait);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21271) [amv2] Don't throw UnsupportedOperationException when rollback called on Assign/Unassign; spiral of death
[ https://issues.apache.org/jira/browse/HBASE-21271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21271: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: (was: 2.2.0) Status: Resolved (was: Patch Available) Pushed to branch-2.0 and branch-2.1. Doesn't make sense on branch-2+. I see it makes a difference in testing in that the failed rollback eventually finishes where previous it would just cycle forever. > [amv2] Don't throw UnsupportedOperationException when rollback called on > Assign/Unassign; spiral of death > - > > Key: HBASE-21271 > URL: https://issues.apache.org/jira/browse/HBASE-21271 > Project: HBase > Issue Type: Bug > Components: amv2 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.1.1, 2.0.3 > > Attachments: HBASE-21271.branch-2.1.001.patch, > HBASE-21271.branch-2.1.002.patch > > > I can't repro reliably but if an AssignProcedure or UnassignProcedure is a > subprocedure of an Enable/Disable and for whatever reason the parent decides > it needs to rollback -- can't get an entity lock -- it will ask the > subprocedures to rollback. UP and AP don't support rollback on all steps. For > steps where not supported, we have been throwing a > UnsupportedOperationException The Framework reschedules the rollback. And > so on filling logs and Procedure WALs. > Instead just note no rollback supported and intervention may be needed (until > we to to 2.2 when AP/UP go away). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21291) Add a test for bypassing stuck state-machine procedures
[
https://issues.apache.org/jira/browse/HBASE-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648751#comment-16648751
]
Jingyun Tian commented on HBASE-21291:
--
[~stack] Yes, I think pass one second or a certain number should be good.
Otherwise HBCK will get stuck, that will make user confusing.
> Add a test for bypassing stuck state-machine procedures
> ---
>
> Key: HBASE-21291
> URL: https://issues.apache.org/jira/browse/HBASE-21291
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Attachments: HBASE-21291.master.001.patch,
> HBASE-21291.master.002.patch, HBASE-21291.master.003.patch,
> HBASE-21291.master.004.patch
>
>
> {code}
> if (!procedure.isFailed()) {
> if (subprocs != null) {
> if (subprocs.length == 1 && subprocs[0] == procedure) {
> // Procedure returned itself. Quick-shortcut for a state
> machine-like procedure;
> // i.e. we go around this loop again rather than go back out on
> the scheduler queue.
> subprocs = null;
> reExecute = true;
> LOG.trace("Short-circuit to next step on pid={}",
> procedure.getProcId());
> } else {
> // Yield the current procedure, and make the subprocedure runnable
> // subprocs may come back 'null'.
> subprocs = initializeChildren(procStack, procedure, subprocs);
> LOG.info("Initialized subprocedures=" +
> (subprocs == null? null:
> Stream.of(subprocs).map(e -> "{" + e.toString() + "}").
> collect(Collectors.toList()).toString()));
> }
> } else if (procedure.getState() == ProcedureState.WAITING_TIMEOUT) {
> LOG.debug("Added to timeoutExecutor {}", procedure);
> timeoutExecutor.add(procedure);
> } else if (!suspended) {
> // No subtask, so we are done
> procedure.setState(ProcedureState.SUCCESS);
> }
> }
> {code}
> Currently implementation of ProcedureExecutor will set the reExcecute to true
> for state machine like procedure. Then if this procedure is stuck at one
> certain state, it will loop forever.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.getLockEntry(proc.getProcId());
> try {
> executeProcedure(proc);
> } catch (AssertionError e) {
> LOG.info("ASSERT pid=" + proc.getProcId(), e);
> throw e;
> } finally {
> procExecutionLock.releaseLockEntry(lockEntry);
> {code}
> Since procedure will get the IdLock and release it after execution done,
> state machine procedure will never release IdLock until it is finished.
> Then bypassProcedure doesn't work because is will try to grab the IdLock at
> first.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.tryLockEntry(procedure.getProcId(), lockWait);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21291) Add a test for bypassing stuck state-machine procedures
[
https://issues.apache.org/jira/browse/HBASE-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648750#comment-16648750
]
stack commented on HBASE-21291:
---
Sorry for the dumb questions, so problem is that we've been passing '0' so we
wait for ever?
Idea is that instead, when bypassing, we'd pass one second or something?
Thanks [~tianjingyun]
> Add a test for bypassing stuck state-machine procedures
> ---
>
> Key: HBASE-21291
> URL: https://issues.apache.org/jira/browse/HBASE-21291
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Attachments: HBASE-21291.master.001.patch,
> HBASE-21291.master.002.patch, HBASE-21291.master.003.patch,
> HBASE-21291.master.004.patch
>
>
> {code}
> if (!procedure.isFailed()) {
> if (subprocs != null) {
> if (subprocs.length == 1 && subprocs[0] == procedure) {
> // Procedure returned itself. Quick-shortcut for a state
> machine-like procedure;
> // i.e. we go around this loop again rather than go back out on
> the scheduler queue.
> subprocs = null;
> reExecute = true;
> LOG.trace("Short-circuit to next step on pid={}",
> procedure.getProcId());
> } else {
> // Yield the current procedure, and make the subprocedure runnable
> // subprocs may come back 'null'.
> subprocs = initializeChildren(procStack, procedure, subprocs);
> LOG.info("Initialized subprocedures=" +
> (subprocs == null? null:
> Stream.of(subprocs).map(e -> "{" + e.toString() + "}").
> collect(Collectors.toList()).toString()));
> }
> } else if (procedure.getState() == ProcedureState.WAITING_TIMEOUT) {
> LOG.debug("Added to timeoutExecutor {}", procedure);
> timeoutExecutor.add(procedure);
> } else if (!suspended) {
> // No subtask, so we are done
> procedure.setState(ProcedureState.SUCCESS);
> }
> }
> {code}
> Currently implementation of ProcedureExecutor will set the reExcecute to true
> for state machine like procedure. Then if this procedure is stuck at one
> certain state, it will loop forever.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.getLockEntry(proc.getProcId());
> try {
> executeProcedure(proc);
> } catch (AssertionError e) {
> LOG.info("ASSERT pid=" + proc.getProcId(), e);
> throw e;
> } finally {
> procExecutionLock.releaseLockEntry(lockEntry);
> {code}
> Since procedure will get the IdLock and release it after execution done,
> state machine procedure will never release IdLock until it is finished.
> Then bypassProcedure doesn't work because is will try to grab the IdLock at
> first.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.tryLockEntry(procedure.getProcId(), lockWait);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21291) Add a test for bypassing stuck state-machine procedures
[
https://issues.apache.org/jira/browse/HBASE-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648745#comment-16648745
]
Hadoop QA commented on HBASE-21291:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
37s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} hbase-procedure: The patch generated 13 new + 13
unchanged - 0 fixed = 26 total (was 13) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
30s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 59s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
12s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21291 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12943742/HBASE-21291.master.004.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 16a568ae067b 4.4.0-134-generic #160~14.04.1-Ubuntu SMP Fri Aug
17 11:07:07 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 7464e2ef9d |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/14682/artifact/patchprocess/diff-checkstyle-hbase-procedure.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14682/testReport/ |
| Max. process+thread count | 268 (vs. ulimit of 1) |
| modules | C: hbase-procedure U: hbase-procedure |
|
[jira] [Commented] (HBASE-21291) Add a test for bypassing stuck state-machine procedures
[
https://issues.apache.org/jira/browse/HBASE-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648739#comment-16648739
]
Jingyun Tian commented on HBASE-21291:
--
{code}
IdLock.Entry lockEntry =
procExecutionLock.tryLockEntry(procedure.getProcId(), lockWait);
if (lockEntry == null && !force) {
LOG.debug("Waited {} ms, but {} is still running, skipping bypass with
force={}",
lockWait, procedure, force);
return false;
} else if (lockEntry == null) {
LOG.debug("Waited {} ms, but {} is still running, begin bypass with
force={}",
lockWait, procedure, force);
}
{code}
[~stack] For a stuck procedure, set the force flag to true will skip grabbing
the lock. But lockWait must be > 0 otherwise tryLockEntry will wait for the
lock forever. Thus my simple fix is add a condition check that lockWait should
be positive.
> Add a test for bypassing stuck state-machine procedures
> ---
>
> Key: HBASE-21291
> URL: https://issues.apache.org/jira/browse/HBASE-21291
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Attachments: HBASE-21291.master.001.patch,
> HBASE-21291.master.002.patch, HBASE-21291.master.003.patch,
> HBASE-21291.master.004.patch
>
>
> {code}
> if (!procedure.isFailed()) {
> if (subprocs != null) {
> if (subprocs.length == 1 && subprocs[0] == procedure) {
> // Procedure returned itself. Quick-shortcut for a state
> machine-like procedure;
> // i.e. we go around this loop again rather than go back out on
> the scheduler queue.
> subprocs = null;
> reExecute = true;
> LOG.trace("Short-circuit to next step on pid={}",
> procedure.getProcId());
> } else {
> // Yield the current procedure, and make the subprocedure runnable
> // subprocs may come back 'null'.
> subprocs = initializeChildren(procStack, procedure, subprocs);
> LOG.info("Initialized subprocedures=" +
> (subprocs == null? null:
> Stream.of(subprocs).map(e -> "{" + e.toString() + "}").
> collect(Collectors.toList()).toString()));
> }
> } else if (procedure.getState() == ProcedureState.WAITING_TIMEOUT) {
> LOG.debug("Added to timeoutExecutor {}", procedure);
> timeoutExecutor.add(procedure);
> } else if (!suspended) {
> // No subtask, so we are done
> procedure.setState(ProcedureState.SUCCESS);
> }
> }
> {code}
> Currently implementation of ProcedureExecutor will set the reExcecute to true
> for state machine like procedure. Then if this procedure is stuck at one
> certain state, it will loop forever.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.getLockEntry(proc.getProcId());
> try {
> executeProcedure(proc);
> } catch (AssertionError e) {
> LOG.info("ASSERT pid=" + proc.getProcId(), e);
> throw e;
> } finally {
> procExecutionLock.releaseLockEntry(lockEntry);
> {code}
> Since procedure will get the IdLock and release it after execution done,
> state machine procedure will never release IdLock until it is finished.
> Then bypassProcedure doesn't work because is will try to grab the IdLock at
> first.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.tryLockEntry(procedure.getProcId(), lockWait);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21266) Not running balancer because processing dead regionservers, but empty dead rs list
[
https://issues.apache.org/jira/browse/HBASE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648738#comment-16648738
]
Hadoop QA commented on HBASE-21266:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
55s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
14s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
24s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
51s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} branch-2.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
18s{color} | {color:red} hbase-server: The patch generated 2 new + 2 unchanged
- 5 fixed = 4 total (was 7) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
8s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 9s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}111m
39s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}152m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:42ca976 |
| JIRA Issue | HBASE-21266 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12943733/HBASE-21266.branch-2.1.001.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 02728695d635 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-2.1 / 72af27b8c9 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/14680/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14680/testReport/ |
| Max. process+thread count | 4368 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
[jira] [Commented] (HBASE-21271) [amv2] Don't throw UnsupportedOperationException when rollback called on Assign/Unassign; spiral of death
[
https://issues.apache.org/jira/browse/HBASE-21271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648737#comment-16648737
]
Hadoop QA commented on HBASE-21271:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2.1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
34s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
40s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s{color} | {color:green} branch-2.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} branch-2.1 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
39s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 15s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}110m
29s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}151m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:42ca976 |
| JIRA Issue | HBASE-21271 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12943737/HBASE-21271.branch-2.1.002.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 52dc1e6e485d 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-2.1 / 72af27b8c9 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14679/testReport/ |
| Max. process+thread count | 4904 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
[jira] [Commented] (HBASE-21291) Add a test for bypassing stuck state-machine procedures
[
https://issues.apache.org/jira/browse/HBASE-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648735#comment-16648735
]
stack commented on HBASE-21291:
---
Is there a fix here [~tianjingyun]? If bypass is called what happens? Thanks.
> Add a test for bypassing stuck state-machine procedures
> ---
>
> Key: HBASE-21291
> URL: https://issues.apache.org/jira/browse/HBASE-21291
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Attachments: HBASE-21291.master.001.patch,
> HBASE-21291.master.002.patch, HBASE-21291.master.003.patch,
> HBASE-21291.master.004.patch
>
>
> {code}
> if (!procedure.isFailed()) {
> if (subprocs != null) {
> if (subprocs.length == 1 && subprocs[0] == procedure) {
> // Procedure returned itself. Quick-shortcut for a state
> machine-like procedure;
> // i.e. we go around this loop again rather than go back out on
> the scheduler queue.
> subprocs = null;
> reExecute = true;
> LOG.trace("Short-circuit to next step on pid={}",
> procedure.getProcId());
> } else {
> // Yield the current procedure, and make the subprocedure runnable
> // subprocs may come back 'null'.
> subprocs = initializeChildren(procStack, procedure, subprocs);
> LOG.info("Initialized subprocedures=" +
> (subprocs == null? null:
> Stream.of(subprocs).map(e -> "{" + e.toString() + "}").
> collect(Collectors.toList()).toString()));
> }
> } else if (procedure.getState() == ProcedureState.WAITING_TIMEOUT) {
> LOG.debug("Added to timeoutExecutor {}", procedure);
> timeoutExecutor.add(procedure);
> } else if (!suspended) {
> // No subtask, so we are done
> procedure.setState(ProcedureState.SUCCESS);
> }
> }
> {code}
> Currently implementation of ProcedureExecutor will set the reExcecute to true
> for state machine like procedure. Then if this procedure is stuck at one
> certain state, it will loop forever.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.getLockEntry(proc.getProcId());
> try {
> executeProcedure(proc);
> } catch (AssertionError e) {
> LOG.info("ASSERT pid=" + proc.getProcId(), e);
> throw e;
> } finally {
> procExecutionLock.releaseLockEntry(lockEntry);
> {code}
> Since procedure will get the IdLock and release it after execution done,
> state machine procedure will never release IdLock until it is finished.
> Then bypassProcedure doesn't work because is will try to grab the IdLock at
> first.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.tryLockEntry(procedure.getProcId(), lockWait);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21242) [amv2] Miscellaneous minor log and assign procedure create improvements
[
https://issues.apache.org/jira/browse/HBASE-21242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-21242:
--
Attachment: HBASE-21242.branch-2.002.patch
> [amv2] Miscellaneous minor log and assign procedure create improvements
> ---
>
> Key: HBASE-21242
> URL: https://issues.apache.org/jira/browse/HBASE-21242
> Project: HBase
> Issue Type: Bug
> Components: amv2, Operability
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21242.addendum.fix.testhregioninfo.patch,
> HBASE-21242.branch-2.0.001.patch, HBASE-21242.branch-2.0.001.patch,
> HBASE-21242.branch-2.0.002.patch, HBASE-21242.branch-2.001.patch,
> HBASE-21242.branch-2.001.patch, HBASE-21242.branch-2.002.patch,
> HBASE-21242.branch-2.002.patch, HBASE-21242.branch-2.1.001.patch,
> HBASE-21242.branch-2.1.001.patch, HBASE-21242.branch-2.1.001.patch,
> HBASE-21242.branch-2.1.002.patch
>
>
> Some minor fixups:
> {code}
> For RIT Duration, do better than print ms/seconds. Remove redundant UI
> column dedicated to duration when we log it in the status field too.
> Make bypass log at INFO level -- when DEBUG we can miss important
> fixup detail like why we failed.
> Make it so on complete of subprocedure, we note count of outstanding
> siblings so we have a clue how much further the parent has to go before
> it is done (Helpful when hundreds of servers doing SCP).
> Have the SCP run the AP preflight check before creating an AP; saves
> creation of hundreds of thousands of APs during fixup of this big cluster
> of mine.
> Don't log tablename three times when reporting remote call failed.
> If lock is held already, note who has it. Also log after we get lock
> or if we have to wait rather than log on entrance though we may
> later have to wait (or we may have just picked up the lock).
> {code}
> Posting patch in a sec but let me try it on cluster too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21291) Add a test for bypassing stuck state-machine procedures
[
https://issues.apache.org/jira/browse/HBASE-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648730#comment-16648730
]
Jingyun Tian commented on HBASE-21291:
--
[~stack] Just uploaded the fixed one. Sorry for the late.
> Add a test for bypassing stuck state-machine procedures
> ---
>
> Key: HBASE-21291
> URL: https://issues.apache.org/jira/browse/HBASE-21291
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Attachments: HBASE-21291.master.001.patch,
> HBASE-21291.master.002.patch, HBASE-21291.master.003.patch,
> HBASE-21291.master.004.patch
>
>
> {code}
> if (!procedure.isFailed()) {
> if (subprocs != null) {
> if (subprocs.length == 1 && subprocs[0] == procedure) {
> // Procedure returned itself. Quick-shortcut for a state
> machine-like procedure;
> // i.e. we go around this loop again rather than go back out on
> the scheduler queue.
> subprocs = null;
> reExecute = true;
> LOG.trace("Short-circuit to next step on pid={}",
> procedure.getProcId());
> } else {
> // Yield the current procedure, and make the subprocedure runnable
> // subprocs may come back 'null'.
> subprocs = initializeChildren(procStack, procedure, subprocs);
> LOG.info("Initialized subprocedures=" +
> (subprocs == null? null:
> Stream.of(subprocs).map(e -> "{" + e.toString() + "}").
> collect(Collectors.toList()).toString()));
> }
> } else if (procedure.getState() == ProcedureState.WAITING_TIMEOUT) {
> LOG.debug("Added to timeoutExecutor {}", procedure);
> timeoutExecutor.add(procedure);
> } else if (!suspended) {
> // No subtask, so we are done
> procedure.setState(ProcedureState.SUCCESS);
> }
> }
> {code}
> Currently implementation of ProcedureExecutor will set the reExcecute to true
> for state machine like procedure. Then if this procedure is stuck at one
> certain state, it will loop forever.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.getLockEntry(proc.getProcId());
> try {
> executeProcedure(proc);
> } catch (AssertionError e) {
> LOG.info("ASSERT pid=" + proc.getProcId(), e);
> throw e;
> } finally {
> procExecutionLock.releaseLockEntry(lockEntry);
> {code}
> Since procedure will get the IdLock and release it after execution done,
> state machine procedure will never release IdLock until it is finished.
> Then bypassProcedure doesn't work because is will try to grab the IdLock at
> first.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.tryLockEntry(procedure.getProcId(), lockWait);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21291) Add a test for bypassing stuck state-machine procedures
[
https://issues.apache.org/jira/browse/HBASE-21291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingyun Tian updated HBASE-21291:
-
Attachment: HBASE-21291.master.004.patch
> Add a test for bypassing stuck state-machine procedures
> ---
>
> Key: HBASE-21291
> URL: https://issues.apache.org/jira/browse/HBASE-21291
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Major
> Attachments: HBASE-21291.master.001.patch,
> HBASE-21291.master.002.patch, HBASE-21291.master.003.patch,
> HBASE-21291.master.004.patch
>
>
> {code}
> if (!procedure.isFailed()) {
> if (subprocs != null) {
> if (subprocs.length == 1 && subprocs[0] == procedure) {
> // Procedure returned itself. Quick-shortcut for a state
> machine-like procedure;
> // i.e. we go around this loop again rather than go back out on
> the scheduler queue.
> subprocs = null;
> reExecute = true;
> LOG.trace("Short-circuit to next step on pid={}",
> procedure.getProcId());
> } else {
> // Yield the current procedure, and make the subprocedure runnable
> // subprocs may come back 'null'.
> subprocs = initializeChildren(procStack, procedure, subprocs);
> LOG.info("Initialized subprocedures=" +
> (subprocs == null? null:
> Stream.of(subprocs).map(e -> "{" + e.toString() + "}").
> collect(Collectors.toList()).toString()));
> }
> } else if (procedure.getState() == ProcedureState.WAITING_TIMEOUT) {
> LOG.debug("Added to timeoutExecutor {}", procedure);
> timeoutExecutor.add(procedure);
> } else if (!suspended) {
> // No subtask, so we are done
> procedure.setState(ProcedureState.SUCCESS);
> }
> }
> {code}
> Currently implementation of ProcedureExecutor will set the reExcecute to true
> for state machine like procedure. Then if this procedure is stuck at one
> certain state, it will loop forever.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.getLockEntry(proc.getProcId());
> try {
> executeProcedure(proc);
> } catch (AssertionError e) {
> LOG.info("ASSERT pid=" + proc.getProcId(), e);
> throw e;
> } finally {
> procExecutionLock.releaseLockEntry(lockEntry);
> {code}
> Since procedure will get the IdLock and release it after execution done,
> state machine procedure will never release IdLock until it is finished.
> Then bypassProcedure doesn't work because is will try to grab the IdLock at
> first.
> {code}
> IdLock.Entry lockEntry =
> procExecutionLock.tryLockEntry(procedure.getProcId(), lockWait);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20952) Re-visit the WAL API
[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648715#comment-16648715
]
Hudson commented on HBASE-20952:
Results for branch HBASE-20952
[build #16 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/16/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/16//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/16//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/16//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Re-visit the WAL API
>
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21251) Refactor RegionMover
[ https://issues.apache.org/jira/browse/HBASE-21251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648712#comment-16648712 ] Guanghao Zhang commented on HBASE-21251: pushed to branch-2.0. > Refactor RegionMover > > > Key: HBASE-21251 > URL: https://issues.apache.org/jira/browse/HBASE-21251 > Project: HBase > Issue Type: Improvement > Components: Operability >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21251.master.001.patch, > HBASE-21251.master.002.patch, HBASE-21251.master.003.patch, > HBASE-21251.master.004.patch > > > 1. Move connection and admin to RegionMover's member variables. No need > create connection many times. > 2. use try-with-resource to reduce code > 3. use ServerName instead of String > 4. don't use Deprecated method > 5. remove duplicate code > .. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-21305) TestHRegionInfo failing because descriptive strings have different times
[ https://issues.apache.org/jira/browse/HBASE-21305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-21305. --- Resolution: Implemented Resolving. Fixed with an amendment on HBASE-21242. > TestHRegionInfo failing because descriptive strings have different times > > > Key: HBASE-21305 > URL: https://issues.apache.org/jira/browse/HBASE-21305 > Project: HBase > Issue Type: Task > Components: test >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > > We see TestHRegionInfo.testRegionDetailsForDisplay failing because the > strings now show timestamps as created at differing points in the past. Can > use EnvironmentEdge to fix this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21251) Refactor RegionMover
[ https://issues.apache.org/jira/browse/HBASE-21251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-21251: --- Resolution: Fixed Fix Version/s: 2.0.3 Status: Resolved (was: Patch Available) > Refactor RegionMover > > > Key: HBASE-21251 > URL: https://issues.apache.org/jira/browse/HBASE-21251 > Project: HBase > Issue Type: Improvement > Components: Operability >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21251.master.001.patch, > HBASE-21251.master.002.patch, HBASE-21251.master.003.patch, > HBASE-21251.master.004.patch > > > 1. Move connection and admin to RegionMover's member variables. No need > create connection many times. > 2. use try-with-resource to reduce code > 3. use ServerName instead of String > 4. don't use Deprecated method > 5. remove duplicate code > .. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21281) Update bouncycastle dependency.
[ https://issues.apache.org/jira/browse/HBASE-21281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21281: --- Attachment: 21281.addendum.patch > Update bouncycastle dependency. > --- > > Key: HBASE-21281 > URL: https://issues.apache.org/jira/browse/HBASE-21281 > Project: HBase > Issue Type: Task > Components: dependencies, test >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: 21281.addendum.patch, HBASE-21281.001.branch-2.0.patch > > > Looks like we still depend on bcprov-jdk16 for some x509 certificate > generation in our tests. Bouncycastle has moved beyond this in 1.47, changing > the artifact names. > [http://www.bouncycastle.org/wiki/display/JA1/Porting+from+earlier+BC+releases+to+1.47+and+later] > There are some API changes too, but it looks like we don't use any of these. > It seems like we also have vestiges in the POMs from when we were depending > on a specific BC version that came in from Hadoop. We now have a > KeyStoreTestUtil class in HBase, which makes me think we can also clean up > some dependencies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21281) Update bouncycastle dependency.
[ https://issues.apache.org/jira/browse/HBASE-21281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21281: --- Status: Patch Available (was: Reopened) I ran TestDelegationTokenWithEncryption with addendum against both hadoop2 and hadoop3. Both passed. > Update bouncycastle dependency. > --- > > Key: HBASE-21281 > URL: https://issues.apache.org/jira/browse/HBASE-21281 > Project: HBase > Issue Type: Task > Components: dependencies, test >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: 21281.addendum.patch, HBASE-21281.001.branch-2.0.patch > > > Looks like we still depend on bcprov-jdk16 for some x509 certificate > generation in our tests. Bouncycastle has moved beyond this in 1.47, changing > the artifact names. > [http://www.bouncycastle.org/wiki/display/JA1/Porting+from+earlier+BC+releases+to+1.47+and+later] > There are some API changes too, but it looks like we don't use any of these. > It seems like we also have vestiges in the POMs from when we were depending > on a specific BC version that came in from Hadoop. We now have a > KeyStoreTestUtil class in HBase, which makes me think we can also clean up > some dependencies. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21281) Update bouncycastle dependency.
[
https://issues.apache.org/jira/browse/HBASE-21281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648708#comment-16648708
]
Ted Yu commented on HBASE-21281:
The following dependency tree output shows that old bouncycastle dependency was
pulled in:
{code}
[INFO] +- org.apache.hadoop:hadoop-minikdc:jar:2.7.7:test
[INFO] | \-
org.apache.directory.server:apacheds-protocol-ldap:jar:2.0.0-M15:test
[INFO] | +- org.apache.directory.api:api-asn1-ber:jar:1.0.0-M20:test
[INFO] | +-
org.apache.directory.api:api-ldap-extras-codec-api:jar:1.0.0-M20:test
[INFO] | +-
org.apache.directory.api:api-ldap-extras-codec:jar:1.0.0-M20:test
[INFO] | +- org.apache.directory.api:api-ldap-extras-sp:jar:1.0.0-M20:test
[INFO] | \- bouncycastle:bcprov-jdk15:jar:140:test
{code}
which we should exclude.
> Update bouncycastle dependency.
> ---
>
> Key: HBASE-21281
> URL: https://issues.apache.org/jira/browse/HBASE-21281
> Project: HBase
> Issue Type: Task
> Components: dependencies, test
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21281.001.branch-2.0.patch
>
>
> Looks like we still depend on bcprov-jdk16 for some x509 certificate
> generation in our tests. Bouncycastle has moved beyond this in 1.47, changing
> the artifact names.
> [http://www.bouncycastle.org/wiki/display/JA1/Porting+from+earlier+BC+releases+to+1.47+and+later]
> There are some API changes too, but it looks like we don't use any of these.
> It seems like we also have vestiges in the POMs from when we were depending
> on a specific BC version that came in from Hadoop. We now have a
> KeyStoreTestUtil class in HBase, which makes me think we can also clean up
> some dependencies.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21114) website should have a copy of 2.1 release docs
[ https://issues.apache.org/jira/browse/HBASE-21114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648692#comment-16648692 ] Sean Busbey commented on HBASE-21114: - menu item is there now. > website should have a copy of 2.1 release docs > -- > > Key: HBASE-21114 > URL: https://issues.apache.org/jira/browse/HBASE-21114 > Project: HBase > Issue Type: Task > Components: community, documentation >Affects Versions: 2.1.0 >Reporter: Sean Busbey >Assignee: Mike Drob >Priority: Major > Fix For: 2.1.1 > > Attachments: HBASE-21114.patch > > > in the "Documentation and API" menu we have entries for 2.0 and 1.2. should > also add in 2.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21290) No need to instantiate BlockCache for master which not carry table
[ https://issues.apache.org/jira/browse/HBASE-21290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648675#comment-16648675 ] Guanghao Zhang commented on HBASE-21290: Yes, related. But we still have this problem in branch-2 and master... > No need to instantiate BlockCache for master which not carry table > -- > > Key: HBASE-21290 > URL: https://issues.apache.org/jira/browse/HBASE-21290 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-21290.master.001.patch > > > In our production clusters, we use different jvm config for > master/regionserver but use same hbase-site.xml for master/regionserver. And > master has a small heap/offheap config. So the regionserver's > hbase.bucketcache.size is not suitable for master. I thought we don't need to > instantiate BlockCache for master which not carry table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21271) [amv2] Don't throw UnsupportedOperationException when rollback called on Assign/Unassign; spiral of death
[ https://issues.apache.org/jira/browse/HBASE-21271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21271: -- Attachment: HBASE-21271.branch-2.1.002.patch > [amv2] Don't throw UnsupportedOperationException when rollback called on > Assign/Unassign; spiral of death > - > > Key: HBASE-21271 > URL: https://issues.apache.org/jira/browse/HBASE-21271 > Project: HBase > Issue Type: Bug > Components: amv2 >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.2.0, 2.1.1, 2.0.3 > > Attachments: HBASE-21271.branch-2.1.001.patch, > HBASE-21271.branch-2.1.002.patch > > > I can't repro reliably but if an AssignProcedure or UnassignProcedure is a > subprocedure of an Enable/Disable and for whatever reason the parent decides > it needs to rollback -- can't get an entity lock -- it will ask the > subprocedures to rollback. UP and AP don't support rollback on all steps. For > steps where not supported, we have been throwing a > UnsupportedOperationException The Framework reschedules the rollback. And > so on filling logs and Procedure WALs. > Instead just note no rollback supported and intervention may be needed (until > we to to 2.2 when AP/UP go away). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21259) [amv2] Revived deadservers; recreated serverstatenode
[
https://issues.apache.org/jira/browse/HBASE-21259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-21259:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: (was: 2.2.0)
Status: Resolved (was: Patch Available)
Pushed to branch-2.0 and branch-2.1. Filed subtask for forward-port.
> [amv2] Revived deadservers; recreated serverstatenode
> -
>
> Key: HBASE-21259
> URL: https://issues.apache.org/jira/browse/HBASE-21259
> Project: HBase
> Issue Type: Bug
> Components: amv2
>Affects Versions: 2.1.0
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.1.1, 2.0.3
>
> Attachments: HBASE-21259.branch-2.1.001.patch,
> HBASE-21259.branch-2.1.002.patch, HBASE-21259.branch-2.1.003.patch,
> HBASE-21259.branch-2.1.004.patch, HBASE-21259.branch-2.1.005.patch,
> HBASE-21259.branch-2.1.006.patch
>
>
> On startup, I see servers being revived; i.e. their serverstatenode is
> getting marked online even though its just been processed by
> ServerCrashProcedure. It looks like this (in a patched server that reports on
> whenever a serverstatenode is created):
> {code}
> 2018-09-29 03:45:40,963 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=3982597,
> state=SUCCESS; ServerCrashProcedure
> server=vb1442.halxg.cloudera.com,22101,1536675314426, splitWal=true,
> meta=false in 1.0130sec
> ...
> 2018-09-29 03:45:43,733 INFO
> org.apache.hadoop.hbase.master.assignment.RegionStates: CREATING!
> vb1442.halxg.cloudera.com,22101,1536675314426
> java.lang.RuntimeException: WHERE AM I?
> at
> org.apache.hadoop.hbase.master.assignment.RegionStates.getOrCreateServer(RegionStates.java:1116)
> at
> org.apache.hadoop.hbase.master.assignment.RegionStates.addRegionToServer(RegionStates.java:1143)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.markRegionAsClosing(AssignmentManager.java:1464)
> at
> org.apache.hadoop.hbase.master.assignment.UnassignProcedure.updateTransition(UnassignProcedure.java:200)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:369)
> at
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:97)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:953)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1716)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1494)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:75)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2022)
> {code}
> See how we've just finished a SCP which will have removed the
> serverstatenode... but then we come across an unassign that references the
> server that was just processed. The unassign will attempt to update the
> serverstatenode and therein we create one if one not present. We shouldn't be
> creating one.
> I think I see this a lot because I am scheduling unassigns with hbck2. The
> servers crash and then come up with SCPs doing cleanup of old server and
> unassign procedures in the procedure executor queue to be processed still
> but could happen at any time on cluster should an unassign happen get
> scheduled near an SCP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21308) Forward-port to branch-2 "HBASE-21259 [amv2] Revived deadservers; recreated serverstatenode"
stack created HBASE-21308: - Summary: Forward-port to branch-2 "HBASE-21259 [amv2] Revived deadservers; recreated serverstatenode" Key: HBASE-21308 URL: https://issues.apache.org/jira/browse/HBASE-21308 Project: HBase Issue Type: Sub-task Reporter: stack Fix For: 3.0.0, 2.2.0 TODO: Recast HBASE-21259 so it works for branch-2; stuff is different in branch-2. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21266) Not running balancer because processing dead regionservers, but empty dead rs list
[ https://issues.apache.org/jira/browse/HBASE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648642#comment-16648642 ] stack commented on HBASE-21266: --- 2.1.001 is forward-port of Andrew's patch; took a bit of jiggering. > Not running balancer because processing dead regionservers, but empty dead rs > list > -- > > Key: HBASE-21266 > URL: https://issues.apache.org/jira/browse/HBASE-21266 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0, 1.4.9 > > Attachments: HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266.branch-2.1.001.patch > > > Found during ITBLL testing. AM in master gets into a state where manual > attempts from the shell to run the balancer always return false and this is > printed in the master log: > 2018-10-03 19:17:14,892 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=21,queue=0,port=8100] master.HMaster: > Not running balancer because processing dead regionserver(s): > Note the empty list. > This errant state did not recover without intervention by way of master > restart, but the test environment was chaotic so needs investigation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21266) Not running balancer because processing dead regionservers, but empty dead rs list
[ https://issues.apache.org/jira/browse/HBASE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21266: -- Attachment: HBASE-21266.branch-2.1.001.patch > Not running balancer because processing dead regionservers, but empty dead rs > list > -- > > Key: HBASE-21266 > URL: https://issues.apache.org/jira/browse/HBASE-21266 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0, 1.4.9 > > Attachments: HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266.branch-2.1.001.patch > > > Found during ITBLL testing. AM in master gets into a state where manual > attempts from the shell to run the balancer always return false and this is > printed in the master log: > 2018-10-03 19:17:14,892 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=21,queue=0,port=8100] master.HMaster: > Not running balancer because processing dead regionserver(s): > Note the empty list. > This errant state did not recover without intervention by way of master > restart, but the test environment was chaotic so needs investigation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21247) Custom WAL Provider cannot be specified by configuration whose value is outside the enums in Providers
[
https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648630#comment-16648630
]
Hadoop QA commented on HBASE-21247:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
2s{color} | {color:blue} The patch file was not named according to hbase's
naming conventions. Please see
https://yetus.apache.org/documentation/0.8.0/precommit-patchnames for
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
32s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
21s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 21s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}205m
41s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}252m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21247 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12943708/21247.v8.txt |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux a85336aa9389 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / e736168567 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14677/testReport/ |
| Max. process+thread count | 4301 (vs. ulimit of 1) |
|
[jira] [Commented] (HBASE-21299) List counts of actual region states in master UI tables section
[
https://issues.apache.org/jira/browse/HBASE-21299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648629#comment-16648629
]
Hudson commented on HBASE-21299:
Results for branch branch-2.1
[build #457 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/457/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/457//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/457//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/457//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> List counts of actual region states in master UI tables section
> ---
>
> Key: HBASE-21299
> URL: https://issues.apache.org/jira/browse/HBASE-21299
> Project: HBase
> Issue Type: Improvement
> Components: UI
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-21299.branch-2.1.001.patch, Screen Shot 2018-10-11
> at 9.44.39 PM.png, Screen Shot 2018-10-11 at 9.44.56 PM.png
>
>
> It the tables panel, we list Open Regions, Offline Regions, Failed Regions,
> Split Region, and Other. This is not very useful. It is from a time before
> region states were edited down. Better to list OPEN, CLOSED, OPENING,
> CLOSING...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21289) Remove the log "'hbase.regionserver.maxlogs' was deprecated." in AbstractFSWAL
[
https://issues.apache.org/jira/browse/HBASE-21289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648628#comment-16648628
]
Hudson commented on HBASE-21289:
Results for branch branch-2.1
[build #457 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/457/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/457//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/457//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/457//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove the log "'hbase.regionserver.maxlogs' was deprecated." in AbstractFSWAL
> --
>
> Key: HBASE-21289
> URL: https://issues.apache.org/jira/browse/HBASE-21289
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21289.master.001.patch
>
>
> This log was added by HBASE-14951. And the description and release note never
> said this config was deprecated. I thought HBASE-14951 only changed the
> default value of maxlogs (Please correct me if I am wrong). And we still use
> this config in our hbase book. So the log "'hbase.regionserver.maxlogs' was
> deprecated." in AbstractFSWAL is confused. Let's remove it. FYI [~vrodionov]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21242) [amv2] Miscellaneous minor log and assign procedure create improvements
[
https://issues.apache.org/jira/browse/HBASE-21242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648625#comment-16648625
]
Hadoop QA commented on HBASE-21242:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
21s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
21s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 7s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}205m
42s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}249m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21242 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12943705/HBASE-21242.addendum.fix.testhregioninfo.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux b10283056ff7 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / e736168567 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/14676/testReport/ |
| Max. process+thread count | 4054 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/14676/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
>
[jira] [Updated] (HBASE-21307) [amv2] Deadlock when we move a Region from a not-online RegionServer
[
https://issues.apache.org/jira/browse/HBASE-21307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-21307:
--
Priority: Critical (was: Major)
> [amv2] Deadlock when we move a Region from a not-online RegionServer
>
>
> Key: HBASE-21307
> URL: https://issues.apache.org/jira/browse/HBASE-21307
> Project: HBase
> Issue Type: Bug
> Components: amv2
>Affects Versions: 2.1.1
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.1.1
>
>
> Perhaps this doesn't happen in branch-2, but its problem in branch-2.1.
> Highlevel, we go to move a region, its unassign subprocedure fails its
> dispatch because the server is not online so it queues a SCP and waits on it
> to break the RPC. The SCP can't run though because the MRP holds lock on the
> region.
> I can bypass the MRP but then the SCP fails because Region is 'owned' by the
> MRP. See below:
> {code}
> 2018-10-12 16:29:53,423 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Begin bypass
> pid=411982, ppid=411981, state=RUNNABLE:REGION_TRANSITION_DISPATCH,
> locked=true; UnassignProcedure
> table=IntegrationTestBigLinkedList_20180709093726,
> region=f5f9ff1e4b0f2d9555dabfcca71df568, override=true,
> server=va1002.halxg.cloudera.com,22101,1539368318649 with lockWait=0,
> override=true, recursive=true
> 2018-10-12 16:29:53,424 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Bypassing pid=411982,
> ppid=411981, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true;
> UnassignProcedure table=IntegrationTestBigLinkedList_20180709093726,
> region=f5f9ff1e4b0f2d9555dabfcca71df568, override=true,
> server=va1002.halxg.cloudera.com,22101,1539368318649
> 2018-10-12 16:29:53,712 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Bypassing pid=411981,
> state=WAITING:MOVE_REGION_ASSIGN, locked=true; MoveRegionProcedure
> hri=f5f9ff1e4b0f2d9555dabfcca71df568,
> source=va1002.halxg.cloudera.com,22101,1539368318649,
> destination=vd1021.halxg.cloudera.com,22101,1539368317897
> 2018-10-12 16:29:53,838 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Bypassing pid=411982,
> ppid=411981, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true,
> bypass=LOG-REDACTED UnassignProcedure
> table=IntegrationTestBigLinkedList_20180709093726,
> region=f5f9ff1e4b0f2d9555dabfcca71df568, override=true,
> server=va1002.halxg.cloudera.com,22101,1539368318649 and its ancestors
> successfully, adding to queue
> 2018-10-12 16:29:53,839 INFO org.apache.hadoop.hbase.procedure2.Procedure:
> pid=411982, ppid=411981, state=RUNNABLE:REGION_TRANSITION_DISPATCH,
> locked=true, bypass=LOG-REDACTED UnassignProcedure
> table=IntegrationTestBigLinkedList_20180709093726,
> region=f5f9ff1e4b0f2d9555dabfcca71df568, override=true,
> server=va1002.halxg.cloudera.com,22101,1539368318649 bypassed, returning null
> to finish it
> 2018-10-12 16:29:53,954 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished subprocedure
> pid=411982, resume processing parent pid=411981,
> state=RUNNABLE:MOVE_REGION_ASSIGN, locked=true, bypass=LOG-REDACTED
> MoveRegionProcedure hri=f5f9ff1e4b0f2d9555dabfcca71df568,
> source=va1002.halxg.cloudera.com,22101,1539368318649,
> destination=vd1021.halxg.cloudera.com,22101,1539368317897
> 2018-10-12 16:29:53,954 INFO org.apache.hadoop.hbase.procedure2.Procedure:
> pid=411981, state=RUNNABLE:MOVE_REGION_ASSIGN, locked=true,
> bypass=LOG-REDACTED MoveRegionProcedure hri=f5f9ff1e4b0f2d9555dabfcca71df568,
> source=va1002.halxg.cloudera.com,22101,1539368318649,
> destination=vd1021.halxg.cloudera.com,22101,1539368317897 bypassed, returning
> null to finish it
> 2018-10-12 16:29:53,956 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=411982,
> ppid=411981, state=SUCCESS, bypass=LOG-REDACTED UnassignProcedure
> table=IntegrationTestBigLinkedList_20180709093726,
> region=f5f9ff1e4b0f2d9555dabfcca71df568, override=true,
> server=va1002.halxg.cloudera.com,22101,1539368318649 in 3hrs, 49mins,
> 12.419sec, unfinishedSiblingCount=0
> 2018-10-12 16:29:54,058 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=411981,
> state=SUCCESS, bypass=LOG-REDACTED MoveRegionProcedure
> hri=f5f9ff1e4b0f2d9555dabfcca71df568,
> source=va1002.halxg.cloudera.com,22101,1539368318649,
> destination=vd1021.halxg.cloudera.com,22101,1539368317897 in 3hrs, 49mins,
> 12.878sec
> 2018-10-12 16:29:54,059 INFO
> org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler: xlock for
> pid=412210, ppid=411983, state=RUNNABLE:REGION_TRANSITION_QUEUE;
> AssignProcedure table=IntegrationTestBigLinkedList_20180709093726,
> region=f5f9ff1e4b0f2d9555dabfcca71df568
> 2018-10-12 16:29:54,105 WARN
[jira] [Commented] (HBASE-21307) [amv2] Deadlock when we move a Region from a not-online RegionServer
[
https://issues.apache.org/jira/browse/HBASE-21307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648618#comment-16648618
]
stack commented on HBASE-21307:
---
Two things:
* Bypass leaving the old Procedure reference in place as a fencing mechanism
is messing up follow-on Procedures. This was 'expected' but Procedures don't
always react nicely as in above where SCP does rollback and ends up in an
UnsupportedOperationException.
* A MRP calling a UP which fails and then schedules an SCP locks up.
> [amv2] Deadlock when we move a Region from a not-online RegionServer
>
>
> Key: HBASE-21307
> URL: https://issues.apache.org/jira/browse/HBASE-21307
> Project: HBase
> Issue Type: Bug
> Components: amv2
>Affects Versions: 2.1.1
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.1.1
>
>
> Perhaps this doesn't happen in branch-2, but its problem in branch-2.1.
> Highlevel, we go to move a region, its unassign subprocedure fails its
> dispatch because the server is not online so it queues a SCP and waits on it
> to break the RPC. The SCP can't run though because the MRP holds lock on the
> region.
> I can bypass the MRP but then the SCP fails because Region is 'owned' by the
> MRP. See below:
> {code}
> 2018-10-12 16:29:53,423 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Begin bypass
> pid=411982, ppid=411981, state=RUNNABLE:REGION_TRANSITION_DISPATCH,
> locked=true; UnassignProcedure
> table=IntegrationTestBigLinkedList_20180709093726,
> region=f5f9ff1e4b0f2d9555dabfcca71df568, override=true,
> server=va1002.halxg.cloudera.com,22101,1539368318649 with lockWait=0,
> override=true, recursive=true
> 2018-10-12 16:29:53,424 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Bypassing pid=411982,
> ppid=411981, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true;
> UnassignProcedure table=IntegrationTestBigLinkedList_20180709093726,
> region=f5f9ff1e4b0f2d9555dabfcca71df568, override=true,
> server=va1002.halxg.cloudera.com,22101,1539368318649
> 2018-10-12 16:29:53,712 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Bypassing pid=411981,
> state=WAITING:MOVE_REGION_ASSIGN, locked=true; MoveRegionProcedure
> hri=f5f9ff1e4b0f2d9555dabfcca71df568,
> source=va1002.halxg.cloudera.com,22101,1539368318649,
> destination=vd1021.halxg.cloudera.com,22101,1539368317897
> 2018-10-12 16:29:53,838 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Bypassing pid=411982,
> ppid=411981, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true,
> bypass=LOG-REDACTED UnassignProcedure
> table=IntegrationTestBigLinkedList_20180709093726,
> region=f5f9ff1e4b0f2d9555dabfcca71df568, override=true,
> server=va1002.halxg.cloudera.com,22101,1539368318649 and its ancestors
> successfully, adding to queue
> 2018-10-12 16:29:53,839 INFO org.apache.hadoop.hbase.procedure2.Procedure:
> pid=411982, ppid=411981, state=RUNNABLE:REGION_TRANSITION_DISPATCH,
> locked=true, bypass=LOG-REDACTED UnassignProcedure
> table=IntegrationTestBigLinkedList_20180709093726,
> region=f5f9ff1e4b0f2d9555dabfcca71df568, override=true,
> server=va1002.halxg.cloudera.com,22101,1539368318649 bypassed, returning null
> to finish it
> 2018-10-12 16:29:53,954 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished subprocedure
> pid=411982, resume processing parent pid=411981,
> state=RUNNABLE:MOVE_REGION_ASSIGN, locked=true, bypass=LOG-REDACTED
> MoveRegionProcedure hri=f5f9ff1e4b0f2d9555dabfcca71df568,
> source=va1002.halxg.cloudera.com,22101,1539368318649,
> destination=vd1021.halxg.cloudera.com,22101,1539368317897
> 2018-10-12 16:29:53,954 INFO org.apache.hadoop.hbase.procedure2.Procedure:
> pid=411981, state=RUNNABLE:MOVE_REGION_ASSIGN, locked=true,
> bypass=LOG-REDACTED MoveRegionProcedure hri=f5f9ff1e4b0f2d9555dabfcca71df568,
> source=va1002.halxg.cloudera.com,22101,1539368318649,
> destination=vd1021.halxg.cloudera.com,22101,1539368317897 bypassed, returning
> null to finish it
> 2018-10-12 16:29:53,956 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=411982,
> ppid=411981, state=SUCCESS, bypass=LOG-REDACTED UnassignProcedure
> table=IntegrationTestBigLinkedList_20180709093726,
> region=f5f9ff1e4b0f2d9555dabfcca71df568, override=true,
> server=va1002.halxg.cloudera.com,22101,1539368318649 in 3hrs, 49mins,
> 12.419sec, unfinishedSiblingCount=0
> 2018-10-12 16:29:54,058 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=411981,
> state=SUCCESS, bypass=LOG-REDACTED MoveRegionProcedure
> hri=f5f9ff1e4b0f2d9555dabfcca71df568,
> source=va1002.halxg.cloudera.com,22101,1539368318649,
> destination=vd1021.halxg.cloudera.com,22101,1539368317897 in
[jira] [Commented] (HBASE-21266) Not running balancer because processing dead regionservers, but empty dead rs list
[
https://issues.apache.org/jira/browse/HBASE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648600#comment-16648600
]
Hadoop QA commented on HBASE-21266:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
5s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
4s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
57s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
56s{color} | {color:green} hbase-server: The patch generated 0 new + 68
unchanged - 5 fixed = 68 total (was 73) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
48s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
2m 36s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}144m 32s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}180m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.TestMasterBalanceThrottling |
| | hadoop.hbase.regionserver.TestPerColumnFamilyFlush |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA Issue | HBASE-21266 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12943710/HBASE-21266-branch-1.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars
[jira] [Commented] (HBASE-21307) [amv2] Deadlock when we move a Region from a not-online RegionServer
[
https://issues.apache.org/jira/browse/HBASE-21307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648588#comment-16648588
]
stack commented on HBASE-21307:
---
Eventually the rollback fails as follows still complaining the region is owned
by another:
{code}
rocedureAbortedException: f5f9ff1e4b0f2d9555dabfcca71df568 owned by pid=411982,
CANNOT run 'this' (pid=412210).; ServerCrashProcedure
server=va1002.halxg.cloudera.com,22101,1539237389315, splitWal=true, meta=false
java.lang.UnsupportedOperationException: unhandled
state=SERVER_CRASH_HANDLE_RIT2
at
org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.rollbackState(ServerCrashProcedure.java:262)
at
org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.rollbackState(ServerCrashProcedure.java:59)
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.rollback(StateMachineProcedure.java:208)
at
org.apache.hadoop.hbase.procedure2.Procedure.doRollback(Procedure.java:970)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1618)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1580)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1451)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:75)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2022)
2018-10-12 16:33:09,150 DEBUG
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: pid=412210, ppid=411983,
state=ROLLEDBACK,
exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via
AssignProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException:
f5f9ff1e4b0f2d9555dabfcca71df568 owned by pid=411982, CANNOT run 'this'
(pid=412210).; AssignProcedure
table=IntegrationTestBigLinkedList_20180709093726,
region=f5f9ff1e4b0f2d9555dabfcca71df568 is already finished, skipping execution
2018-10-12 16:35:10,099 DEBUG
org.apache.hadoop.hbase.regionserver.ChunkCreator: data stats (chunk
size=2097152): current pool size=0, created chunk count=0, reused chunk
count=0, reuseRatio=0
{code}
This holds up the general assign until its unblocked.
> [amv2] Deadlock when we move a Region from a not-online RegionServer
>
>
> Key: HBASE-21307
> URL: https://issues.apache.org/jira/browse/HBASE-21307
> Project: HBase
> Issue Type: Bug
> Components: amv2
>Affects Versions: 2.1.1
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.1.1
>
>
> Perhaps this doesn't happen in branch-2, but its problem in branch-2.1.
> Highlevel, we go to move a region, its unassign subprocedure fails its
> dispatch because the server is not online so it queues a SCP and waits on it
> to break the RPC. The SCP can't run though because the MRP holds lock on the
> region.
> I can bypass the MRP but then the SCP fails because Region is 'owned' by the
> MRP. See below:
> {code}
> 2018-10-12 16:29:53,423 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Begin bypass
> pid=411982, ppid=411981, state=RUNNABLE:REGION_TRANSITION_DISPATCH,
> locked=true; UnassignProcedure
> table=IntegrationTestBigLinkedList_20180709093726,
> region=f5f9ff1e4b0f2d9555dabfcca71df568, override=true,
> server=va1002.halxg.cloudera.com,22101,1539368318649 with lockWait=0,
> override=true, recursive=true
> 2018-10-12 16:29:53,424 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Bypassing pid=411982,
> ppid=411981, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true;
> UnassignProcedure table=IntegrationTestBigLinkedList_20180709093726,
> region=f5f9ff1e4b0f2d9555dabfcca71df568, override=true,
> server=va1002.halxg.cloudera.com,22101,1539368318649
> 2018-10-12 16:29:53,712 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Bypassing pid=411981,
> state=WAITING:MOVE_REGION_ASSIGN, locked=true; MoveRegionProcedure
> hri=f5f9ff1e4b0f2d9555dabfcca71df568,
> source=va1002.halxg.cloudera.com,22101,1539368318649,
> destination=vd1021.halxg.cloudera.com,22101,1539368317897
> 2018-10-12 16:29:53,838 INFO
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Bypassing pid=411982,
> ppid=411981, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true,
> bypass=LOG-REDACTED UnassignProcedure
> table=IntegrationTestBigLinkedList_20180709093726,
> region=f5f9ff1e4b0f2d9555dabfcca71df568, override=true,
> server=va1002.halxg.cloudera.com,22101,1539368318649 and its ancestors
> successfully, adding to queue
> 2018-10-12 16:29:53,839 INFO org.apache.hadoop.hbase.procedure2.Procedure:
> pid=411982, ppid=411981,
[jira] [Created] (HBASE-21307) [amv2] Deadlock when we move a Region from a not-online RegionServer
stack created HBASE-21307:
-
Summary: [amv2] Deadlock when we move a Region from a not-online
RegionServer
Key: HBASE-21307
URL: https://issues.apache.org/jira/browse/HBASE-21307
Project: HBase
Issue Type: Bug
Components: amv2
Affects Versions: 2.1.1
Reporter: stack
Assignee: stack
Fix For: 2.1.1
Perhaps this doesn't happen in branch-2, but its problem in branch-2.1.
Highlevel, we go to move a region, its unassign subprocedure fails its dispatch
because the server is not online so it queues a SCP and waits on it to break
the RPC. The SCP can't run though because the MRP holds lock on the region.
I can bypass the MRP but then the SCP fails because Region is 'owned' by the
MRP. See below:
{code}
2018-10-12 16:29:53,423 INFO
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Begin bypass pid=411982,
ppid=411981, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true;
UnassignProcedure table=IntegrationTestBigLinkedList_20180709093726,
region=f5f9ff1e4b0f2d9555dabfcca71df568, override=true,
server=va1002.halxg.cloudera.com,22101,1539368318649 with lockWait=0,
override=true, recursive=true
2018-10-12 16:29:53,424 INFO
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Bypassing pid=411982,
ppid=411981, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true;
UnassignProcedure table=IntegrationTestBigLinkedList_20180709093726,
region=f5f9ff1e4b0f2d9555dabfcca71df568, override=true,
server=va1002.halxg.cloudera.com,22101,1539368318649
2018-10-12 16:29:53,712 INFO
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Bypassing pid=411981,
state=WAITING:MOVE_REGION_ASSIGN, locked=true; MoveRegionProcedure
hri=f5f9ff1e4b0f2d9555dabfcca71df568,
source=va1002.halxg.cloudera.com,22101,1539368318649,
destination=vd1021.halxg.cloudera.com,22101,1539368317897
2018-10-12 16:29:53,838 INFO
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Bypassing pid=411982,
ppid=411981, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true,
bypass=LOG-REDACTED UnassignProcedure
table=IntegrationTestBigLinkedList_20180709093726,
region=f5f9ff1e4b0f2d9555dabfcca71df568, override=true,
server=va1002.halxg.cloudera.com,22101,1539368318649 and its ancestors
successfully, adding to queue
2018-10-12 16:29:53,839 INFO org.apache.hadoop.hbase.procedure2.Procedure:
pid=411982, ppid=411981, state=RUNNABLE:REGION_TRANSITION_DISPATCH,
locked=true, bypass=LOG-REDACTED UnassignProcedure
table=IntegrationTestBigLinkedList_20180709093726,
region=f5f9ff1e4b0f2d9555dabfcca71df568, override=true,
server=va1002.halxg.cloudera.com,22101,1539368318649 bypassed, returning null
to finish it
2018-10-12 16:29:53,954 INFO
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished subprocedure
pid=411982, resume processing parent pid=411981,
state=RUNNABLE:MOVE_REGION_ASSIGN, locked=true, bypass=LOG-REDACTED
MoveRegionProcedure hri=f5f9ff1e4b0f2d9555dabfcca71df568,
source=va1002.halxg.cloudera.com,22101,1539368318649,
destination=vd1021.halxg.cloudera.com,22101,1539368317897
2018-10-12 16:29:53,954 INFO org.apache.hadoop.hbase.procedure2.Procedure:
pid=411981, state=RUNNABLE:MOVE_REGION_ASSIGN, locked=true, bypass=LOG-REDACTED
MoveRegionProcedure hri=f5f9ff1e4b0f2d9555dabfcca71df568,
source=va1002.halxg.cloudera.com,22101,1539368318649,
destination=vd1021.halxg.cloudera.com,22101,1539368317897 bypassed, returning
null to finish it
2018-10-12 16:29:53,956 INFO
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=411982,
ppid=411981, state=SUCCESS, bypass=LOG-REDACTED UnassignProcedure
table=IntegrationTestBigLinkedList_20180709093726,
region=f5f9ff1e4b0f2d9555dabfcca71df568, override=true,
server=va1002.halxg.cloudera.com,22101,1539368318649 in 3hrs, 49mins,
12.419sec, unfinishedSiblingCount=0
2018-10-12 16:29:54,058 INFO
org.apache.hadoop.hbase.procedure2.ProcedureExecutor: Finished pid=411981,
state=SUCCESS, bypass=LOG-REDACTED MoveRegionProcedure
hri=f5f9ff1e4b0f2d9555dabfcca71df568,
source=va1002.halxg.cloudera.com,22101,1539368318649,
destination=vd1021.halxg.cloudera.com,22101,1539368317897 in 3hrs, 49mins,
12.878sec
2018-10-12 16:29:54,059 INFO
org.apache.hadoop.hbase.master.procedure.MasterProcedureScheduler: xlock for
pid=412210, ppid=411983, state=RUNNABLE:REGION_TRANSITION_QUEUE;
AssignProcedure table=IntegrationTestBigLinkedList_20180709093726,
region=f5f9ff1e4b0f2d9555dabfcca71df568
2018-10-12 16:29:54,105 WARN
org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure:
f5f9ff1e4b0f2d9555dabfcca71df568 owned by pid=411982, CANNOT run 'this'
(pid=412210).
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21242) [amv2] Miscellaneous minor log and assign procedure create improvements
[
https://issues.apache.org/jira/browse/HBASE-21242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648582#comment-16648582
]
Hadoop QA commented on HBASE-21242:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
4s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
40s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
3s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
31s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
15s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
25s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
0s{color} | {color:red} hbase-server: The patch generated 1 new + 7 unchanged -
0 fixed = 8 total (was 7) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
7s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
7m 41s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
9s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
53s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}261m 3s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
48s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}312m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.wal.TestAsyncWALReplay |
| | hadoop.hbase.master.procedure.TestServerCrashProcedure |
| | hadoop.hbase.master.TestServerCrashProcedureStuck |
| |
hadoop.hbase.replication.multiwal.TestReplicationSyncUpToolWithMultipleWAL |
| |
hadoop.hbase.security.visibility.TestVisibilityLabelsWithDefaultVisLabelService
|
| | hadoop.hbase.regionserver.wal.TestWALReplayCompressed |
| |
[jira] [Commented] (HBASE-21178) [BC break] : Get and Scan operation with a custom converter_class not working
[
https://issues.apache.org/jira/browse/HBASE-21178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648572#comment-16648572
]
Hudson commented on HBASE-21178:
Results for branch branch-2
[build #1381 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1381/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1381//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1381//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1381//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> [BC break] : Get and Scan operation with a custom converter_class not working
> -
>
> Key: HBASE-21178
> URL: https://issues.apache.org/jira/browse/HBASE-21178
> Project: HBase
> Issue Type: Bug
> Components: shell
>Affects Versions: 2.0.0
>Reporter: Subrat Mishra
>Assignee: Subrat Mishra
>Priority: Critical
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21178.master.001.patch,
> HBASE-21178.master.002.patch, HBASE-21178.master.003.patch
>
>
> Consider a simple scenario:
> {code:java}
> create 'foo', {NAME => 'f1'}
> put 'foo','r1','f1:a',1000
> get 'foo','r1',{COLUMNS =>
> ['f1:a:c(org.apache.hadoop.hbase.util.Bytes).len']}
> scan 'foo',{COLUMNS =>
> ['f1:a:c(org.apache.hadoop.hbase.util.Bytes).len']}{code}
> Both get and scan fails with ERROR
> {code:java}
> ERROR: wrong number of arguments (3 for 1) {code}
> Looks like in table.rb file converter_method expects 3 arguments [(bytes,
> offset, len)] since version 2.0.0, prior to version 2.0.0 it was taking only
> 1 argument [(bytes)]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21289) Remove the log "'hbase.regionserver.maxlogs' was deprecated." in AbstractFSWAL
[
https://issues.apache.org/jira/browse/HBASE-21289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648571#comment-16648571
]
Hudson commented on HBASE-21289:
Results for branch branch-2
[build #1381 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1381/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1381//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1381//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1381//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove the log "'hbase.regionserver.maxlogs' was deprecated." in AbstractFSWAL
> --
>
> Key: HBASE-21289
> URL: https://issues.apache.org/jira/browse/HBASE-21289
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21289.master.001.patch
>
>
> This log was added by HBASE-14951. And the description and release note never
> said this config was deprecated. I thought HBASE-14951 only changed the
> default value of maxlogs (Please correct me if I am wrong). And we still use
> this config in our hbase book. So the log "'hbase.regionserver.maxlogs' was
> deprecated." in AbstractFSWAL is confused. Let's remove it. FYI [~vrodionov]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21303) [shell] clear_deadservers with no args fails
[
https://issues.apache.org/jira/browse/HBASE-21303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648574#comment-16648574
]
Hudson commented on HBASE-21303:
Results for branch branch-2
[build #1381 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1381/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1381//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1381//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1381//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> [shell] clear_deadservers with no args fails
>
>
> Key: HBASE-21303
> URL: https://issues.apache.org/jira/browse/HBASE-21303
> Project: HBase
> Issue Type: Improvement
>Reporter: Stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21303.branch-2.1.001.patch
>
>
> Tried running clear_deadservers on tip of branch-2.1 and it fails
> {code}
> hbase(main):007:0> help "clear_deadservers"
> Clear the dead region servers that are never used.
> Examples:
> Clear all dead region servers:
> hbase> clear_deadservers
> Clear the specified dead region servers:
> hbase> clear_deadservers 'host187.example.com,60020,1289493121758'
> or
> hbase> clear_deadservers 'host187.example.com,60020,1289493121758',
>'host188.example.com,60020,1289493121758'
> hbase(main):008:0> clear_deadservers
> ERROR: servers cannot be null or empty
> For usage try 'help "clear_deadservers"'
> Took 0.0094 seconds
> hbase(main):009:0> clear_deadservers
> ERROR: servers cannot be null or empty
> For usage try 'help "clear_deadservers"'
> Took 0.0084 seconds
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21299) List counts of actual region states in master UI tables section
[
https://issues.apache.org/jira/browse/HBASE-21299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648573#comment-16648573
]
Hudson commented on HBASE-21299:
Results for branch branch-2
[build #1381 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1381/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1381//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1381//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1381//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> List counts of actual region states in master UI tables section
> ---
>
> Key: HBASE-21299
> URL: https://issues.apache.org/jira/browse/HBASE-21299
> Project: HBase
> Issue Type: Improvement
> Components: UI
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-21299.branch-2.1.001.patch, Screen Shot 2018-10-11
> at 9.44.39 PM.png, Screen Shot 2018-10-11 at 9.44.56 PM.png
>
>
> It the tables panel, we list Open Regions, Offline Regions, Failed Regions,
> Split Region, and Other. This is not very useful. It is from a time before
> region states were edited down. Better to list OPEN, CLOSED, OPENING,
> CLOSING...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21266) Not running balancer because processing dead regionservers, but empty dead rs list
[ https://issues.apache.org/jira/browse/HBASE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648562#comment-16648562 ] Andrew Purtell commented on HBASE-21266: Just waiting for a 1B row ITBLL with serverKilling chaos policy to complete on the latest patch. Still running. Looks good so far. Previously, problems surfaced quickly. Unit tests all look good. > Not running balancer because processing dead regionservers, but empty dead rs > list > -- > > Key: HBASE-21266 > URL: https://issues.apache.org/jira/browse/HBASE-21266 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0, 1.4.9 > > Attachments: HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch > > > Found during ITBLL testing. AM in master gets into a state where manual > attempts from the shell to run the balancer always return false and this is > printed in the master log: > 2018-10-03 19:17:14,892 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=21,queue=0,port=8100] master.HMaster: > Not running balancer because processing dead regionserver(s): > Note the empty list. > This errant state did not recover without intervention by way of master > restart, but the test environment was chaotic so needs investigation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21303) [shell] clear_deadservers with no args fails
[
https://issues.apache.org/jira/browse/HBASE-21303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648561#comment-16648561
]
Hudson commented on HBASE-21303:
Results for branch branch-2.0
[build #942 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/942/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/942//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/942//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/942//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> [shell] clear_deadservers with no args fails
>
>
> Key: HBASE-21303
> URL: https://issues.apache.org/jira/browse/HBASE-21303
> Project: HBase
> Issue Type: Improvement
>Reporter: Stack
>Assignee: stack
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21303.branch-2.1.001.patch
>
>
> Tried running clear_deadservers on tip of branch-2.1 and it fails
> {code}
> hbase(main):007:0> help "clear_deadservers"
> Clear the dead region servers that are never used.
> Examples:
> Clear all dead region servers:
> hbase> clear_deadservers
> Clear the specified dead region servers:
> hbase> clear_deadservers 'host187.example.com,60020,1289493121758'
> or
> hbase> clear_deadservers 'host187.example.com,60020,1289493121758',
>'host188.example.com,60020,1289493121758'
> hbase(main):008:0> clear_deadservers
> ERROR: servers cannot be null or empty
> For usage try 'help "clear_deadservers"'
> Took 0.0094 seconds
> hbase(main):009:0> clear_deadservers
> ERROR: servers cannot be null or empty
> For usage try 'help "clear_deadservers"'
> Took 0.0084 seconds
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21289) Remove the log "'hbase.regionserver.maxlogs' was deprecated." in AbstractFSWAL
[
https://issues.apache.org/jira/browse/HBASE-21289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648560#comment-16648560
]
Hudson commented on HBASE-21289:
Results for branch branch-2.0
[build #942 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/942/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/942//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/942//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/942//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Remove the log "'hbase.regionserver.maxlogs' was deprecated." in AbstractFSWAL
> --
>
> Key: HBASE-21289
> URL: https://issues.apache.org/jira/browse/HBASE-21289
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21289.master.001.patch
>
>
> This log was added by HBASE-14951. And the description and release note never
> said this config was deprecated. I thought HBASE-14951 only changed the
> default value of maxlogs (Please correct me if I am wrong). And we still use
> this config in our hbase book. So the log "'hbase.regionserver.maxlogs' was
> deprecated." in AbstractFSWAL is confused. Let's remove it. FYI [~vrodionov]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21242) [amv2] Miscellaneous minor log and assign procedure create improvements
[
https://issues.apache.org/jira/browse/HBASE-21242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648556#comment-16648556
]
stack commented on HBASE-21242:
---
Ok. [~mdrob] pointed out that TestRegionInfoDisplay was also failing for same
reason; the test was copy/pasted but fixed up to use non-deprecated API.
So, I reverted the addendum and pushed a patch that undoes the duplication --
only one instance now (Fix is still hacky though drob if you are reading this
far).
> [amv2] Miscellaneous minor log and assign procedure create improvements
> ---
>
> Key: HBASE-21242
> URL: https://issues.apache.org/jira/browse/HBASE-21242
> Project: HBase
> Issue Type: Bug
> Components: amv2, Operability
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21242.addendum.fix.testhregioninfo.patch,
> HBASE-21242.branch-2.0.001.patch, HBASE-21242.branch-2.0.001.patch,
> HBASE-21242.branch-2.0.002.patch, HBASE-21242.branch-2.001.patch,
> HBASE-21242.branch-2.001.patch, HBASE-21242.branch-2.002.patch,
> HBASE-21242.branch-2.1.001.patch, HBASE-21242.branch-2.1.001.patch,
> HBASE-21242.branch-2.1.001.patch, HBASE-21242.branch-2.1.002.patch
>
>
> Some minor fixups:
> {code}
> For RIT Duration, do better than print ms/seconds. Remove redundant UI
> column dedicated to duration when we log it in the status field too.
> Make bypass log at INFO level -- when DEBUG we can miss important
> fixup detail like why we failed.
> Make it so on complete of subprocedure, we note count of outstanding
> siblings so we have a clue how much further the parent has to go before
> it is done (Helpful when hundreds of servers doing SCP).
> Have the SCP run the AP preflight check before creating an AP; saves
> creation of hundreds of thousands of APs during fixup of this big cluster
> of mine.
> Don't log tablename three times when reporting remote call failed.
> If lock is held already, note who has it. Also log after we get lock
> or if we have to wait rather than log on entrance though we may
> later have to wait (or we may have just picked up the lock).
> {code}
> Posting patch in a sec but let me try it on cluster too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21114) website should have a copy of 2.1 release docs
[ https://issues.apache.org/jira/browse/HBASE-21114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648536#comment-16648536 ] Sean Busbey commented on HBASE-21114: - oh I see now. the push for the actual docs went through and is live. but the website publishing jenkins job hasn't run yet, so the menu option isn't there. (ex: http://hbase.apache.org/2.1/book.html ) I triggered the site building job again instead of waiting for the next timer: https://builds.apache.org/view/H-L/view/HBase/job/hbase_generate_website/ > website should have a copy of 2.1 release docs > -- > > Key: HBASE-21114 > URL: https://issues.apache.org/jira/browse/HBASE-21114 > Project: HBase > Issue Type: Task > Components: community, documentation >Affects Versions: 2.1.0 >Reporter: Sean Busbey >Assignee: Mike Drob >Priority: Major > Fix For: 2.1.1 > > Attachments: HBASE-21114.patch > > > in the "Documentation and API" menu we have entries for 2.0 and 1.2. should > also add in 2.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21114) website should have a copy of 2.1 release docs
[ https://issues.apache.org/jira/browse/HBASE-21114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648537#comment-16648537 ] Sean Busbey commented on HBASE-21114: - it takes about 20 minutes. > website should have a copy of 2.1 release docs > -- > > Key: HBASE-21114 > URL: https://issues.apache.org/jira/browse/HBASE-21114 > Project: HBase > Issue Type: Task > Components: community, documentation >Affects Versions: 2.1.0 >Reporter: Sean Busbey >Assignee: Mike Drob >Priority: Major > Fix For: 2.1.1 > > Attachments: HBASE-21114.patch > > > in the "Documentation and API" menu we have entries for 2.0 and 1.2. should > also add in 2.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21302) Release 1.2.8
[ https://issues.apache.org/jira/browse/HBASE-21302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648531#comment-16648531 ] Sean Busbey commented on HBASE-21302: - RC0 posted: https://lists.apache.org/thread.html/b89ab5edca81aad46fba06f6999360c5e4a562568938fa4a6c14a0dd@%3Cdev.hbase.apache.org%3E > Release 1.2.8 > - > > Key: HBASE-21302 > URL: https://issues.apache.org/jira/browse/HBASE-21302 > Project: HBase > Issue Type: Task > Components: community >Affects Versions: 1.2.8 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.2.8 > > > 1.4.8 is out, time to make 1.2.8. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21306) website should have 1.4.x release docs
[ https://issues.apache.org/jira/browse/HBASE-21306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21306: --- Fix Version/s: 1.4.9 > website should have 1.4.x release docs > -- > > Key: HBASE-21306 > URL: https://issues.apache.org/jira/browse/HBASE-21306 > Project: HBase > Issue Type: Task > Components: community, documentation, website >Affects Versions: 1.4.7 >Reporter: Mike Drob >Priority: Major > Fix For: 1.4.9 > > > Since 1.4 is the stable line now, we should include the API docs on the > website instead of (in addition to?) the 1.2 docs we have now. > See also HBASE-21114 and HBASE-21119 for the process. > FYI: [~busbey], [~apurtell] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21275) Thrift Server (branch 1 fix) -> Disable TRACE HTTP method for thrift http server (branch 1 only)
[ https://issues.apache.org/jira/browse/HBASE-21275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648505#comment-16648505 ] Andrew Purtell commented on HBASE-21275: Ok, thanks, lgtm > Thrift Server (branch 1 fix) -> Disable TRACE HTTP method for thrift http > server (branch 1 only) > > > Key: HBASE-21275 > URL: https://issues.apache.org/jira/browse/HBASE-21275 > Project: HBase > Issue Type: Bug > Components: Thrift >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Fix For: 1.4.8, 1.2.7 > > Attachments: HBASE-21275-branch-1.2.001.patch, > HBASE-21275-branch-1.2.002.patch, HBASE-21275-branch-1.2.003.patch, > HBASE-21275-branch-1.2.003.patch > > > There's been a reasonable number of users running thrift http server on hbase > 1.x suffering with security audit tests pointing thrift server allows TRACE > requests. > After doing some search, I can see HBASE-20406 added restrictions for > TRACE/OPTIONS method when Thrift is running over http, but it relies on many > other commits applied to thrift http server. This patch was later reverted > from master. Then again later, HBASE-20004 had made TRACE/OPTIONS > configurable via "*hbase.thrift.http.allow.options.method*" property, with > both methods being disabled by default. This also seems to rely on many > changes applied to thrift http server, and a branch 1 compatible patch does > not seem feasible. > A solution for branch 1 is pretty simple though, am proposing a patch that > simply uses *WebAppContext*, instead of *Context*, as the context for the > *HttpServer* instance. *WebAppContext* will already restrict TRACE methods by > default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21285) Enhanced TableSnapshotInputFormat to allow a size based splitting
[ https://issues.apache.org/jira/browse/HBASE-21285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648501#comment-16648501 ] Ted Yu commented on HBASE-21285: I applied 21285.branch-1.4.003.patch on branch-1 but didn't reproduce the above error locally. > Enhanced TableSnapshotInputFormat to allow a size based splitting > - > > Key: HBASE-21285 > URL: https://issues.apache.org/jira/browse/HBASE-21285 > Project: HBase > Issue Type: Improvement > Components: snapshots >Affects Versions: 1.4.0 >Reporter: Lavinia-Stefania Sirbu >Assignee: Lavinia-Stefania Sirbu >Priority: Minor > Attachments: HBASE-21285.branch-1.4.001.patch, > HBASE-21285.branch-1.4.002.patch, HBASE-21285.branch-1.4.003.patch > > > Currently, all the splits generated by a snapshot are having length 0. Right > now, we have a configuration for the number of splits per region, but it's a > general one and not very helpful when the sizes for regions are really > different. The modification must be done in TableSnapshotInputFormatImpl > where the length must be computed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21306) website should have 1.4.x release docs
[ https://issues.apache.org/jira/browse/HBASE-21306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648491#comment-16648491 ] Sean Busbey commented on HBASE-21306: - 1.2 isn't EOM yet so we shouldn't remove it > website should have 1.4.x release docs > -- > > Key: HBASE-21306 > URL: https://issues.apache.org/jira/browse/HBASE-21306 > Project: HBase > Issue Type: Task > Components: community, documentation, website >Affects Versions: 1.4.7 >Reporter: Mike Drob >Priority: Major > > Since 1.4 is the stable line now, we should include the API docs on the > website instead of (in addition to?) the 1.2 docs we have now. > See also HBASE-21114 and HBASE-21119 for the process. > FYI: [~busbey], [~apurtell] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21285) Enhanced TableSnapshotInputFormat to allow a size based splitting
[
https://issues.apache.org/jira/browse/HBASE-21285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648488#comment-16648488
]
Ted Yu commented on HBASE-21285:
Please check the following test error:
{code}
[ERROR]
testNoDuplicateResultsWhenSplitting(org.apache.hadoop.hbase.mapreduce.TestTableSnapshotInputFormat)
Time elapsed: 25.208 s <<< FAILURE!
java.lang.AssertionError
at
org.apache.hadoop.hbase.mapreduce.TestTableSnapshotInputFormat.verifyWithMockedMapReduce(TestTableSnapshotInputFormat.java:374)
at
org.apache.hadoop.hbase.mapreduce.TestTableSnapshotInputFormat.testNoDuplicateResultsWhenSplitting(TestTableSnapshotInputFormat.java:351)
{code}
Please attach patch for master branch.
> Enhanced TableSnapshotInputFormat to allow a size based splitting
> -
>
> Key: HBASE-21285
> URL: https://issues.apache.org/jira/browse/HBASE-21285
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
>Affects Versions: 1.4.0
>Reporter: Lavinia-Stefania Sirbu
>Assignee: Lavinia-Stefania Sirbu
>Priority: Minor
> Attachments: HBASE-21285.branch-1.4.001.patch,
> HBASE-21285.branch-1.4.002.patch, HBASE-21285.branch-1.4.003.patch
>
>
> Currently, all the splits generated by a snapshot are having length 0. Right
> now, we have a configuration for the number of splits per region, but it's a
> general one and not very helpful when the sizes for regions are really
> different. The modification must be done in TableSnapshotInputFormatImpl
> where the length must be computed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-21285) Enhanced TableSnapshotInputFormat to allow a size based splitting
[ https://issues.apache.org/jira/browse/HBASE-21285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reassigned HBASE-21285: -- Assignee: Lavinia-Stefania Sirbu > Enhanced TableSnapshotInputFormat to allow a size based splitting > - > > Key: HBASE-21285 > URL: https://issues.apache.org/jira/browse/HBASE-21285 > Project: HBase > Issue Type: Improvement > Components: snapshots >Affects Versions: 1.4.0 >Reporter: Lavinia-Stefania Sirbu >Assignee: Lavinia-Stefania Sirbu >Priority: Minor > Attachments: HBASE-21285.branch-1.4.001.patch, > HBASE-21285.branch-1.4.002.patch, HBASE-21285.branch-1.4.003.patch > > > Currently, all the splits generated by a snapshot are having length 0. Right > now, we have a configuration for the number of splits per region, but it's a > general one and not very helpful when the sizes for regions are really > different. The modification must be done in TableSnapshotInputFormatImpl > where the length must be computed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-21286) Parallelize computeHDFSBlocksDistribution when getting splits of a HBaseSnapshot
[ https://issues.apache.org/jira/browse/HBASE-21286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reassigned HBASE-21286: -- Assignee: Lavinia-Stefania Sirbu > Parallelize computeHDFSBlocksDistribution when getting splits of a > HBaseSnapshot > > > Key: HBASE-21286 > URL: https://issues.apache.org/jira/browse/HBASE-21286 > Project: HBase > Issue Type: Improvement > Components: snapshots >Affects Versions: 1.4.0 >Reporter: Lavinia-Stefania Sirbu >Assignee: Lavinia-Stefania Sirbu >Priority: Minor > Attachments: HBASE-21286.branch-1.4.001.patch, > HBASE-21286.branch-1.4.002.patch > > > Even if this step is called computeHDFSBlocksDistribution, this is executed > no matter the file system of the snapshot. For example, we have observed an > important slowness when we have a snapshot in s3 (~26k regions, 5column > families, 2 files per column family) the getsplits time is ~40min due to the > calls in s3 for listing the files to get the best locations. > Parallelizing this operation can reduce the overall setup time. The thread > pool should be configurable and a good choice could be > "hbase.snapshot.thread.pool.max" that is also used in RestoreSnapshotHelper. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21306) website should have 1.4.x release docs
Mike Drob created HBASE-21306: - Summary: website should have 1.4.x release docs Key: HBASE-21306 URL: https://issues.apache.org/jira/browse/HBASE-21306 Project: HBase Issue Type: Task Components: community, documentation, website Affects Versions: 1.4.7 Reporter: Mike Drob Since 1.4 is the stable line now, we should include the API docs on the website instead of (in addition to?) the 1.2 docs we have now. See also HBASE-21114 and HBASE-21119 for the process. FYI: [~busbey], [~apurtell] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21114) website should have a copy of 2.1 release docs
[ https://issues.apache.org/jira/browse/HBASE-21114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648473#comment-16648473 ] Mike Drob commented on HBASE-21114: --- Committed, but I don't see it on the live site yet. Which Jenkins do I go look for to figure out if things are broken or still processing? > website should have a copy of 2.1 release docs > -- > > Key: HBASE-21114 > URL: https://issues.apache.org/jira/browse/HBASE-21114 > Project: HBase > Issue Type: Task > Components: community, documentation >Affects Versions: 2.1.0 >Reporter: Sean Busbey >Assignee: Mike Drob >Priority: Major > Fix For: 2.1.1 > > Attachments: HBASE-21114.patch > > > in the "Documentation and API" menu we have entries for 2.0 and 1.2. should > also add in 2.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21302) Release 1.2.8
[
https://issues.apache.org/jira/browse/HBASE-21302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648459#comment-16648459
]
Hudson commented on HBASE-21302:
Results for branch branch-1.2
[build #509 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/509/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/509//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/509//JDK7_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.2/509//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Release 1.2.8
> -
>
> Key: HBASE-21302
> URL: https://issues.apache.org/jira/browse/HBASE-21302
> Project: HBase
> Issue Type: Task
> Components: community
>Affects Versions: 1.2.8
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 1.2.8
>
>
> 1.4.8 is out, time to make 1.2.8.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21266) Not running balancer because processing dead regionservers, but empty dead rs list
[
https://issues.apache.org/jira/browse/HBASE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648433#comment-16648433
]
Hadoop QA commented on HBASE-21266:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
57s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
58s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} branch-1 passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} branch-1 passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} hbase-server: The patch generated 0 new + 68
unchanged - 5 fixed = 68 total (was 73) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
7s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 59s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed with JDK v1.8.0_181 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed with JDK v1.7.0_191 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}113m
45s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}141m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA Issue | HBASE-21266 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12943695/HBASE-21266-branch-1.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 1fabe85234fb 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64
[jira] [Updated] (HBASE-21266) Not running balancer because processing dead regionservers, but empty dead rs list
[ https://issues.apache.org/jira/browse/HBASE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21266: --- Status: Patch Available (was: Open) > Not running balancer because processing dead regionservers, but empty dead rs > list > -- > > Key: HBASE-21266 > URL: https://issues.apache.org/jira/browse/HBASE-21266 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0, 1.4.9 > > Attachments: HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch > > > Found during ITBLL testing. AM in master gets into a state where manual > attempts from the shell to run the balancer always return false and this is > printed in the master log: > 2018-10-03 19:17:14,892 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=21,queue=0,port=8100] master.HMaster: > Not running balancer because processing dead regionserver(s): > Note the empty list. > This errant state did not recover without intervention by way of master > restart, but the test environment was chaotic so needs investigation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21266) Not running balancer because processing dead regionservers, but empty dead rs list
[ https://issues.apache.org/jira/browse/HBASE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21266: --- Attachment: HBASE-21266-branch-1.patch > Not running balancer because processing dead regionservers, but empty dead rs > list > -- > > Key: HBASE-21266 > URL: https://issues.apache.org/jira/browse/HBASE-21266 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0, 1.4.9 > > Attachments: HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch > > > Found during ITBLL testing. AM in master gets into a state where manual > attempts from the shell to run the balancer always return false and this is > printed in the master log: > 2018-10-03 19:17:14,892 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=21,queue=0,port=8100] master.HMaster: > Not running balancer because processing dead regionserver(s): > Note the empty list. > This errant state did not recover without intervention by way of master > restart, but the test environment was chaotic so needs investigation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21247) Custom WAL Provider cannot be specified by configuration whose value is outside the enums in Providers
[ https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648372#comment-16648372 ] Ted Yu commented on HBASE-21247: See if patch v8 is better. Class retrieval from ordinary WAL provider works. In v8, if meta WAL provider is not specified, provider class of ordinary WAL would be used. > Custom WAL Provider cannot be specified by configuration whose value is > outside the enums in Providers > -- > > Key: HBASE-21247 > URL: https://issues.apache.org/jira/browse/HBASE-21247 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: 3.0.0 > > Attachments: 21247.v1.txt, 21247.v2.txt, 21247.v3.txt, 21247.v4.tst, > 21247.v4.txt, 21247.v5.txt, 21247.v6.txt, 21247.v7.txt, 21247.v8.txt > > > Currently all the WAL Providers acceptable to hbase are specified in > Providers enum of WALFactory. > This restricts the ability for additional WAL Providers to be supplied - by > class name. > This issue fixes the bug by allowing the specification of new WAL Provider > class name using the config "hbase.wal.provider". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21247) Custom WAL Provider cannot be specified by configuration whose value is outside the enums in Providers
[ https://issues.apache.org/jira/browse/HBASE-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-21247: --- Attachment: 21247.v8.txt > Custom WAL Provider cannot be specified by configuration whose value is > outside the enums in Providers > -- > > Key: HBASE-21247 > URL: https://issues.apache.org/jira/browse/HBASE-21247 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Major > Fix For: 3.0.0 > > Attachments: 21247.v1.txt, 21247.v2.txt, 21247.v3.txt, 21247.v4.tst, > 21247.v4.txt, 21247.v5.txt, 21247.v6.txt, 21247.v7.txt, 21247.v8.txt > > > Currently all the WAL Providers acceptable to hbase are specified in > Providers enum of WALFactory. > This restricts the ability for additional WAL Providers to be supplied - by > class name. > This issue fixes the bug by allowing the specification of new WAL Provider > class name using the config "hbase.wal.provider". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21266) Not running balancer because processing dead regionservers, but empty dead rs list
[ https://issues.apache.org/jira/browse/HBASE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21266: --- Status: Open (was: Patch Available) > Not running balancer because processing dead regionservers, but empty dead rs > list > -- > > Key: HBASE-21266 > URL: https://issues.apache.org/jira/browse/HBASE-21266 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0, 1.4.9 > > Attachments: HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch > > > Found during ITBLL testing. AM in master gets into a state where manual > attempts from the shell to run the balancer always return false and this is > printed in the master log: > 2018-10-03 19:17:14,892 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=21,queue=0,port=8100] master.HMaster: > Not running balancer because processing dead regionserver(s): > Note the empty list. > This errant state did not recover without intervention by way of master > restart, but the test environment was chaotic so needs investigation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21242) [amv2] Miscellaneous minor log and assign procedure create improvements
[
https://issues.apache.org/jira/browse/HBASE-21242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648367#comment-16648367
]
stack commented on HBASE-21242:
---
I pushed the addendum on 2.1. Will fold into the branch-2 patch.
> [amv2] Miscellaneous minor log and assign procedure create improvements
> ---
>
> Key: HBASE-21242
> URL: https://issues.apache.org/jira/browse/HBASE-21242
> Project: HBase
> Issue Type: Bug
> Components: amv2, Operability
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21242.addendum.fix.testhregioninfo.patch,
> HBASE-21242.branch-2.0.001.patch, HBASE-21242.branch-2.0.001.patch,
> HBASE-21242.branch-2.0.002.patch, HBASE-21242.branch-2.001.patch,
> HBASE-21242.branch-2.001.patch, HBASE-21242.branch-2.002.patch,
> HBASE-21242.branch-2.1.001.patch, HBASE-21242.branch-2.1.001.patch,
> HBASE-21242.branch-2.1.001.patch, HBASE-21242.branch-2.1.002.patch
>
>
> Some minor fixups:
> {code}
> For RIT Duration, do better than print ms/seconds. Remove redundant UI
> column dedicated to duration when we log it in the status field too.
> Make bypass log at INFO level -- when DEBUG we can miss important
> fixup detail like why we failed.
> Make it so on complete of subprocedure, we note count of outstanding
> siblings so we have a clue how much further the parent has to go before
> it is done (Helpful when hundreds of servers doing SCP).
> Have the SCP run the AP preflight check before creating an AP; saves
> creation of hundreds of thousands of APs during fixup of this big cluster
> of mine.
> Don't log tablename three times when reporting remote call failed.
> If lock is held already, note who has it. Also log after we get lock
> or if we have to wait rather than log on entrance though we may
> later have to wait (or we may have just picked up the lock).
> {code}
> Posting patch in a sec but let me try it on cluster too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21266) Not running balancer because processing dead regionservers, but empty dead rs list
[ https://issues.apache.org/jira/browse/HBASE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21266: --- Attachment: (was: HBASE-21266-branch-1.patch) > Not running balancer because processing dead regionservers, but empty dead rs > list > -- > > Key: HBASE-21266 > URL: https://issues.apache.org/jira/browse/HBASE-21266 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0, 1.4.9 > > Attachments: HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch > > > Found during ITBLL testing. AM in master gets into a state where manual > attempts from the shell to run the balancer always return false and this is > printed in the master log: > 2018-10-03 19:17:14,892 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=21,queue=0,port=8100] master.HMaster: > Not running balancer because processing dead regionserver(s): > Note the empty list. > This errant state did not recover without intervention by way of master > restart, but the test environment was chaotic so needs investigation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21242) [amv2] Miscellaneous minor log and assign procedure create improvements
[
https://issues.apache.org/jira/browse/HBASE-21242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648366#comment-16648366
]
stack commented on HBASE-21242:
---
I put up addendum HBASE-21242.addendum.fix.testhregioninfo.patch. Let me push
on branch-2.1.
> [amv2] Miscellaneous minor log and assign procedure create improvements
> ---
>
> Key: HBASE-21242
> URL: https://issues.apache.org/jira/browse/HBASE-21242
> Project: HBase
> Issue Type: Bug
> Components: amv2, Operability
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21242.addendum.fix.testhregioninfo.patch,
> HBASE-21242.branch-2.0.001.patch, HBASE-21242.branch-2.0.001.patch,
> HBASE-21242.branch-2.0.002.patch, HBASE-21242.branch-2.001.patch,
> HBASE-21242.branch-2.001.patch, HBASE-21242.branch-2.002.patch,
> HBASE-21242.branch-2.1.001.patch, HBASE-21242.branch-2.1.001.patch,
> HBASE-21242.branch-2.1.001.patch, HBASE-21242.branch-2.1.002.patch
>
>
> Some minor fixups:
> {code}
> For RIT Duration, do better than print ms/seconds. Remove redundant UI
> column dedicated to duration when we log it in the status field too.
> Make bypass log at INFO level -- when DEBUG we can miss important
> fixup detail like why we failed.
> Make it so on complete of subprocedure, we note count of outstanding
> siblings so we have a clue how much further the parent has to go before
> it is done (Helpful when hundreds of servers doing SCP).
> Have the SCP run the AP preflight check before creating an AP; saves
> creation of hundreds of thousands of APs during fixup of this big cluster
> of mine.
> Don't log tablename three times when reporting remote call failed.
> If lock is held already, note who has it. Also log after we get lock
> or if we have to wait rather than log on entrance though we may
> later have to wait (or we may have just picked up the lock).
> {code}
> Posting patch in a sec but let me try it on cluster too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21242) [amv2] Miscellaneous minor log and assign procedure create improvements
[
https://issues.apache.org/jira/browse/HBASE-21242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-21242:
--
Attachment: HBASE-21242.addendum.fix.testhregioninfo.patch
> [amv2] Miscellaneous minor log and assign procedure create improvements
> ---
>
> Key: HBASE-21242
> URL: https://issues.apache.org/jira/browse/HBASE-21242
> Project: HBase
> Issue Type: Bug
> Components: amv2, Operability
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21242.addendum.fix.testhregioninfo.patch,
> HBASE-21242.branch-2.0.001.patch, HBASE-21242.branch-2.0.001.patch,
> HBASE-21242.branch-2.0.002.patch, HBASE-21242.branch-2.001.patch,
> HBASE-21242.branch-2.001.patch, HBASE-21242.branch-2.002.patch,
> HBASE-21242.branch-2.1.001.patch, HBASE-21242.branch-2.1.001.patch,
> HBASE-21242.branch-2.1.001.patch, HBASE-21242.branch-2.1.002.patch
>
>
> Some minor fixups:
> {code}
> For RIT Duration, do better than print ms/seconds. Remove redundant UI
> column dedicated to duration when we log it in the status field too.
> Make bypass log at INFO level -- when DEBUG we can miss important
> fixup detail like why we failed.
> Make it so on complete of subprocedure, we note count of outstanding
> siblings so we have a clue how much further the parent has to go before
> it is done (Helpful when hundreds of servers doing SCP).
> Have the SCP run the AP preflight check before creating an AP; saves
> creation of hundreds of thousands of APs during fixup of this big cluster
> of mine.
> Don't log tablename three times when reporting remote call failed.
> If lock is held already, note who has it. Also log after we get lock
> or if we have to wait rather than log on entrance though we may
> later have to wait (or we may have just picked up the lock).
> {code}
> Posting patch in a sec but let me try it on cluster too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21305) TestHRegionInfo failing because descriptive strings have different times
Mike Drob created HBASE-21305: - Summary: TestHRegionInfo failing because descriptive strings have different times Key: HBASE-21305 URL: https://issues.apache.org/jira/browse/HBASE-21305 Project: HBase Issue Type: Task Components: test Reporter: Mike Drob Assignee: Mike Drob We see TestHRegionInfo.testRegionDetailsForDisplay failing because the strings now show timestamps as created at differing points in the past. Can use EnvironmentEdge to fix this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21158) Empty qualifier cell should not be returned if it does not match QualifierFilter
[
https://issues.apache.org/jira/browse/HBASE-21158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-21158:
Summary: Empty qualifier cell should not be returned if it does not match
QualifierFilter (was: Empty qualifier cell is always returned when using
QualifierFilter)
> Empty qualifier cell should not be returned if it does not match
> QualifierFilter
>
>
> Key: HBASE-21158
> URL: https://issues.apache.org/jira/browse/HBASE-21158
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 1.4.8, 2.1.1, 2.0.3
>
> Attachments: HBASE-21158.branch-1.001.patch,
> HBASE-21158.master.001.patch, HBASE-21158.master.002.patch,
> HBASE-21158.master.003.patch, HBASE-21158.master.004.patch
>
>
> {code:xml}
> hbase(main):002:0> put 'testTable','testrow','f:testcol1','testvalue1'
> 0 row(s) in 0.0040 seconds
> hbase(main):003:0> put 'testTable','testrow','f:','testvalue2'
> 0 row(s) in 0.0070 seconds
> # get row with empty column f:, result is correct.
> hbase(main):004:0> scan 'testTable',{FILTER => "QualifierFilter (=,
> 'binary:')"}
> ROW COLUMN+CELL
>
>
> testrowcolumn=f:,
> timestamp=1536218563581, value=testvalue2
>
> 1 row(s) in 0.0460 seconds
> # get row with column f:testcol1, result is incorrect.
> hbase(main):005:0> scan 'testTable',{FILTER => "QualifierFilter (=,
> 'binary:testcol1')"}
> ROW COLUMN+CELL
>
>
> testrowcolumn=f:,
> timestamp=1536218563581, value=testvalue2
>
> testrowcolumn=f:testcol1,
> timestamp=1536218550827, value=testvalue1
>
> 1 row(s) in 0.0070 seconds
> {code}
> As the above operation, when the row contains empty qualifier column, empty
> qualifier cell is always returned when using QualifierFilter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HBASE-21158) Empty qualifier cell is always returned when using QualifierFilter
[
https://issues.apache.org/jira/browse/HBASE-21158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey resolved HBASE-21158.
-
Resolution: Fixed
Hadoop Flags: Incompatible change
Release Note:
Scans that make use of `QualifierFilter` previously would erroneously return
both columns with an empty qualifier along with those that matched. After this
change that behavior has changed to only return those columns that match.
updated with release note and flagged it for a change in behavior.
> Empty qualifier cell is always returned when using QualifierFilter
> --
>
> Key: HBASE-21158
> URL: https://issues.apache.org/jira/browse/HBASE-21158
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 2.1.1, 2.0.3, 1.4.8
>
> Attachments: HBASE-21158.branch-1.001.patch,
> HBASE-21158.master.001.patch, HBASE-21158.master.002.patch,
> HBASE-21158.master.003.patch, HBASE-21158.master.004.patch
>
>
> {code:xml}
> hbase(main):002:0> put 'testTable','testrow','f:testcol1','testvalue1'
> 0 row(s) in 0.0040 seconds
> hbase(main):003:0> put 'testTable','testrow','f:','testvalue2'
> 0 row(s) in 0.0070 seconds
> # get row with empty column f:, result is correct.
> hbase(main):004:0> scan 'testTable',{FILTER => "QualifierFilter (=,
> 'binary:')"}
> ROW COLUMN+CELL
>
>
> testrowcolumn=f:,
> timestamp=1536218563581, value=testvalue2
>
> 1 row(s) in 0.0460 seconds
> # get row with column f:testcol1, result is incorrect.
> hbase(main):005:0> scan 'testTable',{FILTER => "QualifierFilter (=,
> 'binary:testcol1')"}
> ROW COLUMN+CELL
>
>
> testrowcolumn=f:,
> timestamp=1536218563581, value=testvalue2
>
> testrowcolumn=f:testcol1,
> timestamp=1536218550827, value=testvalue1
>
> 1 row(s) in 0.0070 seconds
> {code}
> As the above operation, when the row contains empty qualifier column, empty
> qualifier cell is always returned when using QualifierFilter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21266) Not running balancer because processing dead regionservers, but empty dead rs list
[ https://issues.apache.org/jira/browse/HBASE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21266: --- Status: Patch Available (was: Open) > Not running balancer because processing dead regionservers, but empty dead rs > list > -- > > Key: HBASE-21266 > URL: https://issues.apache.org/jira/browse/HBASE-21266 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0, 1.4.9 > > Attachments: HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch > > > Found during ITBLL testing. AM in master gets into a state where manual > attempts from the shell to run the balancer always return false and this is > printed in the master log: > 2018-10-03 19:17:14,892 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=21,queue=0,port=8100] master.HMaster: > Not running balancer because processing dead regionserver(s): > Note the empty list. > This errant state did not recover without intervention by way of master > restart, but the test environment was chaotic so needs investigation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21266) Not running balancer because processing dead regionservers, but empty dead rs list
[ https://issues.apache.org/jira/browse/HBASE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21266: --- Attachment: HBASE-21266-branch-1.patch > Not running balancer because processing dead regionservers, but empty dead rs > list > -- > > Key: HBASE-21266 > URL: https://issues.apache.org/jira/browse/HBASE-21266 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0, 1.4.9 > > Attachments: HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch > > > Found during ITBLL testing. AM in master gets into a state where manual > attempts from the shell to run the balancer always return false and this is > printed in the master log: > 2018-10-03 19:17:14,892 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=21,queue=0,port=8100] master.HMaster: > Not running balancer because processing dead regionserver(s): > Note the empty list. > This errant state did not recover without intervention by way of master > restart, but the test environment was chaotic so needs investigation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (HBASE-21158) Empty qualifier cell is always returned when using QualifierFilter
[
https://issues.apache.org/jira/browse/HBASE-21158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey reopened HBASE-21158:
-
> Empty qualifier cell is always returned when using QualifierFilter
> --
>
> Key: HBASE-21158
> URL: https://issues.apache.org/jira/browse/HBASE-21158
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 1.4.8, 2.1.1, 2.0.3
>
> Attachments: HBASE-21158.branch-1.001.patch,
> HBASE-21158.master.001.patch, HBASE-21158.master.002.patch,
> HBASE-21158.master.003.patch, HBASE-21158.master.004.patch
>
>
> {code:xml}
> hbase(main):002:0> put 'testTable','testrow','f:testcol1','testvalue1'
> 0 row(s) in 0.0040 seconds
> hbase(main):003:0> put 'testTable','testrow','f:','testvalue2'
> 0 row(s) in 0.0070 seconds
> # get row with empty column f:, result is correct.
> hbase(main):004:0> scan 'testTable',{FILTER => "QualifierFilter (=,
> 'binary:')"}
> ROW COLUMN+CELL
>
>
> testrowcolumn=f:,
> timestamp=1536218563581, value=testvalue2
>
> 1 row(s) in 0.0460 seconds
> # get row with column f:testcol1, result is incorrect.
> hbase(main):005:0> scan 'testTable',{FILTER => "QualifierFilter (=,
> 'binary:testcol1')"}
> ROW COLUMN+CELL
>
>
> testrowcolumn=f:,
> timestamp=1536218563581, value=testvalue2
>
> testrowcolumn=f:testcol1,
> timestamp=1536218550827, value=testvalue1
>
> 1 row(s) in 0.0070 seconds
> {code}
> As the above operation, when the row contains empty qualifier column, empty
> qualifier cell is always returned when using QualifierFilter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21266) Not running balancer because processing dead regionservers, but empty dead rs list
[ https://issues.apache.org/jira/browse/HBASE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648338#comment-16648338 ] Andrew Purtell commented on HBASE-21266: If I remove an entry from the processing servers map when registering a new instance of the dead server coming back online, then I'll trip over an assert I added. Back soon. > Not running balancer because processing dead regionservers, but empty dead rs > list > -- > > Key: HBASE-21266 > URL: https://issues.apache.org/jira/browse/HBASE-21266 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0, 1.4.9 > > Attachments: HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch > > > Found during ITBLL testing. AM in master gets into a state where manual > attempts from the shell to run the balancer always return false and this is > printed in the master log: > 2018-10-03 19:17:14,892 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=21,queue=0,port=8100] master.HMaster: > Not running balancer because processing dead regionserver(s): > Note the empty list. > This errant state did not recover without intervention by way of master > restart, but the test environment was chaotic so needs investigation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21166) Creating a CoprocessorHConnection re-retrieves the cluster id from ZK
[ https://issues.apache.org/jira/browse/HBASE-21166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-21166: Component/s: Zookeeper Client > Creating a CoprocessorHConnection re-retrieves the cluster id from ZK > - > > Key: HBASE-21166 > URL: https://issues.apache.org/jira/browse/HBASE-21166 > Project: HBase > Issue Type: Bug > Components: Client, Zookeeper >Affects Versions: 1.5.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl >Priority: Major > Fix For: 1.5.0, 1.3.3, 1.2.8, 1.4.8 > > Attachments: HBASE-21166.branch-1.001.patch > > > CoprocessorHConnections are created for example during a call of > CoprocessorHost$Environent.getTable(...). The region server already know the > cluster id, yet, we're resolving it over and over again. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-15832) memory leak in FSHLog.
[ https://issues.apache.org/jira/browse/HBASE-15832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-15832: Resolution: Duplicate Assignee: (was: Vladimir Rodionov) Status: Resolved (was: Patch Available) fixed by HBASE-21228 > memory leak in FSHLog. > -- > > Key: HBASE-15832 > URL: https://issues.apache.org/jira/browse/HBASE-15832 > Project: HBase > Issue Type: Bug >Affects Versions: 1.1.2 >Reporter: Jeongdae Kim >Priority: Major > Attachments: HBASE-15832-v1.patch, Screenshot-Java - > -home-jeongdae-work-regionserver_jmap_104p_sn5_20160509-sn5_heap.hprof - > Eclipse -1.png > > > FSHLog module uses a map to reuse SyncFuture objects, and assumes that this > map will be used by RPC Handler threads only. but, in some cases, this > assumption is wrong. > for example, if some coprocessors are registered, and these coprocessors uses > CoprocessorHConnection insteadof HConnection, and request some puts/ or > deletes throgh CoprocessorHConnection, all mutations will be handled by > hconnection's batchPool, not RPC Handlers. because hconnection's batchPool is > dynamically growing or shrinking, all new threads in hconnection are put to > the map in FSHLog, and this map will grow continuously. > in attached image file, the map to reuse SyncFuture occupies about 4GB memory > and has (almost all) entries holding hconnection's thread. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21228) Memory leak since AbstractFSWAL caches Thread object and never clean later
[
https://issues.apache.org/jira/browse/HBASE-21228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-21228:
Priority: Critical (was: Major)
> Memory leak since AbstractFSWAL caches Thread object and never clean later
> --
>
> Key: HBASE-21228
> URL: https://issues.apache.org/jira/browse/HBASE-21228
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.1.0, 2.0.2, 1.4.7
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 1.4.8, 2.1.1, 2.0.3
>
> Attachments: HBASE-21228.branch-2.0.001.patch,
> HBASE-21228.branch-2.0.002.patch, HBASE-21228.branch-2.0.003.patch
>
>
> In AbstractFSWAL(FSHLog in branch-1), we have a map caches thread and
> SyncFutures.
> {code:java}
> /**
>* Map of {@link SyncFuture}s keyed by Handler objects. Used so we reuse
> SyncFutures.
>*
>* TODO: Reuse FSWALEntry's rather than create them anew each time as we do
> SyncFutures here.
>*
>* TODO: Add a FSWalEntry and SyncFuture as thread locals on handlers
> rather than have them get
>* them from this Map?
>*/
> private final ConcurrentMap syncFuturesByHandler;
> {code}
> A colleague of mine find a memory leak case caused by this map.
> Every thread who writes WAL will be cached in this map, And no one will clean
> the threads in the map even after the thread is dead.
> In one of our customer's cluster, we noticed that even though there is no
> requests, the heap of the RS is almost full and CMS GC was triggered every
> second.
> We dumped the heap and then found out there were more than 30 thousands
> threads with Terminated state. which are all cached in this map above.
> Everything referenced in these threads were leaked. Most of the threads are:
> 1.PostOpenDeployTasksThread, which will write Open Region mark in WAL
> 2. hconnection-0x1f838e31-shared--pool, which are used to write index short
> circuit(Phoenix), and WAL will be write and sync in these threads.
> 3. Index writer thread(Phoenix), which referenced by
> RegionCoprocessorHost$RegionEnvironment then by HRegion and finally been
> referenced by PostOpenDeployTasksThread.
> We should turn this map into a thread local one, let JVM GC the terminated
> thread for us.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21266) Not running balancer because processing dead regionservers, but empty dead rs list
[ https://issues.apache.org/jira/browse/HBASE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21266: --- Status: Open (was: Patch Available) > Not running balancer because processing dead regionservers, but empty dead rs > list > -- > > Key: HBASE-21266 > URL: https://issues.apache.org/jira/browse/HBASE-21266 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0, 1.4.9 > > Attachments: HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch > > > Found during ITBLL testing. AM in master gets into a state where manual > attempts from the shell to run the balancer always return false and this is > printed in the master log: > 2018-10-03 19:17:14,892 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=21,queue=0,port=8100] master.HMaster: > Not running balancer because processing dead regionserver(s): > Note the empty list. > This errant state did not recover without intervention by way of master > restart, but the test environment was chaotic so needs investigation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21228) Memory leak since AbstractFSWAL caches Thread object and never clean later
[
https://issues.apache.org/jira/browse/HBASE-21228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-21228:
Component/s: wal
> Memory leak since AbstractFSWAL caches Thread object and never clean later
> --
>
> Key: HBASE-21228
> URL: https://issues.apache.org/jira/browse/HBASE-21228
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 2.1.0, 2.0.2, 1.4.7
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 1.4.8, 2.1.1, 2.0.3
>
> Attachments: HBASE-21228.branch-2.0.001.patch,
> HBASE-21228.branch-2.0.002.patch, HBASE-21228.branch-2.0.003.patch
>
>
> In AbstractFSWAL(FSHLog in branch-1), we have a map caches thread and
> SyncFutures.
> {code:java}
> /**
>* Map of {@link SyncFuture}s keyed by Handler objects. Used so we reuse
> SyncFutures.
>*
>* TODO: Reuse FSWALEntry's rather than create them anew each time as we do
> SyncFutures here.
>*
>* TODO: Add a FSWalEntry and SyncFuture as thread locals on handlers
> rather than have them get
>* them from this Map?
>*/
> private final ConcurrentMap syncFuturesByHandler;
> {code}
> A colleague of mine find a memory leak case caused by this map.
> Every thread who writes WAL will be cached in this map, And no one will clean
> the threads in the map even after the thread is dead.
> In one of our customer's cluster, we noticed that even though there is no
> requests, the heap of the RS is almost full and CMS GC was triggered every
> second.
> We dumped the heap and then found out there were more than 30 thousands
> threads with Terminated state. which are all cached in this map above.
> Everything referenced in these threads were leaked. Most of the threads are:
> 1.PostOpenDeployTasksThread, which will write Open Region mark in WAL
> 2. hconnection-0x1f838e31-shared--pool, which are used to write index short
> circuit(Phoenix), and WAL will be write and sync in these threads.
> 3. Index writer thread(Phoenix), which referenced by
> RegionCoprocessorHost$RegionEnvironment then by HRegion and finally been
> referenced by PostOpenDeployTasksThread.
> We should turn this map into a thread local one, let JVM GC the terminated
> thread for us.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21179) Fix the number of actions in responseTooSlow log
[
https://issues.apache.org/jira/browse/HBASE-21179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-21179:
Component/s: logging
> Fix the number of actions in responseTooSlow log
>
>
> Key: HBASE-21179
> URL: https://issues.apache.org/jira/browse/HBASE-21179
> Project: HBase
> Issue Type: Bug
> Components: logging, rpc
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 1.4.8, 2.1.1, 2.0.3
>
> Attachments: HBASE-21179.branch-1.001.patch,
> HBASE-21179.master.001.patch, HBASE-21179.master.002.patch
>
>
> {panel:title=responseTooSlow|borderStyle=dashed|borderColor=#ccc|titleBGColor=#F7D6C1|bgColor=#CE}
> 2018-09-10 16:13:53,022 WARN
> [B.DefaultRpcServer.handler=209,queue=29,port=60020] ipc.RpcServer:
> (responseTooSlow):
> {"processingtimems":321262,"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","client":"127.0.0.1:56149","param":"region=
>
> tsdb,\\x00\\x00.[\\x89\\x1F\\xB0\\x00\\x00\\x01\\x00\\x01Y\\x00\\x00\\x02\\x00\\x00\\x04,1536133210446.7c752de470bd5558a001117b123a5db5.,
> {color:red}for 1 actions and 1st row{color}
> key=\\x00\\x00.[\\x96\\x16p","starttimems":1536566911759,"queuetimems":0,"class":"HRegionServer","responsesize":2,"method":"Multi"}
> {panel}
> The responseTooSlow log is printed when the processing time of a request
> exceeds the specified threshold. The number of actions and the contents of
> the first rowkey in the request will be included in the log.
> However, the number of actions is inaccurate, and it is actually the number
> of regions that the request needs to visit.
> Just like the logs above, users may be mistaken for using 321262ms to process
> an action, which is incredible, so we need to fix it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21242) [amv2] Miscellaneous minor log and assign procedure create improvements
[
https://issues.apache.org/jira/browse/HBASE-21242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648328#comment-16648328
]
stack commented on HBASE-21242:
---
branch-2.1 and branch-2.0. I saw that failure in branch-2.0 and fixed it.
Wondered about 2.1. Let me fix it there too. Thanks [~mdrob]
> [amv2] Miscellaneous minor log and assign procedure create improvements
> ---
>
> Key: HBASE-21242
> URL: https://issues.apache.org/jira/browse/HBASE-21242
> Project: HBase
> Issue Type: Bug
> Components: amv2, Operability
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21242.branch-2.0.001.patch,
> HBASE-21242.branch-2.0.001.patch, HBASE-21242.branch-2.0.002.patch,
> HBASE-21242.branch-2.001.patch, HBASE-21242.branch-2.001.patch,
> HBASE-21242.branch-2.002.patch, HBASE-21242.branch-2.1.001.patch,
> HBASE-21242.branch-2.1.001.patch, HBASE-21242.branch-2.1.001.patch,
> HBASE-21242.branch-2.1.002.patch
>
>
> Some minor fixups:
> {code}
> For RIT Duration, do better than print ms/seconds. Remove redundant UI
> column dedicated to duration when we log it in the status field too.
> Make bypass log at INFO level -- when DEBUG we can miss important
> fixup detail like why we failed.
> Make it so on complete of subprocedure, we note count of outstanding
> siblings so we have a clue how much further the parent has to go before
> it is done (Helpful when hundreds of servers doing SCP).
> Have the SCP run the AP preflight check before creating an AP; saves
> creation of hundreds of thousands of APs during fixup of this big cluster
> of mine.
> Don't log tablename three times when reporting remote call failed.
> If lock is held already, note who has it. Also log after we get lock
> or if we have to wait rather than log on entrance though we may
> later have to wait (or we may have just picked up the lock).
> {code}
> Posting patch in a sec but let me try it on cluster too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21242) [amv2] Miscellaneous minor log and assign procedure create improvements
[
https://issues.apache.org/jira/browse/HBASE-21242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648327#comment-16648327
]
Mike Drob commented on HBASE-21242:
---
Stack, where did you commit this so far? It seems to be breaking
TestHRegionInfo (see
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.1/Flaky_20Test_20Report/)
We try to compare two stringifications, but they don't match because the "0s
ago" becomes "0.01s ago" when comparing two Objects.
I can file a follow on for it to use EdgeManager?
> [amv2] Miscellaneous minor log and assign procedure create improvements
> ---
>
> Key: HBASE-21242
> URL: https://issues.apache.org/jira/browse/HBASE-21242
> Project: HBase
> Issue Type: Bug
> Components: amv2, Operability
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Fix For: 3.0.0, 2.2.0, 2.1.1, 2.0.3
>
> Attachments: HBASE-21242.branch-2.0.001.patch,
> HBASE-21242.branch-2.0.001.patch, HBASE-21242.branch-2.0.002.patch,
> HBASE-21242.branch-2.001.patch, HBASE-21242.branch-2.001.patch,
> HBASE-21242.branch-2.002.patch, HBASE-21242.branch-2.1.001.patch,
> HBASE-21242.branch-2.1.001.patch, HBASE-21242.branch-2.1.001.patch,
> HBASE-21242.branch-2.1.002.patch
>
>
> Some minor fixups:
> {code}
> For RIT Duration, do better than print ms/seconds. Remove redundant UI
> column dedicated to duration when we log it in the status field too.
> Make bypass log at INFO level -- when DEBUG we can miss important
> fixup detail like why we failed.
> Make it so on complete of subprocedure, we note count of outstanding
> siblings so we have a clue how much further the parent has to go before
> it is done (Helpful when hundreds of servers doing SCP).
> Have the SCP run the AP preflight check before creating an AP; saves
> creation of hundreds of thousands of APs during fixup of this big cluster
> of mine.
> Don't log tablename three times when reporting remote call failed.
> If lock is held already, note who has it. Also log after we get lock
> or if we have to wait rather than log on entrance though we may
> later have to wait (or we may have just picked up the lock).
> {code}
> Posting patch in a sec but let me try it on cluster too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21158) Empty qualifier cell is always returned when using QualifierFilter
[
https://issues.apache.org/jira/browse/HBASE-21158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-21158:
Priority: Critical (was: Major)
> Empty qualifier cell is always returned when using QualifierFilter
> --
>
> Key: HBASE-21158
> URL: https://issues.apache.org/jira/browse/HBASE-21158
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Critical
> Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 1.4.8, 2.1.1, 2.0.3
>
> Attachments: HBASE-21158.branch-1.001.patch,
> HBASE-21158.master.001.patch, HBASE-21158.master.002.patch,
> HBASE-21158.master.003.patch, HBASE-21158.master.004.patch
>
>
> {code:xml}
> hbase(main):002:0> put 'testTable','testrow','f:testcol1','testvalue1'
> 0 row(s) in 0.0040 seconds
> hbase(main):003:0> put 'testTable','testrow','f:','testvalue2'
> 0 row(s) in 0.0070 seconds
> # get row with empty column f:, result is correct.
> hbase(main):004:0> scan 'testTable',{FILTER => "QualifierFilter (=,
> 'binary:')"}
> ROW COLUMN+CELL
>
>
> testrowcolumn=f:,
> timestamp=1536218563581, value=testvalue2
>
> 1 row(s) in 0.0460 seconds
> # get row with column f:testcol1, result is incorrect.
> hbase(main):005:0> scan 'testTable',{FILTER => "QualifierFilter (=,
> 'binary:testcol1')"}
> ROW COLUMN+CELL
>
>
> testrowcolumn=f:,
> timestamp=1536218563581, value=testvalue2
>
> testrowcolumn=f:testcol1,
> timestamp=1536218550827, value=testvalue1
>
> 1 row(s) in 0.0070 seconds
> {code}
> As the above operation, when the row contains empty qualifier column, empty
> qualifier cell is always returned when using QualifierFilter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20307) LoadTestTool prints too much zookeeper logging
[ https://issues.apache.org/jira/browse/HBASE-20307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-20307: Priority: Minor (was: Major) > LoadTestTool prints too much zookeeper logging > -- > > Key: HBASE-20307 > URL: https://issues.apache.org/jira/browse/HBASE-20307 > Project: HBase > Issue Type: Improvement > Components: tooling >Reporter: Mike Drob >Assignee: Colin Garcia >Priority: Minor > Labels: beginner > Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 1.4.8, 2.1.1, 2.0.3 > > Attachments: HBASE-20307.000.patch, HBASE-20307.001.patch > > > When running ltt there is a ton of ZK related cruft that I probably don't > care about. Hide it behind -verbose flag or point people at log4j > configuration but don't print it by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21114) website should have a copy of 2.1 release docs
[
https://issues.apache.org/jira/browse/HBASE-21114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648319#comment-16648319
]
Hadoop QA commented on HBASE-21114:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 15m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 14m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21114 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12943691/HBASE-21114.patch |
| Optional Tests | dupname asflicense mvnsite xml |
| uname | Linux 99a265317eca 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / e736168567 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Max. process+thread count | 87 (vs. ulimit of 1) |
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/14673/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> website should have a copy of 2.1 release docs
> --
>
> Key: HBASE-21114
> URL: https://issues.apache.org/jira/browse/HBASE-21114
> Project: HBase
> Issue Type: Task
> Components: community, documentation
>Affects Versions: 2.1.0
>Reporter: Sean Busbey
>Assignee: Mike Drob
>Priority: Major
> Fix For: 2.1.1
>
> Attachments: HBASE-21114.patch
>
>
> in the "Documentation and API" menu we have entries for 2.0 and 1.2. should
> also add in 2.1.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20307) LoadTestTool prints too much zookeeper logging
[ https://issues.apache.org/jira/browse/HBASE-20307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-20307: Issue Type: Improvement (was: Bug) > LoadTestTool prints too much zookeeper logging > -- > > Key: HBASE-20307 > URL: https://issues.apache.org/jira/browse/HBASE-20307 > Project: HBase > Issue Type: Improvement > Components: tooling >Reporter: Mike Drob >Assignee: Colin Garcia >Priority: Major > Labels: beginner > Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 1.4.8, 2.1.1, 2.0.3 > > Attachments: HBASE-20307.000.patch, HBASE-20307.001.patch > > > When running ltt there is a ton of ZK related cruft that I probably don't > care about. Hide it behind -verbose flag or point people at log4j > configuration but don't print it by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21304) HBaseBulkGetExample throws NullPointerException
[
https://issues.apache.org/jira/browse/HBASE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-21304:
Priority: Minor (was: Major)
> HBaseBulkGetExample throws NullPointerException
> ---
>
> Key: HBASE-21304
> URL: https://issues.apache.org/jira/browse/HBASE-21304
> Project: HBase
> Issue Type: Bug
> Components: spark
>Affects Versions: 3.0.0
>Reporter: Carlos A. Morillo
>Priority: Minor
> Labels: beginner
>
> Running the HBaseContext Examples available at
> [https://github.com/apache/hbase/tree/master/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/example/hbasecontext]
>
> If you run them in sequence, let's say you run first HBaseBulkPutExample
> [https://github.com/apache/hbase/blob/master/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/example/rdd/HBaseBulkPutExample.scala]
> and immediately after you run HBaseBulkGetExample
> [https://github.com/apache/hbase/blob/master/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/example/rdd/HBaseBulkGetExample.scala]
> You get a NullPointerException.
> In the API
> [https://github.com/apache/hbase/blob/master/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/example/hbasecontext/HBaseBulkGetExample.scala]
> we have:
> h4. defhbaseBulkGet(hc:
> [HBaseContext|http://hbase.apache.org/hbase-spark/scaladocs/org/apache/hadoop/hbase/spark/HBaseContext.html],
> tableName: TableName, batchSize: Int, f: (T) ⇒ Get):
> RDD[(ImmutableBytesWritable, Result)]
> Implicit method that gives easy access to HBaseContext's bulk get. This will
> return a new RDD. Think about it as a RDD map function. In that every RDD
> value will get a new value out of HBase. That new value will populate the
> newly generated RDD.
> hc
> The hbaseContext object to identify which HBase cluster connection to use
> tableName
> The tableName that the put will be sent to
> batchSize
> How many gets to execute in a single batch
> f
> The function that will turn the RDD values in HBase Get objects
> returns
> A resulting RDD with type R objects
> So it seems the function f passed to should be modified as an Scala partial
> function to handle the case when the Result is null.
> One possible fix would be {color:#00}to call in an{color}{color:#00}
> if
> {{[Result.isEmpty()|https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Result.html#isEmpty--]
> to make sure it isn't empty.}}{color}
> {color:#00}{{The API for Result.listCells expressly says it can return
> null if there are no results.}}{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21304) HBaseBulkGetExample throws NullPointerException
[
https://issues.apache.org/jira/browse/HBASE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-21304:
Affects Version/s: 3.0.0
> HBaseBulkGetExample throws NullPointerException
> ---
>
> Key: HBASE-21304
> URL: https://issues.apache.org/jira/browse/HBASE-21304
> Project: HBase
> Issue Type: Bug
> Components: spark
>Affects Versions: 3.0.0
>Reporter: Carlos A. Morillo
>Priority: Major
> Labels: beginner
>
> Running the HBaseContext Examples available at
> [https://github.com/apache/hbase/tree/master/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/example/hbasecontext]
>
> If you run them in sequence, let's say you run first HBaseBulkPutExample
> [https://github.com/apache/hbase/blob/master/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/example/rdd/HBaseBulkPutExample.scala]
> and immediately after you run HBaseBulkGetExample
> [https://github.com/apache/hbase/blob/master/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/example/rdd/HBaseBulkGetExample.scala]
> You get a NullPointerException.
> In the API
> [https://github.com/apache/hbase/blob/master/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/example/hbasecontext/HBaseBulkGetExample.scala]
> we have:
> h4. defhbaseBulkGet(hc:
> [HBaseContext|http://hbase.apache.org/hbase-spark/scaladocs/org/apache/hadoop/hbase/spark/HBaseContext.html],
> tableName: TableName, batchSize: Int, f: (T) ⇒ Get):
> RDD[(ImmutableBytesWritable, Result)]
> Implicit method that gives easy access to HBaseContext's bulk get. This will
> return a new RDD. Think about it as a RDD map function. In that every RDD
> value will get a new value out of HBase. That new value will populate the
> newly generated RDD.
> hc
> The hbaseContext object to identify which HBase cluster connection to use
> tableName
> The tableName that the put will be sent to
> batchSize
> How many gets to execute in a single batch
> f
> The function that will turn the RDD values in HBase Get objects
> returns
> A resulting RDD with type R objects
> So it seems the function f passed to should be modified as an Scala partial
> function to handle the case when the Result is null.
> One possible fix would be {color:#00}to call in an{color}{color:#00}
> if
> {{[Result.isEmpty()|https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Result.html#isEmpty--]
> to make sure it isn't empty.}}{color}
> {color:#00}{{The API for Result.listCells expressly says it can return
> null if there are no results.}}{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21304) HBaseBulkGetExample throws NullPointerException
[
https://issues.apache.org/jira/browse/HBASE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-21304:
Labels: beginner (was: )
> HBaseBulkGetExample throws NullPointerException
> ---
>
> Key: HBASE-21304
> URL: https://issues.apache.org/jira/browse/HBASE-21304
> Project: HBase
> Issue Type: Bug
> Components: spark
>Affects Versions: 3.0.0
>Reporter: Carlos A. Morillo
>Priority: Minor
> Labels: beginner
>
> Running the HBaseContext Examples available at
> [https://github.com/apache/hbase/tree/master/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/example/hbasecontext]
>
> If you run them in sequence, let's say you run first HBaseBulkPutExample
> [https://github.com/apache/hbase/blob/master/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/example/rdd/HBaseBulkPutExample.scala]
> and immediately after you run HBaseBulkGetExample
> [https://github.com/apache/hbase/blob/master/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/example/rdd/HBaseBulkGetExample.scala]
> You get a NullPointerException.
> In the API
> [https://github.com/apache/hbase/blob/master/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/example/hbasecontext/HBaseBulkGetExample.scala]
> we have:
> h4. defhbaseBulkGet(hc:
> [HBaseContext|http://hbase.apache.org/hbase-spark/scaladocs/org/apache/hadoop/hbase/spark/HBaseContext.html],
> tableName: TableName, batchSize: Int, f: (T) ⇒ Get):
> RDD[(ImmutableBytesWritable, Result)]
> Implicit method that gives easy access to HBaseContext's bulk get. This will
> return a new RDD. Think about it as a RDD map function. In that every RDD
> value will get a new value out of HBase. That new value will populate the
> newly generated RDD.
> hc
> The hbaseContext object to identify which HBase cluster connection to use
> tableName
> The tableName that the put will be sent to
> batchSize
> How many gets to execute in a single batch
> f
> The function that will turn the RDD values in HBase Get objects
> returns
> A resulting RDD with type R objects
> So it seems the function f passed to should be modified as an Scala partial
> function to handle the case when the Result is null.
> One possible fix would be {color:#00}to call in an{color}{color:#00}
> if
> {{[Result.isEmpty()|https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Result.html#isEmpty--]
> to make sure it isn't empty.}}{color}
> {color:#00}{{The API for Result.listCells expressly says it can return
> null if there are no results.}}{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21168) BloomFilterUtil uses hardcoded randomness
[ https://issues.apache.org/jira/browse/HBASE-21168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-21168: Priority: Trivial (was: Minor) > BloomFilterUtil uses hardcoded randomness > - > > Key: HBASE-21168 > URL: https://issues.apache.org/jira/browse/HBASE-21168 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Trivial > Fix For: 3.0.0, 1.5.0, 1.3.3, 1.2.8, 2.2.0, 1.4.8, 2.1.1 > > Attachments: HBASE-21168.branch-1.002.patch, > HBASE-21168.master.001.patch, HBASE-21168.master.002.patch > > > This was flagged by a Fortify scan and while it doesn't appear to be a real > issue, it's pretty easy to take care of anyway. > The hard coded rand can be moved to the test class that actually needs it to > make the static analysis happy. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21185) WALPrettyPrinter: Additional useful info to be printed by wal printer tool, for debugability purposes
[ https://issues.apache.org/jira/browse/HBASE-21185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-21185: Priority: Minor (was: Trivial) > WALPrettyPrinter: Additional useful info to be printed by wal printer tool, > for debugability purposes > - > > Key: HBASE-21185 > URL: https://issues.apache.org/jira/browse/HBASE-21185 > Project: HBase > Issue Type: Improvement > Components: Operability >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Fix For: 3.0.0, 1.3.3, 1.2.8, 2.2.0, 2.1.1, 2.0.3, 1.4.9 > > Attachments: HBASE-21185.branch-1.4.001.patch, > HBASE-21185.master.001.patch, HBASE-21185.master.002.patch, > HBASE-21185.master.003.patch > > > *WALPrettyPrinter* is very useful for troubleshooting wal issues, such as > faulty replication sinks. An useful information one might want to track is > the size of a single WAL entry edit, as well as size for each edit cell. Am > proposing a patch that adds calculations for these two, as well an option to > seek straight to a given position on the WAL file being analysed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21304) HBaseBulkGetExample throws NullPointerException
Carlos A. Morillo created HBASE-21304:
-
Summary: HBaseBulkGetExample throws NullPointerException
Key: HBASE-21304
URL: https://issues.apache.org/jira/browse/HBASE-21304
Project: HBase
Issue Type: Bug
Components: spark
Reporter: Carlos A. Morillo
Running the HBaseContext Examples available at
[https://github.com/apache/hbase/tree/master/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/example/hbasecontext]
If you run them in sequence, let's say you run first HBaseBulkPutExample
[https://github.com/apache/hbase/blob/master/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/example/rdd/HBaseBulkPutExample.scala]
and immediately after you run HBaseBulkGetExample
[https://github.com/apache/hbase/blob/master/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/example/rdd/HBaseBulkGetExample.scala]
You get a NullPointerException.
In the API
[https://github.com/apache/hbase/blob/master/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/example/hbasecontext/HBaseBulkGetExample.scala]
we have:
h4. defhbaseBulkGet(hc:
[HBaseContext|http://hbase.apache.org/hbase-spark/scaladocs/org/apache/hadoop/hbase/spark/HBaseContext.html],
tableName: TableName, batchSize: Int, f: (T) ⇒ Get):
RDD[(ImmutableBytesWritable, Result)]
Implicit method that gives easy access to HBaseContext's bulk get. This will
return a new RDD. Think about it as a RDD map function. In that every RDD value
will get a new value out of HBase. That new value will populate the newly
generated RDD.
hc
The hbaseContext object to identify which HBase cluster connection to use
tableName
The tableName that the put will be sent to
batchSize
How many gets to execute in a single batch
f
The function that will turn the RDD values in HBase Get objects
returns
A resulting RDD with type R objects
So it seems the function f passed to should be modified as an Scala partial
function to handle the case when the Result is null.
One possible fix would be {color:#00}to call in an{color}{color:#00} if
{{[Result.isEmpty()|https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Result.html#isEmpty--]
to make sure it isn't empty.}}{color}
{color:#00}{{The API for Result.listCells expressly says it can return null
if there are no results.}}{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21302) Release 1.2.8
[ https://issues.apache.org/jira/browse/HBASE-21302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648301#comment-16648301 ] Sean Busbey commented on HBASE-21302: - I have a locally staged RC0 tag based on [ref 56d38a088|https://github.com/apache/hbase/commit/56d38a088319413dc3a70e90b1b8e655c6a22340] and am building RC artifacts now. > Release 1.2.8 > - > > Key: HBASE-21302 > URL: https://issues.apache.org/jira/browse/HBASE-21302 > Project: HBase > Issue Type: Task > Components: community >Affects Versions: 1.2.8 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 1.2.8 > > > 1.4.8 is out, time to make 1.2.8. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21114) website should have a copy of 2.1 release docs
[ https://issues.apache.org/jira/browse/HBASE-21114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648299#comment-16648299 ] Sean Busbey commented on HBASE-21114: - +1 on both. > website should have a copy of 2.1 release docs > -- > > Key: HBASE-21114 > URL: https://issues.apache.org/jira/browse/HBASE-21114 > Project: HBase > Issue Type: Task > Components: community, documentation >Affects Versions: 2.1.0 >Reporter: Sean Busbey >Assignee: Mike Drob >Priority: Major > Fix For: 2.1.1 > > Attachments: HBASE-21114.patch > > > in the "Documentation and API" menu we have entries for 2.0 and 1.2. should > also add in 2.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21266) Not running balancer because processing dead regionservers, but empty dead rs list
[ https://issues.apache.org/jira/browse/HBASE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648268#comment-16648268 ] Andrew Purtell edited comment on HBASE-21266 at 10/12/18 6:24 PM: -- Updated patch with suggestion from [~stack]*, also fixed something dumb I did with logging * - Logging only, no change to API. Ok? was (Author: apurtell): Updated patch with suggestion from [~stack], also fixed something dumb I did with logging > Not running balancer because processing dead regionservers, but empty dead rs > list > -- > > Key: HBASE-21266 > URL: https://issues.apache.org/jira/browse/HBASE-21266 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0, 1.4.9 > > Attachments: HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch > > > Found during ITBLL testing. AM in master gets into a state where manual > attempts from the shell to run the balancer always return false and this is > printed in the master log: > 2018-10-03 19:17:14,892 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=21,queue=0,port=8100] master.HMaster: > Not running balancer because processing dead regionserver(s): > Note the empty list. > This errant state did not recover without intervention by way of master > restart, but the test environment was chaotic so needs investigation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21266) Not running balancer because processing dead regionservers, but empty dead rs list
[ https://issues.apache.org/jira/browse/HBASE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648268#comment-16648268 ] Andrew Purtell commented on HBASE-21266: Updated patch with suggestion from [~stack], also fixed something dumb I did with logging > Not running balancer because processing dead regionservers, but empty dead rs > list > -- > > Key: HBASE-21266 > URL: https://issues.apache.org/jira/browse/HBASE-21266 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.8 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0, 1.4.9 > > Attachments: HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch, > HBASE-21266-branch-1.patch, HBASE-21266-branch-1.patch > > > Found during ITBLL testing. AM in master gets into a state where manual > attempts from the shell to run the balancer always return false and this is > printed in the master log: > 2018-10-03 19:17:14,892 DEBUG > [RpcServer.default.FPBQ.Fifo.handler=21,queue=0,port=8100] master.HMaster: > Not running balancer because processing dead regionserver(s): > Note the empty list. > This errant state did not recover without intervention by way of master > restart, but the test environment was chaotic so needs investigation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
