[
https://issues.apache.org/jira/browse/HBASE-20137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16388571#comment-16388571
]
stack commented on HBASE-20137:
-------------------------------
bq. when is target server null? if the remote peer is down we should still know
who we were trying to reach, and would be good to report why we failed. logging
that we were trying to talk to a null server is likely useless for operators
because they have nowhere to look to corroborate or do RCA on why that server
went down.
Understood. When we log, we generally log all procedure context including the
server we are trying to talk to, always, intentionally, so we can trace through
all steps.
If target server is null at this stage something is seriously wrong. Tracing
across the life of the procedure will hopefully turn up where we went awry (Its
null in test where mock loads of context, see above TestAssignProcedure failure
for full context, where non-null would be awkward to insert -- it used to just
print as 'null'... could have just done same. Gave it little mind because this
is actually redundant info).
Including target server in the FailedRemoteDispatchException message is
redundant (the old message which included the procedure context again was super
redundant and confused). Should probably just do w/o target server as message
in this FailedRemoteDispatchException filler exception.
This FailedRemoteDispatchException is an oddball. Its a marker. It denotes the
case where we failed queue of the unassign close rpc. The only reason we'd fail
at this point is that there is no entry for the server we are trying to contact
-- its crashed most likely (I tried to backfill 'cause' but none to insert.
Refactoring inserting an exception would probably make sense one day... but
would take work to manage the ripple given we are suspended at this point and
are trying to exit the execution promptly... ).
bq. Is adding it here an optimization for MTTR but not required for
correctness?
Right. Something to consider. Old comment IIRC. Would love to spend time on
AMv2. Currently am in minimal, evidence-based changes mode.
bq ....where region server went down and master kept failing to do unassign
That sounds related. AMv2 depends on ServerCrashProcedure running to clean up
ongoing assigns and unassigns that will never succeed (because server went
away... there is NO timeout as yet). The fix here is for a corner case where
the SCP doesn't get scheduled... because we are shutting down. We had no
handling for this scenario.
I looked at the checkstyle and punted on it because it meant changing a bunch
of lines.... let me do it in follow-up.
Appreciate the review. Thanks [~mdrob]
> TestRSGroups is flakey
> ----------------------
>
> Key: HBASE-20137
> URL: https://issues.apache.org/jira/browse/HBASE-20137
> Project: HBase
> Issue Type: Bug
> Components: flakey
> Affects Versions: 2.0.0-beta-2
> Reporter: stack
> Assignee: stack
> Priority: Major
> Fix For: 2.0.0
>
> Attachments: HBASE-20137.branch-2.001.patch,
> HBASE-20137.branch-2.002.patch, HBASE-20137.branch-2.003.patch,
> HBASE-20137.branch-2.003.patch
>
>
> It was the single test that failed the hbase-2 nightlies in #440 at the
> hadoop2 stage.
> The failure manifests as a timeout. It actually has an interesting cause
> calling into question some of the clauses in
> UnassignProcedure#remoteCallFailed.
> We are running a disabletable concurrent with a shutdown. pid=309 is the
> disable. pid=311 is the interesting one. The below is a little hard to read
> -- the exception 'message' is the the current procedure as a String... hard
> to parse, fixing -- but we are trying to unassign as part of a the
> disabletable. Our RPC fails because the server we are trying to rpc too is
> currently being processed as crashed (pid=308 is a servercrashprocedure for
> this server). As part of the processing of the failed RPC we will expire the
> server -- if we can't RPC to it, it must be gone. The current procedure is
> then suspended until it gets woken up by the servercrashprocedure triggered
> by the expire.... only in this case we are shutting down so the expire is
> ignored... The current procedure is left in its suspend state. This prevents
> the Master going down. So we time out.
> 2018-03-05 11:29:22,507 INFO [PEWorker-13]
> assignment.RegionTransitionProcedure(213): Dispatch pid=311, ppid=309,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=Group_ns:testKillRS, region=de7534c208a06502537cd95c248b3043,
> server=1cfd208ff882,40584,1520249102524; rit=CLOSING,
> location=1cfd208ff882,40584,1520249102524
> 2018-03-05 11:29:22,508 WARN [PEWorker-13]
> assignment.RegionTransitionProcedure(187): Remote call failed pid=311,
> ppid=309, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=Group_ns:testKillRS, region=de7534c208a06502537cd95c248b3043,
> server=1cfd208ff882,40584,1520249102524; rit=CLOSING,
> location=1cfd208ff882,40584,1520249102524; exception=pid=311, ppid=309,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=Group_ns:testKillRS, region=de7534c208a06502537cd95c248b3043,
> server=1cfd208ff882,40584,1520249102524 to 1cfd208ff882,40584,1520249102524
> 2018-03-05 11:29:22,508 WARN [PEWorker-13]
> assignment.UnassignProcedure(276): Expiring server pid=311, ppid=309,
> state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure
> table=Group_ns:testKillRS, region=de7534c208a06502537cd95c248b3043,
> server=1cfd208ff882,40584,1520249102524; rit=CLOSING,
> location=1cfd208ff882,40584,1520249102524,
> exception=org.apache.hadoop.hbase.master.assignment.FailedRemoteDispatchException:
> pid=311, ppid=309, state=RUNNABLE:REGION_TRANSITION_DISPATCH;
> UnassignProcedure table=Group_ns:testKillRS,
> region=de7534c208a06502537cd95c248b3043,
> server=1cfd208ff882,40584,1520249102524 to 1cfd208ff882,40584,1520249102524
> 2018-03-05 11:29:22,508 WARN [PEWorker-13] master.ServerManager(580):
> Expiration of 1cfd208ff882,40584,1520249102524 but server shutdown already in
> progress
> I need to cater for case where the expire server is rejected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
