[
https://issues.apache.org/jira/browse/HBASE-29041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nirdosh Kumar Yadav updated HBASE-29041:
----------------------------------------
Description:
In HBase cluster we have encountered a scenario where regionserver server crash
procedure(SCP) waited for more than 3 Hours. Incident was triggered due to
temporary network unavailability in hbase cluster. On Debugging found out SCP
was stuck due to child {{SplitWALProcedure}} which was waiting for completion
of SpliWalRemote procedure by regionserver worker. SplitWALRemote procedure
while running encountered{{{{}} an unknown exception. In logs we can see
*"hdfs*{}}}{*}{{{}.{}}}{{{}DataStreamer - {{}}}}{{{}No ack
}}\{{{}receive{}}}{{{}d{}}}"{*} error while regionserver connecting to Data
Node. After this error thread was stuck or died as there was no related logs
exists{{{}. There were inconsistent regions reported during this period. All
procedure were restarted and completed after Active HMaster service was
bounced. {}}}
Related logs:
{quote}{color:#000000}[HMASTER-4]{color}
{{{}2024{}}}{{{}-{}}}{{{}12{}}}{{{}-{}}}{{{}05{}}}{{
}}{{{}14{}}}{{{}:{}}}{{{}55{}}}{{{}:{}}}{{{}11{}}}{{{},{}}}{{{}264{}}}{{
}}{{INFO}}{{{} [{}}}{{{}PEWorker{}}}{{{}-{}}}{{{}41{}}}{{{}]
{}}}{{{}procedure2{}}}{{{}.{}}}{{{}ProcedureExecutor{}}}{{ }}{{-}}{{
}}{{Initialized}}{{
}}{{{}subprocedures{}}}{{{}=[{{}}}{{{}pid{}}}{{{}={}}}{{{}6003288{}}}{{{},
{}}}{{{}ppid{}}}{{{}={}}}{{{}6002575{}}}{{{},
{}}}{{{}state{}}}{{{}={}}}{{{}RUNNABLE{}}}{{{};
{}}}{{{}SplitWALRemoteProcedure{}}}{{{}{}}}{{{}regionserver{}}}{{{}-{}}}{{{}53{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}hbase33a{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}monitoring{}}}{{{}.{}}}{{{}aws{}}}{{{}-{}}}{{{}esvc1{}}}{{{}-{}}}{{{}useast2{}}}{{{}.{}}}{{{}aws{}}}{{{}.{}}}{{{}sfdc{}}}{{{}.{}}}{{{}is{}}}{{{}%2C{}}}{{{}60020{}}}{{{}%2C{}}}{{{}1730886070174{}}}{{{}.{}}}{{{}1733410178680{}}}{{{},
{}}}{{{}worker{}}}{{{}={}}}{{{}regionserver{}}}{{{}-{}}}{{{}15{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}hbase33a{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}monitoring{}}}{{{}.{}}}{{{}aws{}}}{{{}-{}}}{{{}esvc1{}}}{{{}-{}}}{{{}useast2{}}}{{{}.{}}}{{{}aws{}}}{{{}.{}}}{{{}sfdc{}}}{{{}.{}}}{{{}is{}}}{{{},{}}}{{{}60020{}}}{{{},{}}}{{{}1730878238028{}}}{{{}}]{}}}
{color:#000000}[RS-15]{color}
{{{}2024{}}}{{{}-{}}}{{{}12{}}}{{{}-{}}}{{{}05{}}}{{
}}{{{}14{}}}{{{}:{}}}{{{}55{}}}{{{}:{}}}{{{}11{}}}{{{},{}}}{{{}461{}}}{{
}}{{DEBUG}}{{{}
[{}}}{{{}iority{}}}{{{}.{}}}{{{}RWQ{}}}{{{}.{}}}{{{}Fifo{}}}{{{}.{}}}{{{}read{}}}{{{}.{}}}{{{}handler{}}}{{{}={}}}{{{}83{}}}{{{},{}}}{{{}queue{}}}{{{}={}}}{{{}2{}}}{{{},{}}}{{{}port{}}}{{{}={}}}{{{}60020{}}}{{{}]
{}}}{{{}regionserver{}}}{{{}.{}}}{{{}RSRpcServices{}}}{{ }}{{-}}{{
}}{{Executing}}{{ }}{{remote}}{{ }}{{procedure}}{{
}}{{{}class{}}}{{{}{}}}{{{}org{}}}{{{}.{}}}{{{}apache{}}}{{{}.{}}}{{{}hadoop{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}SplitWALCallable{}}}{{{},
{}}}{{{}pid{}}}{{{}={}}}{{{}6003288{}}}
{color:#000000}[RS-15]{color}
{{{}2024{}}}{{{}-{}}}{{{}12{}}}{{{}-{}}}{{{}05{}}}{{
}}{{{}14{}}}{{{}:{}}}{{{}55{}}}{{{}:{}}}{{{}54{}}}{{{},{}}}{{{}689{}}}{{
}}{{ERROR}}{{{}
[{}}}{{{}split{}}}{{{}-{}}}{{{}log{}}}{{{}-{}}}{{{}closeStream{}}}{{{}-{}}}{{{}pool{}}}{{{}-{}}}{{{}0{}}}{{{}]
{}}}{{{}hdfs{}}}{{{}.{}}}{{{}DataStreamer{}}}{{ }}{{-}}{{ }}{{No}}{{
}}{{ack}}{{ }}{{{}received{}}}{{{}, {}}}{{took}}{{ }}{{25002ms}}{{{}
({}}}{{{}threshold{}}}{{{}={}}}{{{}25000ms{}}}{{{}){}}}{{{}.{}}}{{ }}{{File}}{{
}}{{being}}{{ }}{{{}written{}}}{{{}:
/{}}}{{{}hbase{}}}{{{}/{}}}{{{}data{}}}{{{}/{}}}{{{}default{}}}{{{}/{}}}{{{}tsdb{}}}{{{}/{}}}{{{}c997c5f8dd36481dcd3ebb9b79a35b51{}}}{{{}/{}}}{{{}recovered{}}}{{{}.{}}}{{{}edits{}}}{{{}/{}}}{{{}0000000000539451088{}}}{{{}-{}}}{{{}regionserver{}}}{{{}-{}}}{{{}53{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}hbase33a{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}monitoring{}}}{{{}.{}}}{{{}aws{}}}{{{}-{}}}{{{}esvc1{}}}{{{}-{}}}{{{}useast2{}}}{{{}.{}}}{{{}aws{}}}{{{}.{}}}{{{}sfdc{}}}{{{}.{}}}{{{}is{}}}{{{}%2C{}}}{{{}60020{}}}{{{}%2C{}}}{{{}1730886070174{}}}{{{}.{}}}{{{}1733410178680{}}}{{{}.{}}}{{{}temp{}}}{{{},
{}}}{{{}block{}}}{{{}:
{}}}{{{}BP{}}}{{{}-{}}}{{{}1745262640{}}}{{{}-{}}}{{{}10{}}}{{{}.{}}}{{{}60{}}}{{{}.{}}}{{{}130{}}}{{{}.{}}}{{{}13{}}}{{{}-{}}}{{{}1712173738392{}}}{{{}:{}}}{{{}blk{}}}{{{}_{}}}{{{}1330710217{}}}{{{}_{}}}{{{}257120451{}}}{{{},
{}}}{{Write}}{{ }}{{pipeline}}{{ }}{{{}datanodes{}}}{{{}:
[{}}}{{{}DatanodeInfoWithStorage{}}}{{{}[{}}}{{{}10{}}}{{{}.{}}}{{{}60{}}}{{{}.{}}}{{{}52{}}}{{{}.{}}}{{{}107{}}}{{{}:{}}}{{{}50010{}}}{{{},{}}}{{{}DS{}}}{{{}-{}}}{{{}f2b7ba1a{}}}{{{}-{}}}{{{}68b5{}}}{{{}-{}}}{{{}433a{}}}{{{}-{}}}{{{}9fe8{}}}{{{}-{}}}{{{}99315a172098{}}}{{{},{}}}{{{}SSD{}}}{{{}],
{}}}{{{}DatanodeInfoWithStorage{}}}{{{}[{}}}{{{}10{}}}{{{}.{}}}{{{}60{}}}{{{}.{}}}{{{}75{}}}{{{}.{}}}{{{}52{}}}{{{}:{}}}{{{}50010{}}}{{{},{}}}{{{}DS{}}}{{{}-{}}}{{{}93a433be{}}}{{{}-{}}}{{{}972f{}}}{{{}-{}}}{{{}4457{}}}{{{}-{}}}{{{}92ae{}}}{{{}-{}}}{{{}dd07288e41b5{}}}{{{},{}}}{{{}SSD{}}}{{{}]].{}}}
{color:#000000}[HMASTER-1]{color}
{{{}2024{}}}{{{}-{}}}{{{}12{}}}{{{}-{}}}{{{}05{}}}{{
}}{{{}18{}}}{{{}:{}}}{{{}11{}}}{{{}:{}}}{{{}42{}}}{{{},{}}}{{{}036{}}}{{
}}{{DEBUG}}{{{}
[{}}}{{{}master{}}}{{{}/{}}}{{{}hmaster{}}}{{{}-{}}}{{{}1{}}}{{{}:{}}}{{{}60000{}}}{{{}:{}}}{{{}becomeActiveMaster{}}}{{{}]
{}}}{{{}store{}}}{{{}.{}}}{{{}ProcedureTree{}}}{{
}}{{{}-{}}}{{{}{}}}{{{}Procedure{}}}{{
}}{{{}Procedure{}}}{{{}({}}}{{{}pid{}}}{{{}={}}}{{{}6003288{}}}{{{},
{}}}{{{}ppid{}}}{{{}={}}}{{{}6002575{}}}{{{},
{}}}{{{}class{}}}{{{}={}}}{{{}org{}}}{{{}.{}}}{{{}apache{}}}{{{}.{}}}{{{}hadoop{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}master{}}}{{{}.{}}}{{{}procedure{}}}{{{}.{}}}{{{}SplitWALRemoteProcedure{}}}{{{})
{}}}{{stack}}{{ }}{{{}ids{}}}{{{}=[{}}}{{{}3592]{}}}
{color:#000000}[HMASTER-1]{color}
{{{}2024{}}}{{{}-{}}}{{{}12{}}}{{{}-{}}}{{{}05{}}}{{
}}{{{}18{}}}{{{}:{}}}{{{}11{}}}{{{}:{}}}{{{}42{}}}{{{},{}}}{{{}214{}}}{{
}}{{DEBUG}}{{{}
[{}}}{{{}master{}}}{{{}/{}}}{{{}hmaster{}}}{{{}-{}}}{{{}1{}}}{{{}:{}}}{{{}60000{}}}{{{}:{}}}{{{}becomeActiveMaster{}}}{{{}]
{}}}{{{}procedure2{}}}{{{}.{}}}{{{}ProcedureExecutor{}}}{{ }}{{-}}{{
}}{{Loading}}{{ }}{{{}pid{}}}{{{}={}}}{{{}6003288{}}}{{{},
{}}}{{{}ppid{}}}{{{}={}}}{{{}6002575{}}}{{{},
{}}}{{{}state{}}}{{{}={}}}{{{}RUNNABLE{}}}{{{};
{}}}{{SplitWALRemoteProcedure}}{{
}}{{{}regionserver{}}}{{{}-{}}}{{{}53{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}hbase33a{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}monitoring{}}}{{{}.{}}}{{{}aws{}}}{{{}-{}}}{{{}esvc1{}}}{{{}-{}}}{{{}useast2{}}}{{{}.{}}}{{{}aws{}}}{{{}.{}}}{{{}sfdc{}}}{{{}.{}}}{{{}is{}}}{{{}%2C{}}}{{{}60020{}}}{{{}%2C{}}}{{{}1730886070174{}}}{{{}.{}}}{{{}1733410178680{}}}{{{},
{}}}{{{}worker{}}}{{{}={}}}{{{}regionserver{}}}{{{}-{}}}{{{}15{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}hbase33a{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}monitoring{}}}{{{}.{}}}{{{}aws{}}}{{{}-{}}}{{{}esvc1{}}}{{{}-{}}}{{{}useast2{}}}{{{}.{}}}{{{}aws{}}}{{{}.{}}}{{{}sfdc{}}}{{{}.{}}}{{{}is{}}}{{{},{}}}{{{}60020{}}}{{{},{}}}{{{}1730878238028{}}}
{color:#000000}[RS-15]{color}
{{{}2024{}}}{{{}-{}}}{{{}12{}}}{{{}-{}}}{{{}05{}}}{{
}}{{{}18{}}}{{{}:{}}}{{{}11{}}}{{{}:{}}}{{{}42{}}}{{{},{}}}{{{}769{}}}{{
}}{{DEBUG}}{{{}
[{}}}{{{}iority{}}}{{{}.{}}}{{{}RWQ{}}}{{{}.{}}}{{{}Fifo{}}}{{{}.{}}}{{{}read{}}}{{{}.{}}}{{{}handler{}}}{{{}={}}}{{{}79{}}}{{{},{}}}{{{}queue{}}}{{{}={}}}{{{}7{}}}{{{},{}}}{{{}port{}}}{{{}={}}}{{{}60020{}}}{{{}]
{}}}{{{}regionserver{}}}{{{}.{}}}{{{}RSRpcServices{}}}{{ }}{{-}}{{
}}{{Executing}}{{ }}{{remote}}{{ }}{{procedure}}{{
}}{{{}class{}}}{{{}{}}}{{{}org{}}}{{{}.{}}}{{{}apache{}}}{{{}.{}}}{{{}hadoop{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}SplitWALCallable{}}}{{{},
{}}}{{{}pid{}}}{{{}={}}}{{{}6003288{}}}
{color:#000000}[RS-15]{color}
{{{}2024{}}}{{{}-{}}}{{{}12{}}}{{{}-{}}}{{{}05{}}}{{
}}{{{}18{}}}{{{}:{}}}{{{}11{}}}{{{}:{}}}{{{}48{}}}{{{},{}}}{{{}247{}}}{{
}}{{DEBUG}}{{{}
[{}}}{{{}_{}}}{{{}REPLAY{}}}{{{}_{}}}{{{}OPS{}}}{{{}-{}}}{{{}regionserver{}}}{{{}/{}}}{{{}regionserver{}}}{{{}-{}}}{{{}15{}}}{{{}:{}}}{{{}60020{}}}{{{}-{}}}{{{}192{}}}{{{}]
{}}}{{{}regionserver{}}}{{{}.{}}}{{{}RemoteProcedureResultReporter{}}}{{
}}{{-}}{{ }}{{Successfully}}{{ }}{{complete}}{{ }}{{execution}}{{ }}{{of}}{{
}}{{{}pid{}}}{{{}={}}}{{{}6003288{}}}
{color:#000000}[HMASTER-1]{color}
{{{}2024{}}}{{{}-{}}}{{{}12{}}}{{{}-{}}}{{{}05{}}}{{
}}{{{}18{}}}{{{}:{}}}{{{}11{}}}{{{}:{}}}{{{}48{}}}{{{},{}}}{{{}304{}}}{{
}}{{INFO}}{{{} [{}}}{{{}PEWorker{}}}{{{}-{}}}{{{}2{}}}{{{}]
{}}}{{{}procedure2{}}}{{{}.{}}}{{{}ProcedureExecutor{}}}{{ }}{{-}}{{
}}{{Finished}}{{ }}{{{}pid{}}}{{{}={}}}{{{}6002575{}}}{{{},
{}}}{{{}ppid{}}}{{{}={}}}{{{}6000775{}}}{{{},
{}}}{{{}state{}}}{{{}={}}}{{{}SUCCESS{}}}{{{};
{}}}{{{}SplitWALProcedure{}}}{{{}{}}}{{{}regionserver{}}}{{{}-{}}}{{{}53{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}hbase33a{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}monitoring{}}}{{{}.{}}}{{{}aws{}}}{{{}-{}}}{{{}esvc1{}}}{{{}-{}}}{{{}useast2{}}}{{{}.{}}}{{{}aws{}}}{{{}.{}}}{{{}sfdc{}}}{{{}.{}}}{{{}is{}}}{{{}%2C{}}}{{{}60020{}}}{{{}%2C{}}}{{{}1730886070174{}}}{{{}.{}}}{{{}1733410178680{}}}{{{},
{}}}{{{}worker{}}}{{{}={}}}{{{}regionserver{}}}{{{}-{}}}{{{}15{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}hbase33a{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}monitoring{}}}{{{}.{}}}{{{}aws{}}}{{{}-{}}}{{{}esvc1{}}}{{{}-{}}}{{{}useast2{}}}{{{}.{}}}{{{}aws{}}}{{{}.{}}}{{{}sfdc{}}}{{{}.{}}}{{{}is{}}}{{{},{}}}{{{}60020{}}}{{{},{}}}{{{}1730878238028{}}}{{
}}{{in}}{{ }}*{{3}}{{ }}{{{}hrs{}}}{{{}, {}}}{{16}}{{ }}{{{}mins{}}}{{{},
{}}}{{{}52{}}}{{{}.{}}}{{{}806{}}}{{ }}{{sec}}*
{quote}
was:
In HBase cluster we have encountered a scenario where regionserver server crash
procedure(SCP) waited for more than 3 Hours. Incident was triggered due to
temporary network unavailability in hbase cluster. On Debugging found out SCP
was stuck due to child {{SplitWALProcedure}} which was waiting for completion
of SpliWalRemote procedure by regionserver worker. SplitWALRemote procedure
while running encountered{{{{}} an unknown exception. In logs we can see
*"hdfs*{}}}{*}{{{}.{}}}{{{}DataStreamer - {{}}}}{{{}No ack
}}\{{{}receive{}}}{{{}d{}}}"{*} error while regionserver connecting to Data
Node. After this error thread was stuck or died as there was no related logs
exists{{{}. There were inconsistent regions reported during this period. All
procedure were restarted and completed after Active HMaster service was
bounced. {}}}
Related logs:
{quote}{color:#000000}[HMASTER-4]{color}
{{{}2024{}}}{{{}-{}}}{{{}12{}}}{{{}-{}}}{{{}05{}}}{{
}}{{{}14{}}}{{{}:{}}}{{{}55{}}}{{{}:{}}}{{{}11{}}}{{{},{}}}{{{}264{}}}{{
}}{{INFO}}{{{} [{}}}{{{}PEWorker{}}}{{{}-{}}}{{{}41{}}}{{{}]
{}}}{{{}procedure2{}}}{{{}.{}}}{{{}ProcedureExecutor{}}}{{ }}{{-}}{{
}}{{Initialized}}{{
}}{{{}subprocedures{}}}{{{}=[{{}}}{{{}pid{}}}{{{}={}}}{{{}6003288{}}}{{{},
{}}}{{{}ppid{}}}{{{}={}}}{{{}6002575{}}}{{{},
{}}}{{{}state{}}}{{{}={}}}{{{}RUNNABLE{}}}{{{};
{}}}{{{}SplitWALRemoteProcedure{}}}{{{}{}}}{{{}regionserver{}}}{{{}-{}}}{{{}53{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}hbase33a{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}monitoring{}}}{{{}.{}}}{{{}aws{}}}{{{}-{}}}{{{}esvc1{}}}{{{}-{}}}{{{}useast2{}}}{{{}.{}}}{{{}aws{}}}{{{}.{}}}{{{}sfdc{}}}{{{}.{}}}{{{}is{}}}{{{}%2C{}}}{{{}60020{}}}{{{}%2C{}}}{{{}1730886070174{}}}{{{}.{}}}{{{}1733410178680{}}}{{{},
{}}}{{{}worker{}}}{{{}={}}}{{{}regionserver{}}}{{{}-{}}}{{{}15{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}hbase33a{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}monitoring{}}}{{{}.{}}}{{{}aws{}}}{{{}-{}}}{{{}esvc1{}}}{{{}-{}}}{{{}useast2{}}}{{{}.{}}}{{{}aws{}}}{{{}.{}}}{{{}sfdc{}}}{{{}.{}}}{{{}is{}}}{{{},{}}}{{{}60020{}}}{{{},{}}}{{{}1730878238028{}}}{{{}}]{}}}
{color:#000000}[RS-15]{color}
{{{}2024{}}}{{{}-{}}}{{{}12{}}}{{{}-{}}}{{{}05{}}}{{
}}{{{}14{}}}{{{}:{}}}{{{}55{}}}{{{}:{}}}{{{}11{}}}{{{},{}}}{{{}461{}}}{{
}}{{DEBUG}}{{{}
[{}}}{{{}iority{}}}{{{}.{}}}{{{}RWQ{}}}{{{}.{}}}{{{}Fifo{}}}{{{}.{}}}{{{}read{}}}{{{}.{}}}{{{}handler{}}}{{{}={}}}{{{}83{}}}{{{},{}}}{{{}queue{}}}{{{}={}}}{{{}2{}}}{{{},{}}}{{{}port{}}}{{{}={}}}{{{}60020{}}}{{{}]
{}}}{{{}regionserver{}}}{{{}.{}}}{{{}RSRpcServices{}}}{{ }}{{-}}{{
}}{{Executing}}{{ }}{{remote}}{{ }}{{procedure}}{{
}}{{{}class{}}}{{{}{}}}{{{}org{}}}{{{}.{}}}{{{}apache{}}}{{{}.{}}}{{{}hadoop{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}SplitWALCallable{}}}{{{},
{}}}{{{}pid{}}}{{{}={}}}{{{}6003288{}}}
{color:#000000}[RS-15]{color}
{{{}2024{}}}{{{}-{}}}{{{}12{}}}{{{}-{}}}{{{}05{}}}{{
}}{{{}14{}}}{{{}:{}}}{{{}55{}}}{{{}:{}}}{{{}54{}}}{{{},{}}}{{{}689{}}}{{
}}{{ERROR}}{{{}
[{}}}{{{}split{}}}{{{}-{}}}{{{}log{}}}{{{}-{}}}{{{}closeStream{}}}{{{}-{}}}{{{}pool{}}}{{{}-{}}}{{{}0{}}}{{{}]
{}}}{{{}hdfs{}}}{{{}.{}}}{{{}DataStreamer{}}}{{ }}{{-}}{{ }}{{No}}{{
}}{{ack}}{{ }}{{{}received{}}}{{{}, {}}}{{took}}{{ }}{{25002ms}}{{{}
({}}}{{{}threshold{}}}{{{}={}}}{{{}25000ms{}}}{{{}){}}}{{{}.{}}}{{ }}{{File}}{{
}}{{being}}{{ }}{{{}written{}}}{{{}:
/{}}}{{{}hbase{}}}{{{}/{}}}{{{}data{}}}{{{}/{}}}{{{}default{}}}{{{}/{}}}{{{}tsdb{}}}{{{}/{}}}{{{}c997c5f8dd36481dcd3ebb9b79a35b51{}}}{{{}/{}}}{{{}recovered{}}}{{{}.{}}}{{{}edits{}}}{{{}/{}}}{{{}0000000000539451088{}}}{{{}-{}}}{{{}regionserver{}}}{{{}-{}}}{{{}53{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}hbase33a{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}monitoring{}}}{{{}.{}}}{{{}aws{}}}{{{}-{}}}{{{}esvc1{}}}{{{}-{}}}{{{}useast2{}}}{{{}.{}}}{{{}aws{}}}{{{}.{}}}{{{}sfdc{}}}{{{}.{}}}{{{}is{}}}{{{}%2C{}}}{{{}60020{}}}{{{}%2C{}}}{{{}1730886070174{}}}{{{}.{}}}{{{}1733410178680{}}}{{{}.{}}}{{{}temp{}}}{{{},
{}}}{{{}block{}}}{{{}:
{}}}{{{}BP{}}}{{{}-{}}}{{{}1745262640{}}}{{{}-{}}}{{{}10{}}}{{{}.{}}}{{{}60{}}}{{{}.{}}}{{{}130{}}}{{{}.{}}}{{{}13{}}}{{{}-{}}}{{{}1712173738392{}}}{{{}:{}}}{{{}blk{}}}{{{}_{}}}{{{}1330710217{}}}{{{}_{}}}{{{}257120451{}}}{{{},
{}}}{{Write}}{{ }}{{pipeline}}{{ }}{{{}datanodes{}}}{{{}:
[{}}}{{{}DatanodeInfoWithStorage{}}}{{{}[{}}}{{{}10{}}}{{{}.{}}}{{{}60{}}}{{{}.{}}}{{{}52{}}}{{{}.{}}}{{{}107{}}}{{{}:{}}}{{{}50010{}}}{{{},{}}}{{{}DS{}}}{{{}-{}}}{{{}f2b7ba1a{}}}{{{}-{}}}{{{}68b5{}}}{{{}-{}}}{{{}433a{}}}{{{}-{}}}{{{}9fe8{}}}{{{}-{}}}{{{}99315a172098{}}}{{{},{}}}{{{}SSD{}}}{{{}],
{}}}{{{}DatanodeInfoWithStorage{}}}{{{}[{}}}{{{}10{}}}{{{}.{}}}{{{}60{}}}{{{}.{}}}{{{}75{}}}{{{}.{}}}{{{}52{}}}{{{}:{}}}{{{}50010{}}}{{{},{}}}{{{}DS{}}}{{{}-{}}}{{{}93a433be{}}}{{{}-{}}}{{{}972f{}}}{{{}-{}}}{{{}4457{}}}{{{}-{}}}{{{}92ae{}}}{{{}-{}}}{{{}dd07288e41b5{}}}{{{},{}}}{{{}SSD{}}}{{{}]].{}}}
{color:#000000}[HMASTER-1]{color}
{{{}2024{}}}{{{}-{}}}{{{}12{}}}{{{}-{}}}{{{}05{}}}{{
}}{{{}18{}}}{{{}:{}}}{{{}11{}}}{{{}:{}}}{{{}42{}}}{{{},{}}}{{{}036{}}}{{
}}{{DEBUG}}{{{}
[{}}}{{{}master{}}}{{{}/{}}}{{{}hmaster{}}}{{{}-{}}}{{{}1{}}}{{{}:{}}}{{{}60000{}}}{{{}:{}}}{{{}becomeActiveMaster{}}}{{{}]
{}}}{{{}store{}}}{{{}.{}}}{{{}ProcedureTree{}}}{{
}}{{{}-{}}}{{{}{}}}{{{}Procedure{}}}{{
}}{{{}Procedure{}}}{{{}({}}}{{{}pid{}}}{{{}={}}}{{{}6003288{}}}{{{},
{}}}{{{}ppid{}}}{{{}={}}}{{{}6002575{}}}{{{},
{}}}{{{}class{}}}{{{}={}}}{{{}org{}}}{{{}.{}}}{{{}apache{}}}{{{}.{}}}{{{}hadoop{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}master{}}}{{{}.{}}}{{{}procedure{}}}{{{}.{}}}{{{}SplitWALRemoteProcedure{}}}{{{})
{}}}{{stack}}{{ }}{{{}ids{}}}{{{}=[{}}}{{{}3592]{}}}
{color:#000000}[HMASTER-1]{color}
{{{}2024{}}}{{{}-{}}}{{{}12{}}}{{{}-{}}}{{{}05{}}}{{
}}{{{}18{}}}{{{}:{}}}{{{}11{}}}{{{}:{}}}{{{}42{}}}{{{},{}}}{{{}214{}}}{{
}}{{DEBUG}}{{{}
[{}}}{{{}master{}}}{{{}/{}}}{{{}hmaster{}}}{{{}-{}}}{{{}1{}}}{{{}:{}}}{{{}60000{}}}{{{}:{}}}{{{}becomeActiveMaster{}}}{{{}]
{}}}{{{}procedure2{}}}{{{}.{}}}{{{}ProcedureExecutor{}}}{{ }}{{-}}{{
}}{{Loading}}{{ }}{{{}pid{}}}{{{}={}}}{{{}6003288{}}}{{{},
{}}}{{{}ppid{}}}{{{}={}}}{{{}6002575{}}}{{{},
{}}}{{{}state{}}}{{{}={}}}{{{}RUNNABLE{}}}{{{};
{}}}{{SplitWALRemoteProcedure}}{{
}}{{{}regionserver{}}}{{{}-{}}}{{{}53{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}hbase33a{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}monitoring{}}}{{{}.{}}}{{{}aws{}}}{{{}-{}}}{{{}esvc1{}}}{{{}-{}}}{{{}useast2{}}}{{{}.{}}}{{{}aws{}}}{{{}.{}}}{{{}sfdc{}}}{{{}.{}}}{{{}is{}}}{{{}%2C{}}}{{{}60020{}}}{{{}%2C{}}}{{{}1730886070174{}}}{{{}.{}}}{{{}1733410178680{}}}{{{},
{}}}{{{}worker{}}}{{{}={}}}{{{}regionserver{}}}{{{}-{}}}{{{}15{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}hbase33a{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}monitoring{}}}{{{}.{}}}{{{}aws{}}}{{{}-{}}}{{{}esvc1{}}}{{{}-{}}}{{{}useast2{}}}{{{}.{}}}{{{}aws{}}}{{{}.{}}}{{{}sfdc{}}}{{{}.{}}}{{{}is{}}}{{{},{}}}{{{}60020{}}}{{{},{}}}{{{}1730878238028{}}}
{color:#000000}[RS-15]{color}
{{{}2024{}}}{{{}-{}}}{{{}12{}}}{{{}-{}}}{{{}05{}}}{{
}}{{{}18{}}}{{{}:{}}}{{{}11{}}}{{{}:{}}}{{{}42{}}}{{{},{}}}{{{}769{}}}{{
}}{{DEBUG}}{{{}
[{}}}{{{}iority{}}}{{{}.{}}}{{{}RWQ{}}}{{{}.{}}}{{{}Fifo{}}}{{{}.{}}}{{{}read{}}}{{{}.{}}}{{{}handler{}}}{{{}={}}}{{{}79{}}}{{{},{}}}{{{}queue{}}}{{{}={}}}{{{}7{}}}{{{},{}}}{{{}port{}}}{{{}={}}}{{{}60020{}}}{{{}]
{}}}{{{}regionserver{}}}{{{}.{}}}{{{}RSRpcServices{}}}{{ }}{{-}}{{
}}{{Executing}}{{ }}{{remote}}{{ }}{{procedure}}{{
}}{{{}class{}}}{{{}{}}}{{{}org{}}}{{{}.{}}}{{{}apache{}}}{{{}.{}}}{{{}hadoop{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}SplitWALCallable{}}}{{{},
{}}}{{{}pid{}}}{{{}={}}}{{{}6003288{}}}
{color:#000000}[RS-15]{color}
{{{}2024{}}}{{{}-{}}}{{{}12{}}}{{{}-{}}}{{{}05{}}}{{
}}{{{}18{}}}{{{}:{}}}{{{}11{}}}{{{}:{}}}{{{}48{}}}{{{},{}}}{{{}247{}}}{{
}}{{DEBUG}}{{{}
[{}}}{{{}_{}}}{{{}REPLAY{}}}{{{}_{}}}{{{}OPS{}}}{{{}-{}}}{{{}regionserver{}}}{{{}/{}}}{{{}regionserver{}}}{{{}-{}}}{{{}15{}}}{{{}:{}}}{{{}60020{}}}{{{}-{}}}{{{}192{}}}{{{}]
{}}}{{{}regionserver{}}}{{{}.{}}}{{{}RemoteProcedureResultReporter{}}}{{
}}{{-}}{{ }}{{Successfully}}{{ }}{{complete}}{{ }}{{execution}}{{ }}{{of}}{{
}}{{{}pid{}}}{{{}={}}}{{{}6003288{}}}
{color:#000000}[HMASTER-1]{color}
{{{}2024{}}}{{{}-{}}}{{{}12{}}}{{{}-{}}}{{{}05{}}}{{
}}{{{}18{}}}{{{}:{}}}{{{}11{}}}{{{}:{}}}{{{}48{}}}{{{},{}}}{{{}304{}}}{{
}}{{INFO}}{{{} [{}}}{{{}PEWorker{}}}{{{}-{}}}{{{}2{}}}{{{}]
{}}}{{{}procedure2{}}}{{{}.{}}}{{{}ProcedureExecutor{}}}{{ }}{{-}}{{
}}{{Finished}}{{ }}{{{}pid{}}}{{{}={}}}{{{}6002575{}}}{{{},
{}}}{{{}ppid{}}}{{{}={}}}{{{}6000775{}}}{{{},
{}}}{{{}state{}}}{{{}={}}}{{{}SUCCESS{}}}{{{};
{}}}{{{}SplitWALProcedure{}}}{{{}{}}}{{{}regionserver{}}}{{{}-{}}}{{{}53{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}hbase33a{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}monitoring{}}}{{{}.{}}}{{{}aws{}}}{{{}-{}}}{{{}esvc1{}}}{{{}-{}}}{{{}useast2{}}}{{{}.{}}}{{{}aws{}}}{{{}.{}}}{{{}sfdc{}}}{{{}.{}}}{{{}is{}}}{{{}%2C{}}}{{{}60020{}}}{{{}%2C{}}}{{{}1730886070174{}}}{{{}.{}}}{{{}1733410178680{}}}{{{},
{}}}{{{}worker{}}}{{{}={}}}{{{}regionserver{}}}{{{}-{}}}{{{}15{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}hbase33a{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}monitoring{}}}{{{}.{}}}{{{}aws{}}}{{{}-{}}}{{{}esvc1{}}}{{{}-{}}}{{{}useast2{}}}{{{}.{}}}{{{}aws{}}}{{{}.{}}}{{{}sfdc{}}}{{{}.{}}}{{{}is{}}}{{{},{}}}{{{}60020{}}}{{{},{}}}{{{}1730878238028{}}}{{
}}{{in}}{{ }}*{{3}}{{ }}{{{}hrs{}}}{{{}, {}}}{{16}}{{ }}{{{}mins{}}}{{{},
{}}}{{{}52{}}}{{{}.{}}}{{{}806{}}}{{ }}{{sec}}*
{quote}
> Set UncaughtException Handler for RegionServer ExecutorService
> --------------------------------------------------------------
>
> Key: HBASE-29041
> URL: https://issues.apache.org/jira/browse/HBASE-29041
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 3.0.0, 2.6.1, 2.5.10
> Reporter: Nirdosh Kumar Yadav
> Priority: Minor
>
> In HBase cluster we have encountered a scenario where regionserver server
> crash procedure(SCP) waited for more than 3 Hours. Incident was triggered due
> to temporary network unavailability in hbase cluster. On Debugging found out
> SCP was stuck due to child {{SplitWALProcedure}} which was waiting for
> completion of SpliWalRemote procedure by regionserver worker. SplitWALRemote
> procedure while running encountered{{{{}} an unknown exception. In logs we
> can see *"hdfs*{}}}{*}{{{}.{}}}{{{}DataStreamer - {{}}}}{{{}No ack
> }}\{{{}receive{}}}{{{}d{}}}"{*} error while regionserver connecting to Data
> Node. After this error thread was stuck or died as there was no related logs
> exists{{{}. There were inconsistent regions reported during this period. All
> procedure were restarted and completed after Active HMaster service was
> bounced. {}}}
> Related logs:
> {quote}{color:#000000}[HMASTER-4]{color}
> {{{}2024{}}}{{{}-{}}}{{{}12{}}}{{{}-{}}}{{{}05{}}}{{
> }}{{{}14{}}}{{{}:{}}}{{{}55{}}}{{{}:{}}}{{{}11{}}}{{{},{}}}{{{}264{}}}{{
> }}{{INFO}}{{{} [{}}}{{{}PEWorker{}}}{{{}-{}}}{{{}41{}}}{{{}]
> {}}}{{{}procedure2{}}}{{{}.{}}}{{{}ProcedureExecutor{}}}{{ }}{{-}}{{
> }}{{Initialized}}{{
> }}{{{}subprocedures{}}}{{{}=[{{}}}{{{}pid{}}}{{{}={}}}{{{}6003288{}}}{{{},
> {}}}{{{}ppid{}}}{{{}={}}}{{{}6002575{}}}{{{},
> {}}}{{{}state{}}}{{{}={}}}{{{}RUNNABLE{}}}{{{};
> {}}}{{{}SplitWALRemoteProcedure{}}}{{{}{}}}{{{}regionserver{}}}{{{}-{}}}{{{}53{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}hbase33a{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}monitoring{}}}{{{}.{}}}{{{}aws{}}}{{{}-{}}}{{{}esvc1{}}}{{{}-{}}}{{{}useast2{}}}{{{}.{}}}{{{}aws{}}}{{{}.{}}}{{{}sfdc{}}}{{{}.{}}}{{{}is{}}}{{{}%2C{}}}{{{}60020{}}}{{{}%2C{}}}{{{}1730886070174{}}}{{{}.{}}}{{{}1733410178680{}}}{{{},
>
> {}}}{{{}worker{}}}{{{}={}}}{{{}regionserver{}}}{{{}-{}}}{{{}15{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}hbase33a{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}monitoring{}}}{{{}.{}}}{{{}aws{}}}{{{}-{}}}{{{}esvc1{}}}{{{}-{}}}{{{}useast2{}}}{{{}.{}}}{{{}aws{}}}{{{}.{}}}{{{}sfdc{}}}{{{}.{}}}{{{}is{}}}{{{},{}}}{{{}60020{}}}{{{},{}}}{{{}1730878238028{}}}{{{}}]{}}}
> {color:#000000}[RS-15]{color}
> {{{}2024{}}}{{{}-{}}}{{{}12{}}}{{{}-{}}}{{{}05{}}}{{
> }}{{{}14{}}}{{{}:{}}}{{{}55{}}}{{{}:{}}}{{{}11{}}}{{{},{}}}{{{}461{}}}{{
> }}{{DEBUG}}{{{}
> [{}}}{{{}iority{}}}{{{}.{}}}{{{}RWQ{}}}{{{}.{}}}{{{}Fifo{}}}{{{}.{}}}{{{}read{}}}{{{}.{}}}{{{}handler{}}}{{{}={}}}{{{}83{}}}{{{},{}}}{{{}queue{}}}{{{}={}}}{{{}2{}}}{{{},{}}}{{{}port{}}}{{{}={}}}{{{}60020{}}}{{{}]
> {}}}{{{}regionserver{}}}{{{}.{}}}{{{}RSRpcServices{}}}{{ }}{{-}}{{
> }}{{Executing}}{{ }}{{remote}}{{ }}{{procedure}}{{
> }}{{{}class{}}}{{{}{}}}{{{}org{}}}{{{}.{}}}{{{}apache{}}}{{{}.{}}}{{{}hadoop{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}SplitWALCallable{}}}{{{},
> {}}}{{{}pid{}}}{{{}={}}}{{{}6003288{}}}
> {color:#000000}[RS-15]{color}
> {{{}2024{}}}{{{}-{}}}{{{}12{}}}{{{}-{}}}{{{}05{}}}{{
> }}{{{}14{}}}{{{}:{}}}{{{}55{}}}{{{}:{}}}{{{}54{}}}{{{},{}}}{{{}689{}}}{{
> }}{{ERROR}}{{{}
> [{}}}{{{}split{}}}{{{}-{}}}{{{}log{}}}{{{}-{}}}{{{}closeStream{}}}{{{}-{}}}{{{}pool{}}}{{{}-{}}}{{{}0{}}}{{{}]
> {}}}{{{}hdfs{}}}{{{}.{}}}{{{}DataStreamer{}}}{{ }}{{-}}{{ }}{{No}}{{
> }}{{ack}}{{ }}{{{}received{}}}{{{}, {}}}{{took}}{{ }}{{25002ms}}{{{}
> ({}}}{{{}threshold{}}}{{{}={}}}{{{}25000ms{}}}{{{}){}}}{{{}.{}}}{{
> }}{{File}}{{ }}{{being}}{{ }}{{{}written{}}}{{{}:
> /{}}}{{{}hbase{}}}{{{}/{}}}{{{}data{}}}{{{}/{}}}{{{}default{}}}{{{}/{}}}{{{}tsdb{}}}{{{}/{}}}{{{}c997c5f8dd36481dcd3ebb9b79a35b51{}}}{{{}/{}}}{{{}recovered{}}}{{{}.{}}}{{{}edits{}}}{{{}/{}}}{{{}0000000000539451088{}}}{{{}-{}}}{{{}regionserver{}}}{{{}-{}}}{{{}53{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}hbase33a{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}monitoring{}}}{{{}.{}}}{{{}aws{}}}{{{}-{}}}{{{}esvc1{}}}{{{}-{}}}{{{}useast2{}}}{{{}.{}}}{{{}aws{}}}{{{}.{}}}{{{}sfdc{}}}{{{}.{}}}{{{}is{}}}{{{}%2C{}}}{{{}60020{}}}{{{}%2C{}}}{{{}1730886070174{}}}{{{}.{}}}{{{}1733410178680{}}}{{{}.{}}}{{{}temp{}}}{{{},
> {}}}{{{}block{}}}{{{}:
> {}}}{{{}BP{}}}{{{}-{}}}{{{}1745262640{}}}{{{}-{}}}{{{}10{}}}{{{}.{}}}{{{}60{}}}{{{}.{}}}{{{}130{}}}{{{}.{}}}{{{}13{}}}{{{}-{}}}{{{}1712173738392{}}}{{{}:{}}}{{{}blk{}}}{{{}_{}}}{{{}1330710217{}}}{{{}_{}}}{{{}257120451{}}}{{{},
> {}}}{{Write}}{{ }}{{pipeline}}{{ }}{{{}datanodes{}}}{{{}:
> [{}}}{{{}DatanodeInfoWithStorage{}}}{{{}[{}}}{{{}10{}}}{{{}.{}}}{{{}60{}}}{{{}.{}}}{{{}52{}}}{{{}.{}}}{{{}107{}}}{{{}:{}}}{{{}50010{}}}{{{},{}}}{{{}DS{}}}{{{}-{}}}{{{}f2b7ba1a{}}}{{{}-{}}}{{{}68b5{}}}{{{}-{}}}{{{}433a{}}}{{{}-{}}}{{{}9fe8{}}}{{{}-{}}}{{{}99315a172098{}}}{{{},{}}}{{{}SSD{}}}{{{}],
>
> {}}}{{{}DatanodeInfoWithStorage{}}}{{{}[{}}}{{{}10{}}}{{{}.{}}}{{{}60{}}}{{{}.{}}}{{{}75{}}}{{{}.{}}}{{{}52{}}}{{{}:{}}}{{{}50010{}}}{{{},{}}}{{{}DS{}}}{{{}-{}}}{{{}93a433be{}}}{{{}-{}}}{{{}972f{}}}{{{}-{}}}{{{}4457{}}}{{{}-{}}}{{{}92ae{}}}{{{}-{}}}{{{}dd07288e41b5{}}}{{{},{}}}{{{}SSD{}}}{{{}]].{}}}
> {color:#000000}[HMASTER-1]{color}
> {{{}2024{}}}{{{}-{}}}{{{}12{}}}{{{}-{}}}{{{}05{}}}{{
> }}{{{}18{}}}{{{}:{}}}{{{}11{}}}{{{}:{}}}{{{}42{}}}{{{},{}}}{{{}036{}}}{{
> }}{{DEBUG}}{{{}
> [{}}}{{{}master{}}}{{{}/{}}}{{{}hmaster{}}}{{{}-{}}}{{{}1{}}}{{{}:{}}}{{{}60000{}}}{{{}:{}}}{{{}becomeActiveMaster{}}}{{{}]
> {}}}{{{}store{}}}{{{}.{}}}{{{}ProcedureTree{}}}{{
> }}{{{}-{}}}{{{}{}}}{{{}Procedure{}}}{{
> }}{{{}Procedure{}}}{{{}({}}}{{{}pid{}}}{{{}={}}}{{{}6003288{}}}{{{},
> {}}}{{{}ppid{}}}{{{}={}}}{{{}6002575{}}}{{{},
> {}}}{{{}class{}}}{{{}={}}}{{{}org{}}}{{{}.{}}}{{{}apache{}}}{{{}.{}}}{{{}hadoop{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}master{}}}{{{}.{}}}{{{}procedure{}}}{{{}.{}}}{{{}SplitWALRemoteProcedure{}}}{{{})
> {}}}{{stack}}{{ }}{{{}ids{}}}{{{}=[{}}}{{{}3592]{}}}
> {color:#000000}[HMASTER-1]{color}
> {{{}2024{}}}{{{}-{}}}{{{}12{}}}{{{}-{}}}{{{}05{}}}{{
> }}{{{}18{}}}{{{}:{}}}{{{}11{}}}{{{}:{}}}{{{}42{}}}{{{},{}}}{{{}214{}}}{{
> }}{{DEBUG}}{{{}
> [{}}}{{{}master{}}}{{{}/{}}}{{{}hmaster{}}}{{{}-{}}}{{{}1{}}}{{{}:{}}}{{{}60000{}}}{{{}:{}}}{{{}becomeActiveMaster{}}}{{{}]
> {}}}{{{}procedure2{}}}{{{}.{}}}{{{}ProcedureExecutor{}}}{{ }}{{-}}{{
> }}{{Loading}}{{ }}{{{}pid{}}}{{{}={}}}{{{}6003288{}}}{{{},
> {}}}{{{}ppid{}}}{{{}={}}}{{{}6002575{}}}{{{},
> {}}}{{{}state{}}}{{{}={}}}{{{}RUNNABLE{}}}{{{};
> {}}}{{SplitWALRemoteProcedure}}{{
> }}{{{}regionserver{}}}{{{}-{}}}{{{}53{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}hbase33a{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}monitoring{}}}{{{}.{}}}{{{}aws{}}}{{{}-{}}}{{{}esvc1{}}}{{{}-{}}}{{{}useast2{}}}{{{}.{}}}{{{}aws{}}}{{{}.{}}}{{{}sfdc{}}}{{{}.{}}}{{{}is{}}}{{{}%2C{}}}{{{}60020{}}}{{{}%2C{}}}{{{}1730886070174{}}}{{{}.{}}}{{{}1733410178680{}}}{{{},
>
> {}}}{{{}worker{}}}{{{}={}}}{{{}regionserver{}}}{{{}-{}}}{{{}15{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}hbase33a{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}monitoring{}}}{{{}.{}}}{{{}aws{}}}{{{}-{}}}{{{}esvc1{}}}{{{}-{}}}{{{}useast2{}}}{{{}.{}}}{{{}aws{}}}{{{}.{}}}{{{}sfdc{}}}{{{}.{}}}{{{}is{}}}{{{},{}}}{{{}60020{}}}{{{},{}}}{{{}1730878238028{}}}
> {color:#000000}[RS-15]{color}
> {{{}2024{}}}{{{}-{}}}{{{}12{}}}{{{}-{}}}{{{}05{}}}{{
> }}{{{}18{}}}{{{}:{}}}{{{}11{}}}{{{}:{}}}{{{}42{}}}{{{},{}}}{{{}769{}}}{{
> }}{{DEBUG}}{{{}
> [{}}}{{{}iority{}}}{{{}.{}}}{{{}RWQ{}}}{{{}.{}}}{{{}Fifo{}}}{{{}.{}}}{{{}read{}}}{{{}.{}}}{{{}handler{}}}{{{}={}}}{{{}79{}}}{{{},{}}}{{{}queue{}}}{{{}={}}}{{{}7{}}}{{{},{}}}{{{}port{}}}{{{}={}}}{{{}60020{}}}{{{}]
> {}}}{{{}regionserver{}}}{{{}.{}}}{{{}RSRpcServices{}}}{{ }}{{-}}{{
> }}{{Executing}}{{ }}{{remote}}{{ }}{{procedure}}{{
> }}{{{}class{}}}{{{}{}}}{{{}org{}}}{{{}.{}}}{{{}apache{}}}{{{}.{}}}{{{}hadoop{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}SplitWALCallable{}}}{{{},
> {}}}{{{}pid{}}}{{{}={}}}{{{}6003288{}}}
> {color:#000000}[RS-15]{color}
> {{{}2024{}}}{{{}-{}}}{{{}12{}}}{{{}-{}}}{{{}05{}}}{{
> }}{{{}18{}}}{{{}:{}}}{{{}11{}}}{{{}:{}}}{{{}48{}}}{{{},{}}}{{{}247{}}}{{
> }}{{DEBUG}}{{{}
> [{}}}{{{}_{}}}{{{}REPLAY{}}}{{{}_{}}}{{{}OPS{}}}{{{}-{}}}{{{}regionserver{}}}{{{}/{}}}{{{}regionserver{}}}{{{}-{}}}{{{}15{}}}{{{}:{}}}{{{}60020{}}}{{{}-{}}}{{{}192{}}}{{{}]
> {}}}{{{}regionserver{}}}{{{}.{}}}{{{}RemoteProcedureResultReporter{}}}{{
> }}{{-}}{{ }}{{Successfully}}{{ }}{{complete}}{{ }}{{execution}}{{ }}{{of}}{{
> }}{{{}pid{}}}{{{}={}}}{{{}6003288{}}}
> {color:#000000}[HMASTER-1]{color}
> {{{}2024{}}}{{{}-{}}}{{{}12{}}}{{{}-{}}}{{{}05{}}}{{
> }}{{{}18{}}}{{{}:{}}}{{{}11{}}}{{{}:{}}}{{{}48{}}}{{{},{}}}{{{}304{}}}{{
> }}{{INFO}}{{{} [{}}}{{{}PEWorker{}}}{{{}-{}}}{{{}2{}}}{{{}]
> {}}}{{{}procedure2{}}}{{{}.{}}}{{{}ProcedureExecutor{}}}{{ }}{{-}}{{
> }}{{Finished}}{{ }}{{{}pid{}}}{{{}={}}}{{{}6002575{}}}{{{},
> {}}}{{{}ppid{}}}{{{}={}}}{{{}6000775{}}}{{{},
> {}}}{{{}state{}}}{{{}={}}}{{{}SUCCESS{}}}{{{};
> {}}}{{{}SplitWALProcedure{}}}{{{}{}}}{{{}regionserver{}}}{{{}-{}}}{{{}53{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}hbase33a{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}monitoring{}}}{{{}.{}}}{{{}aws{}}}{{{}-{}}}{{{}esvc1{}}}{{{}-{}}}{{{}useast2{}}}{{{}.{}}}{{{}aws{}}}{{{}.{}}}{{{}sfdc{}}}{{{}.{}}}{{{}is{}}}{{{}%2C{}}}{{{}60020{}}}{{{}%2C{}}}{{{}1730886070174{}}}{{{}.{}}}{{{}1733410178680{}}}{{{},
>
> {}}}{{{}worker{}}}{{{}={}}}{{{}regionserver{}}}{{{}-{}}}{{{}15{}}}{{{}.{}}}{{{}regionserver{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}hbase33a{}}}{{{}.{}}}{{{}hbase{}}}{{{}.{}}}{{{}monitoring{}}}{{{}.{}}}{{{}aws{}}}{{{}-{}}}{{{}esvc1{}}}{{{}-{}}}{{{}useast2{}}}{{{}.{}}}{{{}aws{}}}{{{}.{}}}{{{}sfdc{}}}{{{}.{}}}{{{}is{}}}{{{},{}}}{{{}60020{}}}{{{},{}}}{{{}1730878238028{}}}{{
> }}{{in}}{{ }}*{{3}}{{ }}{{{}hrs{}}}{{{}, {}}}{{16}}{{ }}{{{}mins{}}}{{{},
> {}}}{{{}52{}}}{{{}.{}}}{{{}806{}}}{{ }}{{sec}}*
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
