Re: Rolling upgrade from HBase 2.2.2 to 2.5.4: There are 2336 corrupted procedures

2024-03-25 Thread Duo Zhang
Is it OK if you just restart with the 2.2.2 master?

Adam Sjøgren  于2024年3月25日周一 20:14写道:
>
>   Hi,
>
> I am trying to do a rolling upgrade of an HBase 2.2.2 cluster to
> 2.5.4.
>
> When I try switching over to the new master, it processes the WAL
> files and then ends up saying:
>
>   [... many lines elided ...]
>
>   2024-03-25T12:44:29,374 ERROR [master/s4:16000:becomeActiveMaster]
>   region.RegionProcedureStore: Corrupted procedure pid=221160,
>   ppid=219241, state=RUNNABLE; OpenRegionProcedure
>   47b69716553a2e2587f22e0e7c3d268a, server=node13,16020,1640161141416
>
>   2024-03-25T12:44:29,374 ERROR [master/s4:16000:becomeActiveMaster]
>   region.RegionProcedureStore: Corrupted procedure pid=221161,
>   ppid=219258, state=RUNNABLE; OpenRegionProcedure
>   1071a48e8ab7730a160cf55391d1670a, server=node23,16020,1639980983954
>
>   2024-03-25T12:44:29,374 ERROR [master/s4:16000:becomeActiveMaster]
>   region.RegionProcedureStore: Corrupted procedure pid=4215645,
>   ppid=4215619, state=SUCCESS, bypass=true; OpenRegionProcedure
>   17c468cbf507e548b1c34c74b14b4c40, server=node09,16020,1700398035068
>
>   2024-03-25T12:44:29,375 ERROR
>   [master/s4:16000:becomeActiveMaster] master.HMaster: Failed to
>   become active master
>   java.io.IOException: There are 2336 corrupted procedures when
>   migrating from the old WAL based store to the new region based
>   store, please fix them before upgrading again.
>
> and then it quits.
>
> How do I fix these 2336 corrupted procedures?
>
>   Best regards,
>
> Adam
>
> --
>  "Why does the sky turn red as the sun sets?"   Adam Sjøgren
>  "That's all the oxygen in the atmosphere catching a...@koldfront.dk
>   fire."
>


Rolling upgrade from HBase 2.2.2 to 2.5.4: There are 2336 corrupted procedures

2024-03-25 Thread Adam Sjøgren
  Hi,

I am trying to do a rolling upgrade of an HBase 2.2.2 cluster to
2.5.4.

When I try switching over to the new master, it processes the WAL
files and then ends up saying:

  [... many lines elided ...]

  2024-03-25T12:44:29,374 ERROR [master/s4:16000:becomeActiveMaster]
  region.RegionProcedureStore: Corrupted procedure pid=221160,
  ppid=219241, state=RUNNABLE; OpenRegionProcedure
  47b69716553a2e2587f22e0e7c3d268a, server=node13,16020,1640161141416

  2024-03-25T12:44:29,374 ERROR [master/s4:16000:becomeActiveMaster]
  region.RegionProcedureStore: Corrupted procedure pid=221161,
  ppid=219258, state=RUNNABLE; OpenRegionProcedure
  1071a48e8ab7730a160cf55391d1670a, server=node23,16020,1639980983954

  2024-03-25T12:44:29,374 ERROR [master/s4:16000:becomeActiveMaster]
  region.RegionProcedureStore: Corrupted procedure pid=4215645,
  ppid=4215619, state=SUCCESS, bypass=true; OpenRegionProcedure
  17c468cbf507e548b1c34c74b14b4c40, server=node09,16020,1700398035068

  2024-03-25T12:44:29,375 ERROR
  [master/s4:16000:becomeActiveMaster] master.HMaster: Failed to
  become active master
  java.io.IOException: There are 2336 corrupted procedures when
  migrating from the old WAL based store to the new region based
  store, please fix them before upgrading again.

and then it quits.

How do I fix these 2336 corrupted procedures?

  Best regards,

Adam

-- 
 "Why does the sky turn red as the sun sets?"   Adam Sjøgren
 "That's all the oxygen in the atmosphere catching a...@koldfront.dk
  fire."