I doubt the procedure would finish, considering it has run for so long. You can check the tail of master log to see if it is stuck. If it is stuck, see if you can use abort_procedure.rb to stop.
After the procedure is stopped, see if running hbck can fix the issue (I haven't worked with 1.3 release in production). When running hbck, run without -fix parameter first to see what inconsistencies hbck reports. Cheers On Wed, Aug 29, 2018 at 3:42 PM Antonio Si <[email protected]> wrote: > Forgot to mention that all regions of the table is offline now. Wondering > if the table will eventually got disable as it has been running for almost > 24 hrs now. > > Thanks. > > Antonio. > > On Wed, Aug 29, 2018 at 3:40 PM Antonio Si <[email protected]> wrote: > > > Thanks Ted. > > Now that the table is in neither disable or enable state, will the table > > eventually got disable completely? > > From the "Procedure" tab of the hbase ui, I see the "disable" is still > > running. > > > > Thanks. > > > > Antonio. > > > > On Wed, Aug 29, 2018 at 3:31 PM Ted Yu <[email protected]> wrote: > > > >> The 'missing table descriptor' error should have been fixed by running > >> hbck > >> (with selected parameters). > >> > >> FYI > >> > >> On Wed, Aug 29, 2018 at 2:46 PM Antonio Si <[email protected]> > wrote: > >> > >> > Thanks Ted. > >> > > >> > The log says "java.io.IOException: missing table descriptor for > >> > ba912582f295f7ac0b83e7e419351602 > >> > > >> > [AM.ZK.Worker-pool2-t6552] master.RegionStates: Failed to open/close > >> > ba912582f295f7ac0b83e7e419351602 set to FAILED_OPEN" > >> > > >> > > >> > The version of hbase is 1.3.1 > >> > > >> > > >> > Thanks. > >> > > >> > > >> > Antonio. > >> > > >> > On Wed, Aug 29, 2018 at 2:28 PM Ted Yu <[email protected]> wrote: > >> > > >> > > Do you have access to master / region logs for when FAILED_OPEN > state > >> was > >> > > noticed ? > >> > > > >> > > There should be some hint there as to why some region couldn't open. > >> > > > >> > > The length of table DDL is related to number of regions the table > has. > >> > But > >> > > the length should be less related to data amount. > >> > > > >> > > Which version of hbase are you using ? > >> > > > >> > > Thanks > >> > > > >> > > On Wed, Aug 29, 2018 at 2:22 PM Antonio Si <[email protected]> > >> wrote: > >> > > > >> > > > Hi, > >> > > > > >> > > > We have a table which is stuck in FAILED_OPEN state. So, we > planned > >> to > >> > > drop > >> > > > the table and re-clone the table from an old snapshot. We disabled > >> the > >> > > > table, but the disable procedure has been running for more than 20 > >> hrs. > >> > > > > >> > > > I went to hbase shell and found out "is_disabled" and "is_enabled" > >> both > >> > > > return false. Is that a normal behavior since the table is in the > >> > middle > >> > > of > >> > > > being disabled? > >> > > > > >> > > > Is it normal that the disable took that many hours even though the > >> > table > >> > > is > >> > > > large in size (about 33TB)? > >> > > > > >> > > > Thanks. > >> > > > > >> > > > Antonio. > >> > > > > >> > > > >> > > >> > > >
