Re: a table is neither disable or enable
Thanks Ted. Antonio. On Wed, Aug 29, 2018 at 4:00 PM Ted Yu wrote: > I doubt the procedure would finish, considering it has run for so long. > > You can check the tail of master log to see if it is stuck. > If it is stuck, see if you can use abort_procedure.rb to stop. > > After the procedure is stopped, see if running hbck can fix the issue (I > haven't worked with 1.3 release in production). > When running hbck, run without -fix parameter first to see what > inconsistencies hbck reports. > > Cheers > > On Wed, Aug 29, 2018 at 3:42 PM Antonio Si wrote: > > > Forgot to mention that all regions of the table is offline now. Wondering > > if the table will eventually got disable as it has been running for > almost > > 24 hrs now. > > > > Thanks. > > > > Antonio. > > > > On Wed, Aug 29, 2018 at 3:40 PM Antonio Si wrote: > > > > > Thanks Ted. > > > Now that the table is in neither disable or enable state, will the > table > > > eventually got disable completely? > > > From the "Procedure" tab of the hbase ui, I see the "disable" is still > > > running. > > > > > > Thanks. > > > > > > Antonio. > > > > > > On Wed, Aug 29, 2018 at 3:31 PM Ted Yu wrote: > > > > > >> The 'missing table descriptor' error should have been fixed by running > > >> hbck > > >> (with selected parameters). > > >> > > >> FYI > > >> > > >> On Wed, Aug 29, 2018 at 2:46 PM Antonio Si > > wrote: > > >> > > >> > Thanks Ted. > > >> > > > >> > The log says "java.io.IOException: missing table descriptor for > > >> > ba912582f295f7ac0b83e7e419351602 > > >> > > > >> > [AM.ZK.Worker-pool2-t6552] master.RegionStates: Failed to open/close > > >> > ba912582f295f7ac0b83e7e419351602 set to FAILED_OPEN" > > >> > > > >> > > > >> > The version of hbase is 1.3.1 > > >> > > > >> > > > >> > Thanks. > > >> > > > >> > > > >> > Antonio. > > >> > > > >> > On Wed, Aug 29, 2018 at 2:28 PM Ted Yu wrote: > > >> > > > >> > > Do you have access to master / region logs for when FAILED_OPEN > > state > > >> was > > >> > > noticed ? > > >> > > > > >> > > There should be some hint there as to why some region couldn't > open. > > >> > > > > >> > > The length of table DDL is related to number of regions the table > > has. > > >> > But > > >> > > the length should be less related to data amount. > > >> > > > > >> > > Which version of hbase are you using ? > > >> > > > > >> > > Thanks > > >> > > > > >> > > On Wed, Aug 29, 2018 at 2:22 PM Antonio Si > > >> wrote: > > >> > > > > >> > > > Hi, > > >> > > > > > >> > > > We have a table which is stuck in FAILED_OPEN state. So, we > > planned > > >> to > > >> > > drop > > >> > > > the table and re-clone the table from an old snapshot. We > disabled > > >> the > > >> > > > table, but the disable procedure has been running for more than > 20 > > >> hrs. > > >> > > > > > >> > > > I went to hbase shell and found out "is_disabled" and > "is_enabled" > > >> both > > >> > > > return false. Is that a normal behavior since the table is in > the > > >> > middle > > >> > > of > > >> > > > being disabled? > > >> > > > > > >> > > > Is it normal that the disable took that many hours even though > the > > >> > table > > >> > > is > > >> > > > large in size (about 33TB)? > > >> > > > > > >> > > > Thanks. > > >> > > > > > >> > > > Antonio. > > >> > > > > > >> > > > > >> > > > >> > > > > > >
Re: a table is neither disable or enable
I doubt the procedure would finish, considering it has run for so long. You can check the tail of master log to see if it is stuck. If it is stuck, see if you can use abort_procedure.rb to stop. After the procedure is stopped, see if running hbck can fix the issue (I haven't worked with 1.3 release in production). When running hbck, run without -fix parameter first to see what inconsistencies hbck reports. Cheers On Wed, Aug 29, 2018 at 3:42 PM Antonio Si wrote: > Forgot to mention that all regions of the table is offline now. Wondering > if the table will eventually got disable as it has been running for almost > 24 hrs now. > > Thanks. > > Antonio. > > On Wed, Aug 29, 2018 at 3:40 PM Antonio Si wrote: > > > Thanks Ted. > > Now that the table is in neither disable or enable state, will the table > > eventually got disable completely? > > From the "Procedure" tab of the hbase ui, I see the "disable" is still > > running. > > > > Thanks. > > > > Antonio. > > > > On Wed, Aug 29, 2018 at 3:31 PM Ted Yu wrote: > > > >> The 'missing table descriptor' error should have been fixed by running > >> hbck > >> (with selected parameters). > >> > >> FYI > >> > >> On Wed, Aug 29, 2018 at 2:46 PM Antonio Si > wrote: > >> > >> > Thanks Ted. > >> > > >> > The log says "java.io.IOException: missing table descriptor for > >> > ba912582f295f7ac0b83e7e419351602 > >> > > >> > [AM.ZK.Worker-pool2-t6552] master.RegionStates: Failed to open/close > >> > ba912582f295f7ac0b83e7e419351602 set to FAILED_OPEN" > >> > > >> > > >> > The version of hbase is 1.3.1 > >> > > >> > > >> > Thanks. > >> > > >> > > >> > Antonio. > >> > > >> > On Wed, Aug 29, 2018 at 2:28 PM Ted Yu wrote: > >> > > >> > > Do you have access to master / region logs for when FAILED_OPEN > state > >> was > >> > > noticed ? > >> > > > >> > > There should be some hint there as to why some region couldn't open. > >> > > > >> > > The length of table DDL is related to number of regions the table > has. > >> > But > >> > > the length should be less related to data amount. > >> > > > >> > > Which version of hbase are you using ? > >> > > > >> > > Thanks > >> > > > >> > > On Wed, Aug 29, 2018 at 2:22 PM Antonio Si > >> wrote: > >> > > > >> > > > Hi, > >> > > > > >> > > > We have a table which is stuck in FAILED_OPEN state. So, we > planned > >> to > >> > > drop > >> > > > the table and re-clone the table from an old snapshot. We disabled > >> the > >> > > > table, but the disable procedure has been running for more than 20 > >> hrs. > >> > > > > >> > > > I went to hbase shell and found out "is_disabled" and "is_enabled" > >> both > >> > > > return false. Is that a normal behavior since the table is in the > >> > middle > >> > > of > >> > > > being disabled? > >> > > > > >> > > > Is it normal that the disable took that many hours even though the > >> > table > >> > > is > >> > > > large in size (about 33TB)? > >> > > > > >> > > > Thanks. > >> > > > > >> > > > Antonio. > >> > > > > >> > > > >> > > >> > > >
Re: a table is neither disable or enable
Forgot to mention that all regions of the table is offline now. Wondering if the table will eventually got disable as it has been running for almost 24 hrs now. Thanks. Antonio. On Wed, Aug 29, 2018 at 3:40 PM Antonio Si wrote: > Thanks Ted. > Now that the table is in neither disable or enable state, will the table > eventually got disable completely? > From the "Procedure" tab of the hbase ui, I see the "disable" is still > running. > > Thanks. > > Antonio. > > On Wed, Aug 29, 2018 at 3:31 PM Ted Yu wrote: > >> The 'missing table descriptor' error should have been fixed by running >> hbck >> (with selected parameters). >> >> FYI >> >> On Wed, Aug 29, 2018 at 2:46 PM Antonio Si wrote: >> >> > Thanks Ted. >> > >> > The log says "java.io.IOException: missing table descriptor for >> > ba912582f295f7ac0b83e7e419351602 >> > >> > [AM.ZK.Worker-pool2-t6552] master.RegionStates: Failed to open/close >> > ba912582f295f7ac0b83e7e419351602 set to FAILED_OPEN" >> > >> > >> > The version of hbase is 1.3.1 >> > >> > >> > Thanks. >> > >> > >> > Antonio. >> > >> > On Wed, Aug 29, 2018 at 2:28 PM Ted Yu wrote: >> > >> > > Do you have access to master / region logs for when FAILED_OPEN state >> was >> > > noticed ? >> > > >> > > There should be some hint there as to why some region couldn't open. >> > > >> > > The length of table DDL is related to number of regions the table has. >> > But >> > > the length should be less related to data amount. >> > > >> > > Which version of hbase are you using ? >> > > >> > > Thanks >> > > >> > > On Wed, Aug 29, 2018 at 2:22 PM Antonio Si >> wrote: >> > > >> > > > Hi, >> > > > >> > > > We have a table which is stuck in FAILED_OPEN state. So, we planned >> to >> > > drop >> > > > the table and re-clone the table from an old snapshot. We disabled >> the >> > > > table, but the disable procedure has been running for more than 20 >> hrs. >> > > > >> > > > I went to hbase shell and found out "is_disabled" and "is_enabled" >> both >> > > > return false. Is that a normal behavior since the table is in the >> > middle >> > > of >> > > > being disabled? >> > > > >> > > > Is it normal that the disable took that many hours even though the >> > table >> > > is >> > > > large in size (about 33TB)? >> > > > >> > > > Thanks. >> > > > >> > > > Antonio. >> > > > >> > > >> > >> >
Re: a table is neither disable or enable
Thanks Ted. Now that the table is in neither disable or enable state, will the table eventually got disable completely? >From the "Procedure" tab of the hbase ui, I see the "disable" is still running. Thanks. Antonio. On Wed, Aug 29, 2018 at 3:31 PM Ted Yu wrote: > The 'missing table descriptor' error should have been fixed by running hbck > (with selected parameters). > > FYI > > On Wed, Aug 29, 2018 at 2:46 PM Antonio Si wrote: > > > Thanks Ted. > > > > The log says "java.io.IOException: missing table descriptor for > > ba912582f295f7ac0b83e7e419351602 > > > > [AM.ZK.Worker-pool2-t6552] master.RegionStates: Failed to open/close > > ba912582f295f7ac0b83e7e419351602 set to FAILED_OPEN" > > > > > > The version of hbase is 1.3.1 > > > > > > Thanks. > > > > > > Antonio. > > > > On Wed, Aug 29, 2018 at 2:28 PM Ted Yu wrote: > > > > > Do you have access to master / region logs for when FAILED_OPEN state > was > > > noticed ? > > > > > > There should be some hint there as to why some region couldn't open. > > > > > > The length of table DDL is related to number of regions the table has. > > But > > > the length should be less related to data amount. > > > > > > Which version of hbase are you using ? > > > > > > Thanks > > > > > > On Wed, Aug 29, 2018 at 2:22 PM Antonio Si > wrote: > > > > > > > Hi, > > > > > > > > We have a table which is stuck in FAILED_OPEN state. So, we planned > to > > > drop > > > > the table and re-clone the table from an old snapshot. We disabled > the > > > > table, but the disable procedure has been running for more than 20 > hrs. > > > > > > > > I went to hbase shell and found out "is_disabled" and "is_enabled" > both > > > > return false. Is that a normal behavior since the table is in the > > middle > > > of > > > > being disabled? > > > > > > > > Is it normal that the disable took that many hours even though the > > table > > > is > > > > large in size (about 33TB)? > > > > > > > > Thanks. > > > > > > > > Antonio. > > > > > > > > > >
Re: a table is neither disable or enable
The 'missing table descriptor' error should have been fixed by running hbck (with selected parameters). FYI On Wed, Aug 29, 2018 at 2:46 PM Antonio Si wrote: > Thanks Ted. > > The log says "java.io.IOException: missing table descriptor for > ba912582f295f7ac0b83e7e419351602 > > [AM.ZK.Worker-pool2-t6552] master.RegionStates: Failed to open/close > ba912582f295f7ac0b83e7e419351602 set to FAILED_OPEN" > > > The version of hbase is 1.3.1 > > > Thanks. > > > Antonio. > > On Wed, Aug 29, 2018 at 2:28 PM Ted Yu wrote: > > > Do you have access to master / region logs for when FAILED_OPEN state was > > noticed ? > > > > There should be some hint there as to why some region couldn't open. > > > > The length of table DDL is related to number of regions the table has. > But > > the length should be less related to data amount. > > > > Which version of hbase are you using ? > > > > Thanks > > > > On Wed, Aug 29, 2018 at 2:22 PM Antonio Si wrote: > > > > > Hi, > > > > > > We have a table which is stuck in FAILED_OPEN state. So, we planned to > > drop > > > the table and re-clone the table from an old snapshot. We disabled the > > > table, but the disable procedure has been running for more than 20 hrs. > > > > > > I went to hbase shell and found out "is_disabled" and "is_enabled" both > > > return false. Is that a normal behavior since the table is in the > middle > > of > > > being disabled? > > > > > > Is it normal that the disable took that many hours even though the > table > > is > > > large in size (about 33TB)? > > > > > > Thanks. > > > > > > Antonio. > > > > > >
Re: a table is neither disable or enable
Thanks Ted. The log says "java.io.IOException: missing table descriptor for ba912582f295f7ac0b83e7e419351602 [AM.ZK.Worker-pool2-t6552] master.RegionStates: Failed to open/close ba912582f295f7ac0b83e7e419351602 set to FAILED_OPEN" The version of hbase is 1.3.1 Thanks. Antonio. On Wed, Aug 29, 2018 at 2:28 PM Ted Yu wrote: > Do you have access to master / region logs for when FAILED_OPEN state was > noticed ? > > There should be some hint there as to why some region couldn't open. > > The length of table DDL is related to number of regions the table has. But > the length should be less related to data amount. > > Which version of hbase are you using ? > > Thanks > > On Wed, Aug 29, 2018 at 2:22 PM Antonio Si wrote: > > > Hi, > > > > We have a table which is stuck in FAILED_OPEN state. So, we planned to > drop > > the table and re-clone the table from an old snapshot. We disabled the > > table, but the disable procedure has been running for more than 20 hrs. > > > > I went to hbase shell and found out "is_disabled" and "is_enabled" both > > return false. Is that a normal behavior since the table is in the middle > of > > being disabled? > > > > Is it normal that the disable took that many hours even though the table > is > > large in size (about 33TB)? > > > > Thanks. > > > > Antonio. > > >
Re: a table is neither disable or enable
Do you have access to master / region logs for when FAILED_OPEN state was noticed ? There should be some hint there as to why some region couldn't open. The length of table DDL is related to number of regions the table has. But the length should be less related to data amount. Which version of hbase are you using ? Thanks On Wed, Aug 29, 2018 at 2:22 PM Antonio Si wrote: > Hi, > > We have a table which is stuck in FAILED_OPEN state. So, we planned to drop > the table and re-clone the table from an old snapshot. We disabled the > table, but the disable procedure has been running for more than 20 hrs. > > I went to hbase shell and found out "is_disabled" and "is_enabled" both > return false. Is that a normal behavior since the table is in the middle of > being disabled? > > Is it normal that the disable took that many hours even though the table is > large in size (about 33TB)? > > Thanks. > > Antonio. >
a table is neither disable or enable
Hi, We have a table which is stuck in FAILED_OPEN state. So, we planned to drop the table and re-clone the table from an old snapshot. We disabled the table, but the disable procedure has been running for more than 20 hrs. I went to hbase shell and found out "is_disabled" and "is_enabled" both return false. Is that a normal behavior since the table is in the middle of being disabled? Is it normal that the disable took that many hours even though the table is large in size (about 33TB)? Thanks. Antonio.
