[jira] [Commented] (HBASE-23103) Survey incidence of table state queries
[ https://issues.apache.org/jira/browse/HBASE-23103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945251#comment-16945251 ] Michael Stack commented on HBASE-23103: --- bq. Master startup needs meta table online as we need to scan meta, and here want to disable meta. bq. ...we may still hit problems, for example, the ModifyTableProcedure is only half done and assume that meta table is disabled, but after restarting, meta table is enabled, and the procedure can not continue... So, master comes up, can read hbase:meta (though its state may be in-between) and we have a stuck procedure. MIght have to do some fixing w/ hbck2 to address this super-rare scenario (edit of meta is rate * failure of master during meta edit * fails in middle of cross-family edit). The alternative of snowflaking hbase:meta in at least three different locations makes for intrigue and complexity around sequences already hard to decipher. Any other ideas? Any ideas around meta state? Thanks [~zhangduo] > Survey incidence of table state queries > --- > > Key: HBASE-23103 > URL: https://issues.apache.org/jira/browse/HBASE-23103 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Blocker > Fix For: 3.0.0 > > > Task that comes of parent issue. Parent makes it so we go via Master to > figure state of a table. It is the authority and since the parent issues adds > being able to enable/disable hbase:meta, table state is now in two places -- > in hbase:meta table... and elsewhere for the hbase:meta's state. Rather than > have client go to two locations dependent on which table is being asked > about, parent made it so we went to master. Parent allows that this puts more > load on the Master. [~zhangduo] brings up the valid concern that it might be > too much or that dependent on the Master for state puts Master too much > in-line with read/writes. > This issue is a survey to figure how much load and how much state-in-master > could mess up inline read/writes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23103) Survey incidence of table state queries
[ https://issues.apache.org/jira/browse/HBASE-23103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16945030#comment-16945030 ] Duo Zhang commented on HBASE-23103: --- AsyncRpcRetryingCaller is the base class for retrying caller of normal read/write. And for altering meta, I do not think keep the state transient can solve all the problems. It has inherent complexity. Master startup needs meta table online as we need to scan meta, and here want to disable meta. Even if we automatically make meta enabled, we may still hit problems, for example, the ModifyTableProcedure is only half done and assume that meta table is disabled, but after restarting, meta table is enabled, and the procedure can not continue... As now, altering a table does not need to disable the table, I suggest we still keep meta always enabled, and add special logic to deal with meta table in ModifyTableProcedure, SplitTableRegionProcedure, MergeTableRegionsProcedure, etc. Thanks. > Survey incidence of table state queries > --- > > Key: HBASE-23103 > URL: https://issues.apache.org/jira/browse/HBASE-23103 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Blocker > Fix For: 3.0.0 > > > Task that comes of parent issue. Parent makes it so we go via Master to > figure state of a table. It is the authority and since the parent issues adds > being able to enable/disable hbase:meta, table state is now in two places -- > in hbase:meta table... and elsewhere for the hbase:meta's state. Rather than > have client go to two locations dependent on which table is being asked > about, parent made it so we went to master. Parent allows that this puts more > load on the Master. [~zhangduo] brings up the valid concern that it might be > too much or that dependent on the Master for state puts Master too much > in-line with read/writes. > This issue is a survey to figure how much load and how much state-in-master > could mess up inline read/writes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23103) Survey incidence of table state queries
[ https://issues.apache.org/jira/browse/HBASE-23103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944954#comment-16944954 ] Michael Stack commented on HBASE-23103: --- Or, if querying meta table state, MetaTableAccessor goes to ask the Master rather than ZK; then I could keep state transient such that if Master dies, we default back to enabled for hbase:meta. That might be better. Could also make it so Admin functions all go to Master for table state and only AsyncRpcRetryingCaller (and thrift) uses this new MTA exclusively. Something like that. > Survey incidence of table state queries > --- > > Key: HBASE-23103 > URL: https://issues.apache.org/jira/browse/HBASE-23103 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Blocker > Fix For: 3.0.0 > > > Task that comes of parent issue. Parent makes it so we go via Master to > figure state of a table. It is the authority and since the parent issues adds > being able to enable/disable hbase:meta, table state is now in two places -- > in hbase:meta table... and elsewhere for the hbase:meta's state. Rather than > have client go to two locations dependent on which table is being asked > about, parent made it so we went to master. Parent allows that this puts more > load on the Master. [~zhangduo] brings up the valid concern that it might be > too much or that dependent on the Master for state puts Master too much > in-line with read/writes. > This issue is a survey to figure how much load and how much state-in-master > could mess up inline read/writes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23103) Survey incidence of table state queries
[ https://issues.apache.org/jira/browse/HBASE-23103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944935#comment-16944935 ] Michael Stack commented on HBASE-23103: --- That make for a bunch of loading. Reviewing the code base, its only non-admin, non-local (i.e. call made by the Master) apart from a swath of calling done by thrift which seems wrong. Let me take a look... I suppose could do an alternative where MetaTableAccessor handles lookups for meta table itself. I could extend the meta table znode so it had a status too and the MTA could ping this when asked of the meta state. > Survey incidence of table state queries > --- > > Key: HBASE-23103 > URL: https://issues.apache.org/jira/browse/HBASE-23103 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Blocker > Fix For: 3.0.0 > > > Task that comes of parent issue. Parent makes it so we go via Master to > figure state of a table. It is the authority and since the parent issues adds > being able to enable/disable hbase:meta, table state is now in two places -- > in hbase:meta table... and elsewhere for the hbase:meta's state. Rather than > have client go to two locations dependent on which table is being asked > about, parent made it so we went to master. Parent allows that this puts more > load on the Master. [~zhangduo] brings up the valid concern that it might be > too much or that dependent on the Master for state puts Master too much > in-line with read/writes. > This issue is a survey to figure how much load and how much state-in-master > could mess up inline read/writes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23103) Survey incidence of table state queries
[ https://issues.apache.org/jira/browse/HBASE-23103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944528#comment-16944528 ] Duo Zhang commented on HBASE-23103: --- See here https://github.com/apache/hbase/blob/800c35a30e647ee6a87bab67b528e2634b38d0c4/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRpcRetryingCaller.java#L186 When hitting NotServingRegionException, we will check if the table is disabled. > Survey incidence of table state queries > --- > > Key: HBASE-23103 > URL: https://issues.apache.org/jira/browse/HBASE-23103 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Blocker > Fix For: 3.0.0 > > > Task that comes of parent issue. Parent makes it so we go via Master to > figure state of a table. It is the authority and since the parent issues adds > being able to enable/disable hbase:meta, table state is now in two places -- > in hbase:meta table... and elsewhere for the hbase:meta's state. Rather than > have client go to two locations dependent on which table is being asked > about, parent made it so we went to master. Parent allows that this puts more > load on the Master. [~zhangduo] brings up the valid concern that it might be > too much or that dependent on the Master for state puts Master too much > in-line with read/writes. > This issue is a survey to figure how much load and how much state-in-master > could mess up inline read/writes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23103) Survey incidence of table state queries
[ https://issues.apache.org/jira/browse/HBASE-23103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944142#comment-16944142 ] Michael Stack commented on HBASE-23103: --- Minor: TableState is cached in Master. Lookup is into a Map. The TableState is not cached when you go via MTA to the hbase:meta table. Each lookup does a full table scan of the table state column to find all states and then returns the state of the table asked for. > Survey incidence of table state queries > --- > > Key: HBASE-23103 > URL: https://issues.apache.org/jira/browse/HBASE-23103 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Blocker > Fix For: 3.0.0 > > > Task that comes of parent issue. Parent makes it so we go via Master to > figure state of a table. It is the authority and since the parent issues adds > being able to enable/disable hbase:meta, table state is now in two places -- > in hbase:meta table... and elsewhere for the hbase:meta's state. Rather than > have client go to two locations dependent on which table is being asked > about, parent made it so we went to master. Parent allows that this puts more > load on the Master. [~zhangduo] brings up the valid concern that it might be > too much or that dependent on the Master for state puts Master too much > in-line with read/writes. > This issue is a survey to figure how much load and how much state-in-master > could mess up inline read/writes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23103) Survey incidence of table state queries
[ https://issues.apache.org/jira/browse/HBASE-23103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944003#comment-16944003 ] Michael Stack commented on HBASE-23103: --- Did a rough survey. Incidence is close to zero where we go to ask table state; just when we are doing admin on table... which requires the Master. Will do some more... What about case where an errant user tool that keeps asking for table state. This would be user putting load on Master. Could warn against it? Otherwise, I suppose could update the hbase:meta zk to handle table state but then the client has to look in two places > Survey incidence of table state queries > --- > > Key: HBASE-23103 > URL: https://issues.apache.org/jira/browse/HBASE-23103 > Project: HBase > Issue Type: Sub-task >Reporter: Michael Stack >Priority: Blocker > Fix For: 3.0.0 > > > Task that comes of parent issue. Parent makes it so we go via Master to > figure state of a table. It is the authority and since the parent issues adds > being able to enable/disable hbase:meta, table state is now in two places -- > in hbase:meta table... and elsewhere for the hbase:meta's state. Rather than > have client go to two locations dependent on which table is being asked > about, parent made it so we went to master. Parent allows that this puts more > load on the Master. [~zhangduo] brings up the valid concern that it might be > too much or that dependent on the Master for state puts Master too much > in-line with read/writes. > This issue is a survey to figure how much load and how much state-in-master > could mess up inline read/writes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
