Works! I enabled async wal replication with the suggested option and ITBLL ran successfully.
generator step: hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList generator 15 1000000 /tmp/hbase-itbll verification step: hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList verify /tmp/hbase-itbll-verify 15 Both succeeded. I also confirmed that meta replicas are written and read by the clients, so must be in load balance mode. Thanks for the help! Andor > On 2022. Mar 27., at 6:59, Huaxiang Sun <huaxiang...@apache.org> wrote: > > It makes sense to turn on async wal replication when hbase.meta.replicas.use > = true. Let me run couple rounds of itbll with > hbase.region.replica.replication.catalog.enabled (lastest 2.4 and 2.5.0 > candidates) to get more confidence before proposing turn on async wal > replication for meta. > > Thanks, > Huaxiang > > On 2022/03/26 04:03:15 Andrew Purtell wrote: >> Just to be clear when I say "it seems pointless to have meta replicas which >> do not actually receive updates (by default)", what I should have said is >> 'timely updates', because a long delay in updating meta might as well be a >> missed update. >> >> On Fri, Mar 25, 2022 at 9:01 PM Andrew Purtell <apurt...@apache.org> wrote: >> >>>> "Async WAL replication for META is added as a new feature in 2.4.0. It >>> is still under active development. Use with caution. Set >>> hbase.region.replica.replication.catalog.enabled to enable async WAL >>> Replication for META region replicas. It is off by default." >>> >>> Do we still need this warning? >>> >>> Should hbase.region.replica.replication.catalog.enabled have a default of >>> 'true' (enabled) if hbase.meta.replicas.use = true ? Otherwise, it seems >>> pointless to have meta replicas which do not actually receive updates (by >>> default). >>> >>> >>> On Fri, Mar 25, 2022 at 10:51 AM Huaxiang Sun <huaxiang...@apache.org> >>> wrote: >>> >>>> Hi Andor, >>>> >>>> I get what you are saying. The HFile refreshing is the old way for >>>> replica regions to refresh hfiles periodically, default is 5 minutes. In >>>> this itbll case, we need to have the wal replication enabled for meta >>>> replica. Please check out, >>>> >>>> https://hbase.apache.org/book.html#_async_wal_replication_for_meta_table_as_of_hbase_2_4_0. >>>> Basically, you need to set >>>> "hbase.region.replica.replication.catalog.enabled" to true in the >>>> configuration and rerun itbll. Otherwise, all meta changes at the primary >>>> meta region wont be updated at the replica meta regions and it will result >>>> in itbll failures. >>>> >>>> Hope this helps, >>>> >>>> Huaxiang >>>> >>>> >>>> On 2022/03/25 13:46:42 Andor Molnar wrote: >>>>> Hi Huaxiang, >>>>> >>>>> We use 2.4.6 for the tests. >>>>> >>>>> I run itbll with the following command: >>>>> >>>>> hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList >>>> generator 15 1000000 /tmp/hbase-itbll >>>>> >>>>> for the generator step and essentially jobs have failed. We can see the >>>> meta request are spanning out to replicas, but writes start failing after >>>> this due to the stale cache which is not getting updated. >>>>> >>>>> Would you please tell me more about ‘hfile refresh’ and how to >>>> configure it? >>>>> >>>>> Thanks, >>>>> Andor >>>>> >>>>> >>>>> >>>>> >>>>>> On 2022. Mar 24., at 17:43, Huaxiang Sun <huaxiang...@apache.org> >>>> wrote: >>>>>> >>>>>> Hi Andor, >>>>>> >>>>>> Which 2.4 release do you test in your lab? We use this feature at >>>> production cluster with 2.4.5. >>>>>> At server side, we use hfile refresh instead of wal replication. I >>>> used to run itbll for each release with this feature enabled. How did you >>>> find the errors, did itbll fail? >>>>>> >>>>>> Regards, >>>>>> Huaxiang >>>>> >>>>> >>>> >>> >>> >>> -- >>> Best regards, >>> Andrew >>> >>> Unrest, ignorance distilled, nihilistic imbeciles - >>> It's what we’ve earned >>> Welcome, apocalypse, what’s taken you so long? >>> Bring us the fitting end that we’ve been counting on >>> - A23, Welcome, Apocalypse >>> >> >> >> -- >> Best regards, >> Andrew >> >> Unrest, ignorance distilled, nihilistic imbeciles - >> It's what we’ve earned >> Welcome, apocalypse, what’s taken you so long? >> Bring us the fitting end that we’ve been counting on >> - A23, Welcome, Apocalypse >>
