Thanks a lot Huaxiang! Sorry I didn’t have a chance to follow-up on this today, but hopefully come back to you tomorrow. Really appreciate your efforts.
Andor > On 2022. Mar 28., at 19:10, Huaxiang Sun <huaxiang...@apache.org> wrote: > > I will start to work on it. > > Thanks > Huaxiang > > On 2022/03/27 18:42:06 Andrew Purtell wrote: >> On Sat, Mar 26, 2022 at 10:00 PM Huaxiang Sun <huaxiang...@apache.org> >> wrote: >> >>> It makes sense to turn on async wal replication when >>> hbase.meta.replicas.use = true. Let me run couple rounds of itbll with >>> hbase.region.replica.replication.catalog.enabled (lastest 2.4 and 2.5.0 >>> candidates) to get more confidence before proposing turn on async wal >>> replication for meta. >>> >> >> +1 >> >> Thanks, >>> Huaxiang >>> >>> On 2022/03/26 04:03:15 Andrew Purtell wrote: >>>> Just to be clear when I say "it seems pointless to have meta replicas >>> which >>>> do not actually receive updates (by default)", what I should have said is >>>> 'timely updates', because a long delay in updating meta might as well be >>> a >>>> missed update. >>>> >>>> On Fri, Mar 25, 2022 at 9:01 PM Andrew Purtell <apurt...@apache.org> >>> wrote: >>>> >>>>>> "Async WAL replication for META is added as a new feature in 2.4.0. >>> It >>>>> is still under active development. Use with caution. Set >>>>> hbase.region.replica.replication.catalog.enabled to enable async WAL >>>>> Replication for META region replicas. It is off by default." >>>>> >>>>> Do we still need this warning? >>>>> >>>>> Should hbase.region.replica.replication.catalog.enabled have a default >>> of >>>>> 'true' (enabled) if hbase.meta.replicas.use = true ? Otherwise, it >>> seems >>>>> pointless to have meta replicas which do not actually receive updates >>> (by >>>>> default). >>>>> >>>>> >>>>> On Fri, Mar 25, 2022 at 10:51 AM Huaxiang Sun <huaxiang...@apache.org> >>>>> wrote: >>>>> >>>>>> Hi Andor, >>>>>> >>>>>> I get what you are saying. The HFile refreshing is the old way for >>>>>> replica regions to refresh hfiles periodically, default is 5 minutes. >>> In >>>>>> this itbll case, we need to have the wal replication enabled for meta >>>>>> replica. Please check out, >>>>>> >>>>>> >>> https://hbase.apache.org/book.html#_async_wal_replication_for_meta_table_as_of_hbase_2_4_0 >>> . >>>>>> Basically, you need to set >>>>>> "hbase.region.replica.replication.catalog.enabled" to true in the >>>>>> configuration and rerun itbll. Otherwise, all meta changes at the >>> primary >>>>>> meta region wont be updated at the replica meta regions and it will >>> result >>>>>> in itbll failures. >>>>>> >>>>>> Hope this helps, >>>>>> >>>>>> Huaxiang >>>>>> >>>>>> >>>>>> On 2022/03/25 13:46:42 Andor Molnar wrote: >>>>>>> Hi Huaxiang, >>>>>>> >>>>>>> We use 2.4.6 for the tests. >>>>>>> >>>>>>> I run itbll with the following command: >>>>>>> >>>>>>> hbase org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList >>>>>> generator 15 1000000 /tmp/hbase-itbll >>>>>>> >>>>>>> for the generator step and essentially jobs have failed. We can see >>> the >>>>>> meta request are spanning out to replicas, but writes start failing >>> after >>>>>> this due to the stale cache which is not getting updated. >>>>>>> >>>>>>> Would you please tell me more about ‘hfile refresh’ and how to >>>>>> configure it? >>>>>>> >>>>>>> Thanks, >>>>>>> Andor >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> On 2022. Mar 24., at 17:43, Huaxiang Sun <huaxiang...@apache.org> >>>>>> wrote: >>>>>>>> >>>>>>>> Hi Andor, >>>>>>>> >>>>>>>> Which 2.4 release do you test in your lab? We use this feature >>> at >>>>>> production cluster with 2.4.5. >>>>>>>> At server side, we use hfile refresh instead of wal replication. I >>>>>> used to run itbll for each release with this feature enabled. How did >>> you >>>>>> find the errors, did itbll fail? >>>>>>>> >>>>>>>> Regards, >>>>>>>> Huaxiang >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Best regards, >>>>> Andrew >>>>> >>>>> Unrest, ignorance distilled, nihilistic imbeciles - >>>>> It's what we’ve earned >>>>> Welcome, apocalypse, what’s taken you so long? >>>>> Bring us the fitting end that we’ve been counting on >>>>> - A23, Welcome, Apocalypse >>>>> >>>> >>>> >>>> -- >>>> Best regards, >>>> Andrew >>>> >>>> Unrest, ignorance distilled, nihilistic imbeciles - >>>> It's what we’ve earned >>>> Welcome, apocalypse, what’s taken you so long? >>>> Bring us the fitting end that we’ve been counting on >>>> - A23, Welcome, Apocalypse >>>> >>> >> >> >> -- >> Best regards, >> Andrew >> >> Unrest, ignorance distilled, nihilistic imbeciles - >> It's what we’ve earned >> Welcome, apocalypse, what’s taken you so long? >> Bring us the fitting end that we’ve been counting on >> - A23, Welcome, Apocalypse >>
