Hi Josh, I’m interested in this, too. I’ll start to read and follow what you suggested to Jack.
Thanks, Toshi > On Aug 7, 2018, at 09:12, Jack Bearden <[email protected]> wrote: > > Thanks Josh that was helpful. I'll start doing some of my own research > around these topics and look into that Ratis ticket. Much appreciated! > > On Mon, Aug 6, 2018 at 8:04 AM, Josh Elser <[email protected]> wrote: > >> Yup, replication is a big one to "unravel". Repeating myself from a branch >> in the thread, but I'd expect some initial suggestions on what a new API >> could be this week. Certainly the first draft won't be the final -- would >> be great to get your input after your AsyncWAL work, Duo. >> >> Using AWS SimpleQueryService, or much anything else, would be great. I >> want to make sure that, while we try to "scratch this one itch", we pave >> the way for whatever else folks want to experiment with. >> >> >> On 8/4/18 5:10 AM, 张铎(Duo Zhang) wrote: >> >>> Yes, maybe we could write WAL to SQS and HFile to S3, then we can deploy >>> HBase on AWS without any local storage volumes... >>> >>> But we also need a good abstraction for Replication, as the current design >>> is file based... >>> >>> 2018-07-27 1:28 GMT+08:00 Zach York <[email protected]>: >>> >>> I would REALLY hope that the WAL interface/API changes would go into >>>> master >>>> even if the feature work for Ratis is going in a feature branch. Not only >>>> would this enable other backends to be developed in parallel with the >>>> Ratis >>>> solution if there are other good fits for a non-HDFS WAL, but also it >>>> would >>>> save the burden of having to rebase these core changes onto the latest >>>> master to maintain compatibility. I'm assuming the Ratis portion of the >>>> code would be mostly new files so these would be less of a concern. >>>> >>>> On Thu, Jul 26, 2018 at 9:24 AM, Josh Elser <[email protected]> wrote: >>>> >>>> On 7/26/18 1:00 AM, Stack wrote: >>>>> >>>>> All this said, I'd like to start moving toward the point where we start >>>>>> >>>>>>> breaking out this work into a feature-branch off of master and start >>>>>>> building code. My hope is that this is amenable to everyone, with the >>>>>>> acknowledge that the Ratis work is considered "experimental" and not >>>>>>> an >>>>>>> attempt to make all of HBase use Ratis-backed WALs. >>>>>>> >>>>>>> >>>>>>> Go for it. >>>>>>> >>>>>> >>>>>> The branch would have WAL API changes only or would it include Ratis >>>>>> WAL >>>>>> dev? (If the latter, would that be better done over on Ratis project?). >>>>>> >>>>>> >>>>> I think we would start with WAL API changes, get those "blessed", and >>>>> >>>> then >>>> >>>>> continue Ratis WAL dev after that. >>>>> >>>>> >>>> >>>
