Yup, replication is a big one to "unravel". Repeating myself from a
branch in the thread, but I'd expect some initial suggestions on what a
new API could be this week. Certainly the first draft won't be the final
-- would be great to get your input after your AsyncWAL work, Duo.
Using AWS SimpleQueryService, or much anything else, would be great. I
want to make sure that, while we try to "scratch this one itch", we pave
the way for whatever else folks want to experiment with.
On 8/4/18 5:10 AM, 张铎(Duo Zhang) wrote:
Yes, maybe we could write WAL to SQS and HFile to S3, then we can deploy
HBase on AWS without any local storage volumes...
But we also need a good abstraction for Replication, as the current design
is file based...
2018-07-27 1:28 GMT+08:00 Zach York <[email protected]>:
I would REALLY hope that the WAL interface/API changes would go into master
even if the feature work for Ratis is going in a feature branch. Not only
would this enable other backends to be developed in parallel with the Ratis
solution if there are other good fits for a non-HDFS WAL, but also it would
save the burden of having to rebase these core changes onto the latest
master to maintain compatibility. I'm assuming the Ratis portion of the
code would be mostly new files so these would be less of a concern.
On Thu, Jul 26, 2018 at 9:24 AM, Josh Elser <[email protected]> wrote:
On 7/26/18 1:00 AM, Stack wrote:
All this said, I'd like to start moving toward the point where we start
breaking out this work into a feature-branch off of master and start
building code. My hope is that this is amenable to everyone, with the
acknowledge that the Ratis work is considered "experimental" and not an
attempt to make all of HBase use Ratis-backed WALs.
Go for it.
The branch would have WAL API changes only or would it include Ratis WAL
dev? (If the latter, would that be better done over on Ratis project?).
I think we would start with WAL API changes, get those "blessed", and
then
continue Ratis WAL dev after that.
