[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16560160#comment-16560160
]
Josh Elser commented on HBASE-20952:
------------------------------------
Talked through this a little bit today with [[email protected]], trying to
break it down into consumable pieces.
* Look at prior art (e.g. Apache Kafka and DistributedLog) to come up with
ideal API (e.g. RATIS-272)
* Go through each "system" in HBase to document how they use WALs, and try to
find an "ideal" HBase API
** Replication
** Backup and restore
** write path
** .. other?
Other tenets we called out:
* Asynchronous API is desired
* Easily supporting group-commit/multiple writers being batched into one "sync"
* Must provide "tail" (hard part will be implementing this for
FSHLog/AsyncFSWal – maybe two interfaces would be good, an extension that
supports tailing?)
Some other things here:
* Look at WALEdit/WALKey as Stack asked
Once we have these pieces written down, I think we should be able to start
finding a middle ground between it all.
> Re-visit the WAL API
> --------------------
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Sub-task
> Components: wal
> Reporter: Josh Elser
> Priority: Major
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup&restore. Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B&R doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
