[
https://issues.apache.org/jira/browse/HBASE-28987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17914289#comment-17914289
]
Duo Zhang commented on HBASE-28987:
-----------------------------------
OK, here I know why you guys want to implement HBASE-28987.
I think this could work, we just need manage the delivered offset in memory and
once there are failures, we resatrt from the last committed offset and reset
the delivered offset.
But I think the problem here is that, usually we do not want to store very
small files on S3.
One possible solution is to close the S3 file when closing the WAL file. But it
could take a very long time and if the region server crashes, the S3 file will
be dropped too and we need to resend it. Then maybe a better solution is to
just upload closed WAL files directly to S3, instead of relying on the tailing
of WAL files...
Another way is to close the S3 file on demand, maybe every 5 minutes? Then we
need to implement compaction, of course this could be implement outside the
regionserver process but anyway, it will increase the complexity.
For me, I prefer we just reuse the replication framework and send the entries
to a message queue, with a customized replication endpoint. I guess here we do
not need to rely on some advanced features so we can provide several different
MQ implementations, like kafka, pulsar, kinesis, etc. In this way it will be
very flexible as you are free to subscribe the data from the MQ and do anything
you want to do.
WDYT guy think?
Thanks.
> Developing a Custom ReplicationEndpoint to Support External Storage
> Integration
> -------------------------------------------------------------------------------
>
> Key: HBASE-28987
> URL: https://issues.apache.org/jira/browse/HBASE-28987
> Project: HBase
> Issue Type: Task
> Components: backup&restore
> Affects Versions: 2.6.0, 3.0.0-alpha-4
> Reporter: Vinayak Hegde
> Assignee: Vinayak Hegde
> Priority: Major
>
> *Develop a Custom Replication Endpoint*
> Implement a custom replication endpoint to support the backup of WALs to
> external storage systems, such as HDFS-compliant storages (including HDFS,
> S3, ADLS, and GCS via respective Hadoop connectors).
> *Support for Bulk-loaded Files*
> Add functionality to back up bulk-loaded files in addition to regular WALs.
> *Ensure Process Durability*
> Ensure the backup process is durable, with no WALs being missed, even in the
> event of issues in the cluster.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
