[
https://issues.apache.org/jira/browse/HBASE-28987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17911459#comment-17911459
]
Vinayak Hegde commented on HBASE-28987:
---------------------------------------
*Option 3: Optimized Offset Management with Dual Tracking*
h4. *Approach*
* Maintain {*}two offsets{*}:
## {*}Delivered Offset{*}: Tracks the last set of WAL entries successfully
sent to the replication endpoint.
## {*}Committed Offset{*}: Tracks the last set of WAL entries confirmed as
persisted to the backup storage (e.g., S3).
* {*}Process{*}:
## The {{replicate()}} method sends entries to the replication endpoint and
updates the *delivered offset* in RegionServer memory.
## The *committed offset* is updated only after the WAL file is successfully
closed, ensuring all entries in the file are written and persisted to the
storage location.
## In case of failure, entries between the *delivered offset* and the
*committed offset* are retried.
* {*}Failure Handling{*}:
** WAL entries that fail to persist to storage are retried without affecting
previously committed data.
** On server restarts or node failures, the system reprocesses WAL entries
starting from the {*}committed offset{*}.
h4. *Considerations*
* Modifications to the replication framework are required to support
dual-offset tracking and handle three distinct states for {{{}replicate(){}}}:
** {*}Failed{*}: Retry entries.
** {*}Delivered{*}: Entries sent but not yet confirmed as persisted.
** {*}Committed{*}: Entries fully persisted to the storage location.
* The framework must ensure memory tracking for *delivered offsets* and
persistence for {*}committed offsets{*}.
> Developing a Custom ReplicationEndpoint to Support External Storage
> Integration
> -------------------------------------------------------------------------------
>
> Key: HBASE-28987
> URL: https://issues.apache.org/jira/browse/HBASE-28987
> Project: HBase
> Issue Type: Task
> Components: backup&restore
> Affects Versions: 2.6.0, 3.0.0-alpha-4
> Reporter: Vinayak Hegde
> Assignee: Vinayak Hegde
> Priority: Major
>
> *Develop a Custom Replication Endpoint*
> Implement a custom replication endpoint to support the backup of WALs to
> external storage systems, such as HDFS-compliant storages (including HDFS,
> S3, ADLS, and GCS via respective Hadoop connectors).
> *Support for Bulk-loaded Files*
> Add functionality to back up bulk-loaded files in addition to regular WALs.
> *Ensure Process Durability*
> Ensure the backup process is durable, with no WALs being missed, even in the
> event of issues in the cluster.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
