[
https://issues.apache.org/jira/browse/HBASE-29064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17931539#comment-17931539
]
Andor Molnar commented on HBASE-29064:
--------------------------------------
I think we already discussed this on the other ticket, but let me repeat here:
Hadoop S3A framework doesn't allow streaming of data and there's no flush
option in the API. The only thing we can do to make sure that data is persisted
in the cloud, is to close() the object completely and create a new one.
We'd like to support this behaviour with the ticket.
> POC: Implement Dual Offset Tracking for WAL Replication
> -------------------------------------------------------
>
> Key: HBASE-29064
> URL: https://issues.apache.org/jira/browse/HBASE-29064
> Project: HBase
> Issue Type: Task
> Reporter: Vinayak Hegde
> Assignee: Andor Molnar
> Priority: Major
> Labels: pull-request-available
>
> The current replication framework updates the WAL offset immediately after
> processing entries, which may lead to data loss in case of partial writes or
> failures before persisting the data to the backup storage (e.g., S3).
> To address this:
> # {*}Delivered Offset{*}: Tracks entries sent to the replication endpoint.
> # {*}Committed Offset{*}: Tracks entries fully persisted to the backup
> storage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
