[
https://issues.apache.org/jira/browse/HBASE-28987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17934369#comment-17934369
]
Andor Molnar commented on HBASE-28987:
--------------------------------------
{quote}Or we can introduce a more general logic, that the replicate method just
means we send the things out, and before persisting, we should make sure that
all the edits which are sent out should be committed, if there are any errors
while committing, we throw a special exception and let the replication restart
from the previous position.
{quote}
I think I need more clarification, because I'm not sure when it's going to
happen exactly.
Currently ReplicationSourceShipper persists the log position
[here|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceShipper.java#L266].
It happens when
{code:java}
if (
batch.isEndOfFile() || !batch.getLastWalPath().equals(currentPath)
|| batch.getLastWalPosition() != currentPosition
) {
source.logPositionAndCleanOldLogs(batch);
updated = true;
} {code}
So, when batch reaches end of file or wal path changed or wal position changed.
Doesn't that happen every time the pointer moves?
If that's the case, are we going to call ReplicationEndpoint and check things
sent out properly (calling close() on S3 file) and persist the pointer if
succeeded?
This seems to be too frequent and will end up with too many small files.
> Developing a Custom ReplicationEndpoint to Support External Storage
> Integration
> -------------------------------------------------------------------------------
>
> Key: HBASE-28987
> URL: https://issues.apache.org/jira/browse/HBASE-28987
> Project: HBase
> Issue Type: Task
> Components: backup&restore
> Affects Versions: 2.6.0, 3.0.0-alpha-4
> Reporter: Vinayak Hegde
> Assignee: Vinayak Hegde
> Priority: Major
>
> *Develop a Custom Replication Endpoint*
> Implement a custom replication endpoint to support the backup of WALs to
> external storage systems, such as HDFS-compliant storages (including HDFS,
> S3, ADLS, and GCS via respective Hadoop connectors).
> *Support for Bulk-loaded Files*
> Add functionality to back up bulk-loaded files in addition to regular WALs.
> *Ensure Process Durability*
> Ensure the backup process is durable, with no WALs being missed, even in the
> event of issues in the cluster.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
