[jira] [Commented] (HBASE-26890) Make the WAL interface async so sync replication can be built up on the WAL interface
[ https://issues.apache.org/jira/browse/HBASE-26890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17733148#comment-17733148 ] Duo Zhang commented on HBASE-26890: --- OK, the marker edits only is for transiting from ACTIVE to STANDBY, and check the code, seems the hacking is OK, we still use the createWriterInstance method in AsyncFSWAL to create AsyncWriter, so it will not block new WAL implementation... > Make the WAL interface async so sync replication can be built up on the WAL > interface > - > > Key: HBASE-26890 > URL: https://issues.apache.org/jira/browse/HBASE-26890 > Project: HBase > Issue Type: Improvement > Components: Replication, wal >Reporter: Duo Zhang >Priority: Major > > Instead of hacking into the WAL implementation. > This could make the implementation more general if later we want to change > the WAL implementation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26890) Make the WAL interface async so sync replication can be built up on the WAL interface
[ https://issues.apache.org/jira/browse/HBASE-26890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17733142#comment-17733142 ] Duo Zhang commented on HBASE-26890: --- After investigating, I do not think this is the correct way to implement sync replication. In the WAL implementation, we have also down sequence id accounting, and it is highly related to how we roll the WAL, so it is impossible for us to have two WAL instances for the same region, as we can not have two different sequence id accountings... So the current abstraction layer is correct, and also for new WAL implementations, they should just implement a new WAL.AsyncWriter(and a WAL.Writer maybe), then they could just reuse all the code in AsyncFSWAL. We could use URI instead of Path to describe a WAL entry, it will be more general. And for sync replication, the only thing is that, it can only work with AsyncWriter, maybe we could make it more general so we do not need to hack into AsyncFSWAL, just pass something like an AsyncWriterFactory is fine. The only thing for now is that, while reopening a region, we need to write some markers out so we have done some hacks in AsyncFSWAL to only allow marker edits out. But I do not think this is necessary, we will only reopen regions when transiting state from ACTIVE to DOWNGRADE_ACTIVE, where we know that we will not be in sync with the peer cluster, so it is OK for us to also write data edits out to the source cluster, next time when we want to convert back to ACTIVE, the async replication will replicate the out to the peer cluster, so there is no problem. Let me file an issue to see if I can make the implementation more general. Thanks. > Make the WAL interface async so sync replication can be built up on the WAL > interface > - > > Key: HBASE-26890 > URL: https://issues.apache.org/jira/browse/HBASE-26890 > Project: HBase > Issue Type: Improvement > Components: Replication, wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > Instead of hacking into the WAL implementation. > This could make the implementation more general if later we want to change > the WAL implementation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26890) Make the WAL interface async so sync replication can be built up on the WAL interface
[ https://issues.apache.org/jira/browse/HBASE-26890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17725443#comment-17725443 ] Duo Zhang commented on HBASE-26890: --- Plan to work on HBASE-20952 again, to build a WAL implementation without relying on external services, so then this one becomes a blocker.. Need to address this first. > Make the WAL interface async so sync replication can be built up on the WAL > interface > - > > Key: HBASE-26890 > URL: https://issues.apache.org/jira/browse/HBASE-26890 > Project: HBase > Issue Type: Improvement > Components: Replication, wal >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > Instead of hacking into the WAL implementation. > This could make the implementation more general if later we want to change > the WAL implementation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HBASE-26890) Make the WAL interface async so sync replication can be built up on the WAL interface
[ https://issues.apache.org/jira/browse/HBASE-26890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17512733#comment-17512733 ] Duo Zhang commented on HBASE-26890: --- This is related to the WAL refactor issue. We should build sync replication upon the WAL interface, otherwise once we want to change the WAL implementation, sync replication will be broken. Of course there will be new issues if we want to build sync replication upon WAL interface, for example, now we can make sure that the files are always the same for local and remote WALs, as they share the same WAL instance and roll at the same time, but if they are two WAL instances, it is not easy to archive this. So when deleting remote WAL files when replicating, we need to find a way to determine whether a file can be deleted. It is also related to how to track the offset of a replication queue. Anyway, they are internal implementations, so we are free to change them at any time, so this is a not a blocker for our 3.0.0 release. > Make the WAL interface async so sync replication can be built up on the WAL > interface > - > > Key: HBASE-26890 > URL: https://issues.apache.org/jira/browse/HBASE-26890 > Project: HBase > Issue Type: Improvement >Reporter: Duo Zhang >Priority: Major > > Instead of hacking into the WAL implementation. > This could make the implementation more general if later we want to change > the WAL implementation. -- This message was sent by Atlassian Jira (v8.20.1#820001)
