[
https://issues.apache.org/jira/browse/HBASE-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17905216#comment-17905216
]
Yutong Xiao commented on HBASE-29027:
-------------------------------------
No. The latency to the downstream was good that time.
The problem is the workload.
In our production env we cannot set too many multi wals for a reasonable MTTR.
In our case, we set the multiwal with group number 3.
As hbase architecture is that 1 queue has one thread to consume, we have 3
replication log queues and there are 3 threads to consume the wals.
So that the log queue is easy to be accumulated when the server is under heavy
write workload, where may be tens/hundreds of writing threads working and this
contributes to a higher frequency of log rolling.
> Introduce a new WALProvider to generate WAL files consumed by Replication
> -------------------------------------------------------------------------
>
> Key: HBASE-29027
> URL: https://issues.apache.org/jira/browse/HBASE-29027
> Project: HBase
> Issue Type: Improvement
> Components: Replication, wal
> Reporter: Yutong Xiao
> Assignee: Yutong Xiao
> Priority: Major
> Labels: pull-request-available
> Attachments: Screenshot 2024-12-11 at 17.46.40.png
>
>
> Under our current Replication implementation, all WAL files will be consumed
> and the data will be filtered by a table attribute
> REPLICATION_SCOPE_GLOBAL
>
> We are using 2.4.11 version in our production env. And we have a small group
> table to use the replication to do CDC, however with the growth of the number
> of other non-replicated tables, we met the log queue accumulation issue and
> the data lag grew to hours level.
>
> To solve the issue we write the tables, which has CF marked the
> replication_scope to global, to separated WAL files and let the replication
> only consume those files. The feature solved our problems and push it to the
> community.
> In our env, we have 19 tables and only 8 of them need to be replicated.
> There are write heavy tables in the the rest of 11. So that our business
> benefits a lot from this feature.
>
> The attached screenshot shows the improvement.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
