[
https://issues.apache.org/jira/browse/HBASE-29027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yutong Xiao updated HBASE-29027:
--------------------------------
Description:
Under our current Replication implementation, all WAL files will be consumed
and the data will be filtered by a table attribute
REPLICATION_SCOPE_GLOBAL
We are using 2.4.11 version in our production env. And we have a small group
table to use the replication to do CDC, however with the growth of the number
of other non-replicated tables, we met the log queue accumulation issue and the
data lag grew to hours level.
To solve the issue we write the tables, which has CF marked the
replication_scope to global, to separated WAL files and let the replication
only consume those files. The feature solved our problems and push it to the
community.
was:
Under our current Replication implementation, all WAL files will be consumed
and the data will be filtered by a table attribute
REPLICATION_SCOPE_GLOBAL
We are using 2.4.11 version in our production env. And we have a small group
table to use the replication to do CDC, however with the growth of the number
of other non-replicated tables, we met the log queue accumulation issue and the
data lag grew to hours level.
To solve the issue we write the tables, which has CF marked the
replication_scope to global, to separated WAL files and let the replication
only consume those files. The feature works good in our env and push it to the
community.
> Introduce a new WALProvider to generate WAL files consumed by Replication
> -------------------------------------------------------------------------
>
> Key: HBASE-29027
> URL: https://issues.apache.org/jira/browse/HBASE-29027
> Project: HBase
> Issue Type: Improvement
> Components: Replication, wal
> Reporter: Yutong Xiao
> Assignee: Yutong Xiao
> Priority: Major
>
> Under our current Replication implementation, all WAL files will be consumed
> and the data will be filtered by a table attribute
> REPLICATION_SCOPE_GLOBAL
>
> We are using 2.4.11 version in our production env. And we have a small group
> table to use the replication to do CDC, however with the growth of the number
> of other non-replicated tables, we met the log queue accumulation issue and
> the data lag grew to hours level.
>
> To solve the issue we write the tables, which has CF marked the
> replication_scope to global, to separated WAL files and let the replication
> only consume those files. The feature solved our problems and push it to the
> community.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
