[
https://issues.apache.org/jira/browse/HBASE-25891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mallikarjun updated HBASE-25891:
--------------------------------
Description:
Currently WAL logs are stored in `backup:system` meta table
{code:java}
// code placeholder
wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:backupId,
timestamp=1622003479895, value=backup_1622003358258
wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:file,
timestamp=1622003479895,
value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621996160175
wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:root,
timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:backupId,
timestamp=1622003479895, value=backup_1622003358258
wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:file,
timestamp=1622003479895,
value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621999760280
wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:root,
timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
{code}
This has several problems
# Ever growing rows of wal's sourced for incremental backup is maintained and
never cleaned up.
# Unnecessary to have wal log listed for performing incremental backup or for
performing logcleaner.
# No support for rsgroup. Hence tables belonging to rsgroups which doesn't
have backup enabled also have to retain wals' and forever.
Proposed Solution:
was:
Currently WAL logs are stored in `backup:system` meta table
{code:java}
// code placeholder
wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:backupId,
timestamp=1622003479895, value=backup_1622003358258
wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:file,
timestamp=1622003479895,
value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621996160175
wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:root,
timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:backupId,
timestamp=1622003479895, value=backup_1622003358258
wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:file,
timestamp=1622003479895,
value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621999760280
wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:root,
timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
{code}
This has several problems
# Ever growing rows of wal's sourced for incremental backup is maintained and
never cleaned up.
# Unnecessary to have wal log listed for performing incremental backup or for
performing logcleaner.
# No support for rsgroup. Hence all rsgroups which doesn't have backup enabled
tables, WAL's are retained forever.
> Remove dependence storing WAL filenames for backup
> --------------------------------------------------
>
> Key: HBASE-25891
> URL: https://issues.apache.org/jira/browse/HBASE-25891
> Project: HBase
> Issue Type: Improvement
> Components: backup&restore
> Affects Versions: 3.0.0-alpha-1
> Reporter: Mallikarjun
> Assignee: Mallikarjun
> Priority: Major
> Fix For: 3.0.0-alpha-1
>
>
> Currently WAL logs are stored in `backup:system` meta table
> {code:java}
> // code placeholder
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:backupId,
> timestamp=1622003479895, value=backup_1622003358258
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:file,
> timestamp=1622003479895,
> value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621996160175
> wals:preprod-dn-1%2C16020%2C1614844389000.1621996160175 column=meta:root,
> timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:backupId,
> timestamp=1622003479895, value=backup_1622003358258
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:file,
> timestamp=1622003479895,
> value=hdfs://store/hbase/oldWALs/preprod-dn-1%2C16020%2C1614844389000.1621999760280
> wals:preprod-dn-1%2C16020%2C1614844389000.1621999760280 column=meta:root,
> timestamp=1622003479895, value=s3a://2021-05-25--21-45-00--full/set1
> {code}
>
> This has several problems
> # Ever growing rows of wal's sourced for incremental backup is maintained
> and never cleaned up.
> # Unnecessary to have wal log listed for performing incremental backup or
> for performing logcleaner.
> # No support for rsgroup. Hence tables belonging to rsgroups which doesn't
> have backup enabled also have to retain wals' and forever.
>
> Proposed Solution:
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
