[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459836#comment-15459836 ] Vladimir Rodionov commented on HBASE-15449: --- lgtm. > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu > Labels: backup > Fix For: 2.0.0 > > Attachments: 15449.v1.txt, 15449.v10.txt, 15449.v11.txt, > 15449.v2.txt, 15449.v4.txt, 15449.v5.txt, 15449.v7.txt, 15449.v8.txt > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457068#comment-15457068 ] Ted Yu commented on HBASE-15449: Patch is on review board: https://reviews.apache.org/r/51586/diff/1#index_header > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu > Labels: backup > Fix For: 2.0.0 > > Attachments: 15449.v1.txt, 15449.v2.txt, 15449.v4.txt, 15449.v5.txt, > 15449.v7.txt, 15449.v8.txt > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457064#comment-15457064 ] Ted Yu commented on HBASE-15449: https://issues.apache.org/jira/secure/attachment/12826716/15449.v8.txt should apply. I checked RestoreServerUtil.java in the above patch just now where the conflict is resolved. > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu > Labels: backup > Fix For: 2.0.0 > > Attachments: 15449.v1.txt, 15449.v2.txt, 15449.v4.txt, 15449.v5.txt, > 15449.v7.txt, 15449.v8.txt > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457049#comment-15457049 ] Vladimir Rodionov commented on HBASE-15449: --- I do not see patch, [~tedyu] Have you attached it? > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu > Labels: backup > Fix For: 2.0.0 > > Attachments: 15449.v1.txt, 15449.v2.txt, 15449.v4.txt, 15449.v5.txt, > 15449.v7.txt, 15449.v8.txt > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[
https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15457041#comment-15457041
]
Vladimir Rodionov commented on HBASE-15449:
---
[~tedyu], still does not apply
{code}
HW10692:hbase vrodionov$ patch -p1 < 15449.v8.txt
patching file
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/impl/RestoreClientImpl.java
patching file
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/util/BackupServerUtil.java
patching file
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/util/RestoreServerUtil.java
Hunk #3 FAILED at 59.
Hunk #4 succeeded at 140 (offset 3 lines).
Hunk #5 succeeded at 200 (offset 3 lines).
Hunk #6 succeeded at 236 (offset 3 lines).
Hunk #7 succeeded at 344 (offset 3 lines).
Hunk #8 succeeded at 567 (offset 3 lines).
Hunk #9 succeeded at 667 (offset 3 lines).
Hunk #10 succeeded at 685 (offset 3 lines).
Hunk #11 succeeded at 706 with fuzz 1 (offset 11 lines).
1 out of 11 hunks FAILED -- saving rejects to file
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/util/RestoreServerUtil.java.rej
patching file
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
patching file
hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestBackupBase.java
patching file
hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestIncrementalBackup.java
{code}
> HBase Backup Phase 3: Support physical table layout change
> ---
>
> Key: HBASE-15449
> URL: https://issues.apache.org/jira/browse/HBASE-15449
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0
>Reporter: Vladimir Rodionov
>Assignee: Ted Yu
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: 15449.v1.txt, 15449.v2.txt, 15449.v4.txt, 15449.v5.txt,
> 15449.v7.txt, 15449.v8.txt
>
>
> Table operation such as add column family, delete column family, truncate ,
> delete table may result in subsequent backup restore failure.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456805#comment-15456805 ] Ted Yu commented on HBASE-15449: https://reviews.apache.org/r/51586/ > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu > Labels: backup > Fix For: 2.0.0 > > Attachments: 15449.v1.txt, 15449.v2.txt, 15449.v4.txt, 15449.v5.txt, > 15449.v7.txt, 15449.v8.txt > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456785#comment-15456785 ] Vladimir Rodionov commented on HBASE-15449: --- [~tedyu], when you update your patch can you put it on review board? > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu > Labels: backup > Fix For: 2.0.0 > > Attachments: 15449.v1.txt, 15449.v2.txt, 15449.v4.txt, 15449.v5.txt, > 15449.v7.txt, 15449.v8.txt > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456766#comment-15456766 ] Vladimir Rodionov commented on HBASE-15449: --- [~tedyu], your last patch does not apply cleanly, can you fix it? > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu > Labels: backup > Fix For: 2.0.0 > > Attachments: 15449.v1.txt, 15449.v2.txt, 15449.v4.txt, 15449.v5.txt, > 15449.v7.txt, 15449.v8.txt > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15453951#comment-15453951 ] Vladimir Rodionov commented on HBASE-15449: --- [~tedyu], can you add test with removing CF between full and incremental? full, add data, incremental 1, remove CF, add data, incremental 2, add some data -> restore to Incremental 2? That test should validate your code. > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu > Labels: backup > Fix For: 2.0.0 > > Attachments: 15449.v1.txt, 15449.v2.txt, 15449.v4.txt, 15449.v5.txt, > 15449.v7.txt > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440265#comment-15440265 ] Ted Yu commented on HBASE-15449: bq. This only works if the user specifies auto restore Auto restore is default now. bq. and specifies the last PIT image to restore to? In the modified TestIncrementalBackup (in next patch), you would see that both full restore and incremental restore work. bq. The LoadIncrementalHFiles change can be done in a separate JIRA for general use Let me log a separate JIRA and continue discussion there. Thanks, Jerry. > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: 15449.v1.txt, 15449.v2.txt, 15449.v4.txt, 15449.v5.txt > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440255#comment-15440255 ] Jerry He commented on HBASE-15449: -- Ok. Is this a optimization/short-cut of the normal sequence of restores? This only works if the user specifies auto restore and specifies the last PIT image to restore to? Does it work if user does one restore at a time manual restore? Does it work for incremental restore WAL replay? The LoadIncrementalHFiles change can be done in a separate JIRA for general use? > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: 15449.v1.txt, 15449.v2.txt, 15449.v4.txt, 15449.v5.txt > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15440108#comment-15440108 ] Vladimir Rodionov commented on HBASE-15449: --- Yes, this is what we need. Go ahead, Ted. > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: 15449.v1.txt, 15449.v2.txt, 15449.v4.txt, 15449.v5.txt > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15439879#comment-15439879 ] Ted Yu commented on HBASE-15449: I am thinking of passing the backup Id of the most recent incremental backup to full restore so that the most recent table descriptor can be retrieved and used to create the new table. Comment is welcome on the above approach. > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: 15449.v1.txt, 15449.v2.txt, 15449.v4.txt, 15449.v5.txt > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15409317#comment-15409317 ] Ted Yu commented on HBASE-15449: In the scenario I added to TestIncrementalBackupNoDataLoss, full restore is performed first. The tableInfo in the backup image is outdated - it lacks the column family added after full backup. That was why I resort to admin.getTableDescriptor(). > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: 15449.v1.txt, 15449.v2.txt > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408769#comment-15408769 ] Jerry He commented on HBASE-15449: -- Is this for incremental restore? For full restore, the tableInfo is in the backup image, which can be used to restore the schema. For incremental restore, the incremental backup image also keeps the PIT tableInfo. Could we do alter table with the new PIT tableInfo in the image? > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: 15449.v1.txt, 15449.v2.txt > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408730#comment-15408730 ] Ted Yu commented on HBASE-15449: The goal is to handle mismatch between current table schema and the schema recorded in backup. Current table descriptor is obtained. If the descriptor has more column families, we use it for restore. If the current descriptor has fewer column families, LoadIncrementalHFiles would ignore absent column families. Planning to log another JIRA for enhancing LoadIncrementalHFiles. > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: 15449.v1.txt, 15449.v2.txt > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408720#comment-15408720 ] Jerry He commented on HBASE-15449: -- I could not follow the above discussion completely. Could you provide outline what we try to do here? Incremental backup has a copy of the tableInfo as of the time of the incremental backup. No? LoadIncrementalHFiles such that unmatched families are skipped during bulk load? If the backup has more cfs, then yes, this is needed. > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: 15449.v1.txt, 15449.v2.txt > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408470#comment-15408470 ] Ted Yu commented on HBASE-15449: Currently in RestoreServerUtil#restoreTableAndCreate(), we obtain HTableDescriptor of tableName from snapshot. If column family is added to table schema after full backup but prior to restoring the backup, we should obtain HTableDescriptor via HBaseAdmin. This way LoadIncrementalHFiles would continue to function. > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408311#comment-15408311 ] Vladimir Rodionov commented on HBASE-15449: --- If you delete CF, then restore will fail for sure. > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408208#comment-15408208 ] Ted Yu commented on HBASE-15449: I am modifying existing backup test by altering table in between full / incremental backup. Will see what error we get. > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15449) HBase Backup Phase 3: Support physical table layout change
[ https://issues.apache.org/jira/browse/HBASE-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15200136#comment-15200136 ] Vladimir Rodionov commented on HBASE-15449: --- Moved to Phase 3. > HBase Backup Phase 3: Support physical table layout change > --- > > Key: HBASE-15449 > URL: https://issues.apache.org/jira/browse/HBASE-15449 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > Table operation such as add column family, delete column family, truncate , > delete table may result in subsequent backup restore failure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
