[ https://issues.apache.org/jira/browse/HIVE-19739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sankar Hariappan updated HIVE-19739: ------------------------------------ Description: Currently. bootstrap REPL LOAD have added checkpoint identifiers in DB/table/partition object properties once the data/metadata related to the object is successfully loaded. If the Db exist and is not empty, then currently we are throwing exception. But need to support it for the retry scenario after a failure. If there is a retry of bootstrap load using the same dump, then instead of throwing error, we should check if any of the tables/partitions are completely loaded using the checkpoint identifiers. If yes, then skip it or else drop/create them again. If the bootstrap load is performed using different dump, then it should throw exception. was: Currently. bootstrap REPL LOAD expect the target database to be empty or not exist to start bootstrap load. But, this adds overhead when there is a failure in between bootstrap load and there is no way to resume it from where it fails. So, it is needed to create checkpoints in table/partitions to skip the completely loaded objects. Use the fully qualified path of the dump directory as a checkpoint identifier. This should be added to the table / partition properties in hive via a task, as the last task in the DAG for table / partition creation. > Bootstrap REPL LOAD to use checkpoints to validate and skip the loaded > data/metadata. > ------------------------------------------------------------------------------------- > > Key: HIVE-19739 > URL: https://issues.apache.org/jira/browse/HIVE-19739 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl > Affects Versions: 3.0.0 > Reporter: Sankar Hariappan > Assignee: Sankar Hariappan > Priority: Major > Labels: DR, pull-request-available, replication > Fix For: 4.0.0 > > > Currently. bootstrap REPL LOAD have added checkpoint identifiers in > DB/table/partition object properties once the data/metadata related to the > object is successfully loaded. > If the Db exist and is not empty, then currently we are throwing exception. > But need to support it for the retry scenario after a failure. > If there is a retry of bootstrap load using the same dump, then instead of > throwing error, we should check if any of the tables/partitions are > completely loaded using the checkpoint identifiers. If yes, then skip it or > else drop/create them again. > If the bootstrap load is performed using different dump, then it should throw > exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005)