ashutosh-bapat commented on a change in pull request #587: HIVE-21213 : Acid
table bootstrap replication needs to handle directory created by compaction
with txn id
URL: https://github.com/apache/hive/pull/587#discussion_r360315217
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##########
@@ -463,7 +464,29 @@ public static Path
getCopyDestination(ReplChangeManager.FileInfo fileInfo, Path
String[] subDirs = fileInfo.getSubDir().split(Path.SEPARATOR);
Path destination = destRoot;
for (String subDir: subDirs) {
- destination = new Path(destination, subDir);
+ // If the directory is created by compactor, then the directory will
have the transaction id also.
+ // In case of replication, the same txn id can not be used at target, as
the txn with same id might be a
+ // aborted or live txn at target.
+ // In case of bootstrap load, we copy only the committed data, so the
directory with only write id
+ // can be created. The validity txn id can be removed from the directory
name.
+ // TODO : Support for incremental load flow. This can be done once
replication of compaction is decided.
Review comment:
The real question is whether there's a bug here, if compaction happens on
the source as well as on the target. It would be better to clarify that this
login needs to be worked again when incremental replication will be worked
again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
