klcopp commented on a change in pull request #2971:
URL: https://github.com/apache/hive/pull/2971#discussion_r793559890
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
##########
@@ -448,15 +461,15 @@ private boolean isFileBelowWatermark(FileStatus child,
long highWatermark) {
Path p = child.getPath();
String fn = p.getName();
if (!child.isDirectory()) {
- return false;
+ return true;
}
if (fn.startsWith(AcidUtils.BASE_PREFIX)) {
ParsedBaseLight b = ParsedBaseLight.parseBase(p);
- return b.getWriteId() < highWatermark;
+ return b.getWriteId() <= highWatermark;
Review comment:
The AcidDirectory could be based off of some older txn id (depending on
how the Cleaner selects which compaction to pick up – if CQ_NEXT_TXN_ID is
filled in the cleaner will only attempt to clean when it can, however if
CQ_NEXT_TXN_ID==NULL then the cleaner will try to clean even if it's blocked by
an obsolete dir). It could be that txnId=99 was opened, table a was created,
table a is compacted many times but the cleaner will never clean anything from
table a because txid=99 sees table a as totally empty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
