Karen Coppage created HIVE-24302: ------------------------------------ Summary: Cleaner should not mark compaction queue entry as cleaned if it doesn't remove obsolete files Key: HIVE-24302 URL: https://issues.apache.org/jira/browse/HIVE-24302 Project: Hive Issue Type: Bug Reporter: Karen Coppage Assignee: Karen Coppage
Example: # open txn 5, leave it open (maybe it's a long-running compaction) # insert into table t in txns 6, 7 with writeids 1, 2 # compactor.Worker runs on table t and compacts writeids 1, 2 # compactor.Cleaner picks up the compaction queue entry, but doesn't delete any files because the min global open txnid is 5, which cannot see writeIds 1, 2. # Cleaner marks the compactor queue entry as cleaned and removes the entry from the queue. delta_1 and delta_2 will remain in the file system until another compaction is run on table t. Step 5 should not happen, we should skip calling markCleaned() and leave it in the queue in "ready to clean" state. MarkCleaned() should be called only after txn 5 is closed and, following that, the cleaner runs successfully. This will potentially slow down the cleaner, but on the other hand it won't silently "fail" i.e. not do its job. -- This message was sent by Atlassian Jira (v8.3.4#803005)