sankarh commented on a change in pull request #2651:
URL: https://github.com/apache/hive/pull/2651#discussion_r712136132
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
##########
@@ -179,6 +180,13 @@ private void clean(CompactionInfo ci, long minOpenTxnGLB,
boolean metricsEnabled
txnHandler.markCleaned(ci);
return;
}
+ if (MetaStoreUtils.isNoCleanUpSet(t.getParameters())) {
+ // The table was marked no clean up true.
+ LOG.info("Skipping " + ci.getFullTableName() + " clean up, as
NO_CLEANUP set to true");
+ txnHandler.markCleaned(ci);
Review comment:
We shouldn't mark these table/partition as cleaned since we skip it
temporarily. If user enable it back, then next cycle should clean the obsolete
files.
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
##########
@@ -189,6 +197,12 @@ private void clean(CompactionInfo ci, long minOpenTxnGLB,
boolean metricsEnabled
txnHandler.markCleaned(ci);
return;
}
+ if(MetaStoreUtils.isNoCleanUpSet(p.getParameters())){
Review comment:
nit: Add space before ( and {
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
##########
@@ -189,6 +197,12 @@ private void clean(CompactionInfo ci, long minOpenTxnGLB,
boolean metricsEnabled
txnHandler.markCleaned(ci);
return;
}
+ if(MetaStoreUtils.isNoCleanUpSet(p.getParameters())){
+ // The table was marked no clean up true.
Review comment:
Update the comment: "The partition is marked with no_cleanup=true"
##########
File path: ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestCleaner.java
##########
@@ -604,4 +608,113 @@ public void tearDown() throws Exception {
compactorTestCleanup();
}
+ @Test
+ public void NoCleanupAfterMajorCompaction() throws Exception {
+ Map<String, String> parameters = new HashMap<>();
+
+ //With no cleanup true
+ parameters.put("no_cleanup", "true");
+ Table t = newTable("default", "dcamc", false, parameters);
+
+ addBaseFile(t, null, 20L, 20);
+ addDeltaFile(t, null, 21L, 22L, 2);
+ addDeltaFile(t, null, 23L, 24L, 2);
+ addBaseFile(t, null, 25L, 25);
+
+ burnThroughTransactions("default", "dcamc", 25);
+
+ CompactionRequest rqst = new CompactionRequest("default", "dcamc",
CompactionType.MAJOR);
+ compactInTxn(rqst);
+
+ startCleaner();
+ // Check there are no compactions requests left.
+ ShowCompactResponse rsp = txnHandler.showCompact(new ShowCompactRequest());
+ Assert.assertEquals(1, rsp.getCompactsSize());
+ Assert.assertEquals(TxnStore.SUCCEEDED_RESPONSE,
rsp.getCompacts().get(0).getState());
+
+ // Check that the files are not removed
+ List<Path> paths = getDirectories(conf, t, null);
+ Assert.assertEquals(4, paths.size());
+
+ //With no clean up false
+ t = ms.getTable(new GetTableRequest("default", "dcamc"));
+ t.getParameters().put("no_cleanup", "false");
+ ms.alter_table("default", "dcamc", t);
+ rqst = new CompactionRequest("default", "dcamc", CompactionType.MAJOR);
+ compactInTxn(rqst);
+
+ startCleaner();
+ // Check there are no compactions requests left.
+ rsp = txnHandler.showCompact(new ShowCompactRequest());
+ Assert.assertEquals(2, rsp.getCompactsSize());
+ Assert.assertEquals(TxnStore.SUCCEEDED_RESPONSE,
rsp.getCompacts().get(0).getState());
+
+ // Check that the files are not removed
+ paths = getDirectories(conf, t, null);
+ Assert.assertEquals(1, paths.size());
+ Assert.assertEquals("base_25", paths.get(0).getName());
+ }
Review comment:
Enable cleanup again and run cleaner which should remove the files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
