nsivabalan commented on a change in pull request #4016:
URL: https://github.com/apache/hudi/pull/4016#discussion_r768957857
##########
File path:
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieActiveTimeline.java
##########
@@ -203,6 +203,11 @@ public void deletePendingIfExists(HoodieInstant.State
state, String action, Stri
deleteInstantFileIfExists(instant);
}
+ public void deleteEmptyIfExists(HoodieInstant instant) {
Review comment:
minor: deleteEmptyInstantIfExists
##########
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java
##########
@@ -317,10 +317,19 @@ public void archive(HoodieEngineContext context,
List<HoodieInstant> instants) t
List<IndexedRecord> records = new ArrayList<>();
for (HoodieInstant hoodieInstant : instants) {
try {
- deleteAnyLeftOverMarkers(context, hoodieInstant);
- records.add(convertToAvroRecord(hoodieInstant));
- if (records.size() >= this.config.getCommitArchivalBatchSize()) {
- writeToFile(wrapperSchema, records);
+ if (table.getActiveTimeline().isEmpty(hoodieInstant)
Review comment:
@vinothchandar @bvaradar : Can you review these changes.
##########
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanner.java
##########
@@ -146,11 +146,15 @@ public CleanPlanner(HoodieEngineContext context,
HoodieTable<T, I, K, O> hoodieT
if (config.incrementalCleanerModeEnabled()) {
Option<HoodieInstant> lastClean =
hoodieTable.getCleanTimeline().filterCompletedInstants().lastInstant();
if (lastClean.isPresent()) {
- HoodieCleanMetadata cleanMetadata = TimelineMetadataUtils
-
.deserializeHoodieCleanMetadata(hoodieTable.getActiveTimeline().getInstantDetails(lastClean.get()).get());
- if ((cleanMetadata.getEarliestCommitToRetain() != null)
- && (cleanMetadata.getEarliestCommitToRetain().length() > 0)) {
- return getPartitionPathsForIncrementalCleaning(cleanMetadata,
instantToRetain);
+ if (hoodieTable.getActiveTimeline().isEmpty(lastClean.get())) {
+ hoodieTable.getActiveTimeline().deleteEmptyIfExists(lastClean.get());
Review comment:
I am wondering how come we could have a empty file for a completed
clean. I thought we might have empty file issues only with inflght ones. can
you help me understand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
