vinothchandar commented on a change in pull request #1009: [HUDI-308] Avoid
Renames for tracking state transitions of all actions on dataset
URL: https://github.com/apache/incubator-hudi/pull/1009#discussion_r347940639
##########
File path:
hudi-client/src/main/java/org/apache/hudi/io/HoodieCommitArchiveLog.java
##########
@@ -108,16 +110,25 @@ private void close() {
*/
public boolean archiveIfRequired(final JavaSparkContext jsc) throws
IOException {
try {
- List<HoodieInstant> instantsToArchive =
getInstantsToArchive(jsc).collect(Collectors.toList());
+ List<HoodieInstant> instantsToDelete =
getInstantsToArchive(jsc).collect(Collectors.toList());
+ List<HoodieInstant> instantsToArchive = instantsToDelete.stream()
+ .filter(HoodieInstant::isCompleted).collect(Collectors.toList());
+
+ Preconditions.checkArgument(instantsToArchive.isEmpty() ==
instantsToDelete.isEmpty(),
Review comment:
not sure I follow this change? are you doing this to keep existing behavior
of only archiving completed instants? if so, is that necessary... Nothing
reads the archived timeline now except cli?
does this help us in someway when we redesign the timeline?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
