[
https://issues.apache.org/jira/browse/HUDI-1518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17396947#comment-17396947
]
ASF GitHub Bot commented on HUDI-1518:
--------------------------------------
satishkotha commented on a change in pull request #3310:
URL: https://github.com/apache/hudi/pull/3310#discussion_r686397627
##########
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java
##########
@@ -331,29 +323,6 @@ private void
deleteAnyLeftOverMarkerFiles(HoodieEngineContext context, HoodieIns
}
}
- private boolean deleteReplacedFileGroups(HoodieEngineContext context,
HoodieInstant instant) {
- if (!instant.isCompleted() ||
!HoodieTimeline.REPLACE_COMMIT_ACTION.equals(instant.getAction())) {
- // only delete files for completed replace instants
- return true;
- }
-
- TableFileSystemView fileSystemView = this.table.getFileSystemView();
- List<String> replacedPartitions = getReplacedPartitions(instant);
- return ReplaceArchivalHelper.deleteReplacedFileGroups(context, metaClient,
fileSystemView, instant, replacedPartitions);
Review comment:
Do we need ReplaceArchivalHelper class?
##########
File path:
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/io/TestHoodieTimelineArchiveLog.java
##########
@@ -233,45 +231,6 @@ public void testArchiveTableWithArchival() throws
IOException {
verifyInflightInstants(metaClient, 2);
}
- @Test
- public void testArchiveTableWithReplacedFiles() throws Exception {
- HoodieTestUtils.init(hadoopConf, basePath);
- HoodieWriteConfig cfg = HoodieWriteConfig.newBuilder().withPath(basePath)
-
.withSchema(HoodieTestDataGenerator.TRIP_EXAMPLE_SCHEMA).withParallelism(2,
2).forTable("test-trip-table")
-
.withCompactionConfig(HoodieCompactionConfig.newBuilder().retainCommits(1).archiveCommitsWith(2,
3).build())
- .build();
-
- // when using insert_overwrite or insert_overwrite_table
- // first commit may without replaceFileIds
- createReplaceMetadataWithoutReplaceFileId("000");
Review comment:
Could we also remove these helper methods?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
> Remove replaced files logic from archival
> -----------------------------------------
>
> Key: HUDI-1518
> URL: https://issues.apache.org/jira/browse/HUDI-1518
> Project: Apache Hudi
> Issue Type: Task
> Affects Versions: 0.9.0
> Reporter: satish
> Priority: Minor
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> See
> https://github.com/apache/hudi/blob/79ec7b4894b997183a6e10fdc19d34f5ab4ea437/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java#L324
> Archival tries to delete replaced files in 'deleteReplacedFileGroups' method.
> We changed cleaner to delete replaced files. So this is no longer needed. We
> can instead validate that replaced files no longer exist on disk during
> archival.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
