zhengyuan-cn commented on issue #6596: URL: https://github.com/apache/hudi/issues/6596#issuecomment-1262020313
> hi xushiyan ,I debugged in flink + hudi local mode, I found CleanPlanner deleted my partion, I have three pations,(2022/09/27,2022/09/28,2022/09/29). CleanPlanner deleted partion '2022/09/27'. detial logs as below. --------------------------------------------------------------------------- detail log : `306975 [pool-16-thread-1] INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView - Took 62 ms to read 16 instants, 171 replaced file groups 306998 [pool-16-thread-1] INFO org.apache.hudi.common.util.ClusteringUtils - Found 109 files in pending clustering operations 306998 [pool-16-thread-1] INFO org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView - Sending request : (http://192.168.1.75:58989/v1/hoodie/view/compactions/pending/?basepath=hdfs%3A%2F%2Fhadoop01%3A9000%2Fhudi%2Fcow-intact-4&lastinstantts=20220929165857714&timelinehash=3446cb10ee80b94e6b37ad4052890146807bbf579bd20bed86c2e7564d09b62d) 307014 [qtp805746605-86] INFO org.apache.hudi.timeline.service.RequestHandler - Syncing view as client passed last known instant 20220929165857714 as last known instant but server has the following last instant on timeline :Option{val=[20220929165857714__commit__COMPLETED]} 307018 [qtp805746605-86] INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants upto : Option{val=[==>20220929165927744__commit__INFLIGHT]} 307049 [qtp805746605-86] INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView - Took 31 ms to read 16 instants, 171 replaced file groups 307072 [qtp805746605-86] INFO org.apache.hudi.common.util.ClusteringUtils - Found 109 files in pending clustering operations 307078 [pool-16-thread-1] INFO org.apache.hudi.table.action.clean.CleanPlanner - Incremental Cleaning mode is enabled. Looking up partition-paths that have since changed since last cleaned at 20220929164457499. New Instant to retain : Option{val=[20220929164559700__replacecommit__COMPLETED]} 307086 [pool-16-thread-1] INFO org.apache.hudi.table.action.clean.CleanPlanner - Total Partitions to clean : 3, with policy KEEP_LATEST_COMMITS 307086 [pool-16-thread-1] INFO org.apache.hudi.table.action.clean.CleanPlanner - Using cleanerParallelism: 3 307086 [pool-16-thread-1] INFO org.apache.hudi.table.action.clean.CleanPlanner - Cleaning 2022/09/27, retaining latest 30 commits. 307086 [ForkJoinPool.commonPool-worker-6] INFO org.apache.hudi.table.action.clean.CleanPlanner - Cleaning 2022/09/28, retaining latest 30 commits. 307086 [ForkJoinPool.commonPool-worker-11] INFO org.apache.hudi.table.action.clean.CleanPlanner - Cleaning 2022/09/29, retaining latest 30 commits. 307087 [ForkJoinPool.commonPool-worker-6] INFO org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView - Sending request : (http://192.168.1.75:58989/v1/hoodie/view/filegroups/replaced/before/?partition=2022%2F09%2F28&maxinstant=20220929164559700&basepath=hdfs%3A%2F%2Fhadoop01%3A9000%2Fhudi%2Fcow-intact-4&lastinstantts=20220929165857714&timelinehash=3446cb10ee80b94e6b37ad4052890146807bbf579bd20bed86c2e7564d09b62d) 307087 [ForkJoinPool.commonPool-worker-11] INFO org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView - Sending request : (http://192.168.1.75:58989/v1/hoodie/view/filegroups/replaced/before/?partition=2022%2F09%2F29&maxinstant=20220929164559700&basepath=hdfs%3A%2F%2Fhadoop01%3A9000%2Fhudi%2Fcow-intact-4&lastinstantts=20220929165857714&timelinehash=3446cb10ee80b94e6b37ad4052890146807bbf579bd20bed86c2e7564d09b62d) 307087 [pool-16-thread-1] INFO org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView - Sending request : (http://192.168.1.75:58989/v1/hoodie/view/filegroups/replaced/before/?partition=2022%2F09%2F27&maxinstant=20220929164559700&basepath=hdfs%3A%2F%2Fhadoop01%3A9000%2Fhudi%2Fcow-intact-4&lastinstantts=20220929165857714&timelinehash=3446cb10ee80b94e6b37ad4052890146807bbf579bd20bed86c2e7564d09b62d) 307089 [qtp805746605-535] INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView - Building file system view for partition (2022/09/27) 307090 [qtp805746605-535] INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView - addFilesToView: NumFiles=3, NumFileGroups=2, FileGroupsCreationTime=0, StoreTimeTaken=0 307093 [qtp805746605-81] INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView - Building file system view for partition (2022/09/29) 307098 [qtp805746605-81] INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView - addFilesToView: NumFiles=214, NumFileGroups=213, FileGroupsCreationTime=2, StoreTimeTaken=0 307110 [qtp805746605-78] INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView - Building file system view for partition (2022/09/28) 307111 [pool-16-thread-1] INFO org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView - Sending request : (http://192.168.1.75:58989/v1/hoodie/view/filegroups/all/partition/?partition=2022%2F09%2F27&basepath=hdfs%3A%2F%2Fhadoop01%3A9000%2Fhudi%2Fcow-intact-4&lastinstantts=20220929165857714&timelinehash=3446cb10ee80b94e6b37ad4052890146807bbf579bd20bed86c2e7564d09b62d) 307112 [qtp805746605-78] INFO org.apache.hudi.common.table.view.AbstractTableFileSystemView - addFilesToView: NumFiles=4, NumFileGroups=3, FileGroupsCreationTime=0, StoreTimeTaken=0 307112 [ForkJoinPool.commonPool-worker-11] INFO org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView - Sending request : (http://192.168.1.75:58989/v1/hoodie/view/filegroups/all/partition/?partition=2022%2F09%2F29&basepath=hdfs%3A%2F%2Fhadoop01%3A9000%2Fhudi%2Fcow-intact-4&lastinstantts=20220929165857714&timelinehash=3446cb10ee80b94e6b37ad4052890146807bbf579bd20bed86c2e7564d09b62d) 307113 [pool-16-thread-1] INFO org.apache.hudi.table.action.clean.CleanPlanner - 1 patterns used to delete in partition path:2022/09/27 307113 [pool-16-thread-1] INFO org.apache.hudi.table.action.clean.CleanPlanner - Partition 2022/09/27 to be deleted 307113 [ForkJoinPool.commonPool-worker-6] INFO org.apache.hudi.common.table.view.RemoteHoodieTableFileSystemView - Sending request : (http://192.168.1.75:58989/v1/hoodie/view/filegroups/all/partition/?partition=2022%2F09%2F28&basepath=hdfs%3A%2F%2Fhadoop01%3A9000%2Fhudi%2Fcow-intact-4&lastinstantts=20220929165857714&timelinehash=3446cb10ee80b94e6b37ad4052890146807bbf579bd20bed86c2e7564d09b62d) 307114 [ForkJoinPool.commonPool-worker-6] INFO org.apache.hudi.table.action.clean.CleanPlanner - 1 patterns used to delete in partition path:2022/09/28 307128 [ForkJoinPool.commonPool-worker-11] INFO org.apache.hudi.table.action.clean.CleanPlanner - 7 patterns used to delete in partition path:2022/09/29 307153 [pool-16-thread-1] INFO org.apache.hudi.table.action.clean.CleanPlanner - Requesting Cleaning with instant time [==>20220929165957292__clean__REQUESTED] 307157 [pool-16-thread-1] INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants upto : Option{val=[==>20220929165957292__clean__REQUESTED]} 307161 [pool-16-thread-1] INFO org.apache.hudi.table.action.clean.CleanActionExecutor - Finishing previously unfinished cleaner instant=[==>20220929165957292__clean__REQUESTED] 307163 [pool-16-thread-1] INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Checking for file exists ?hdfs://hadoop01:9000/hudi/cow-intact-4/.hoodie/20220929165957292.clean.requested 307184 [pool-16-thread-1] INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Create new file for toInstant ?hdfs://hadoop01:9000/hudi/cow-intact-4/.hoodie/20220929165957292.clean.inflight 307184 [pool-16-thread-1] INFO org.apache.hudi.table.action.clean.CleanActionExecutor - Using cleanerParallelism: 3 307214 [pool-16-thread-1] INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants upto : Option{val=[==>20220929165957292__clean__INFLIGHT]} 307215 [pool-16-thread-1] INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Checking for file exists ?hdfs://hadoop01:9000/hudi/cow-intact-4/.hoodie/20220929165957292.clean.inflight 307235 [pool-16-thread-1] INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Create new file for toInstant ?hdfs://hadoop01:9000/hudi/cow-intact-4/.hoodie/20220929165957292.clean 307235 [pool-16-thread-1] INFO org.apache.hudi.table.action.clean.CleanActionExecutor - Marked clean started on 20220929165957292 as complete 307239 [pool-16-thread-1] INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline - Loaded instants upto : Option{val=[20220929165957292__clean__COMPLETED]}` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
