TheR1sing3un opened a new pull request, #13606: URL: https://github.com/apache/hudi/pull/13606
In most of our scenarios, we use the clean incremental scanning form to ensure the stability of clean because our table has a large number of partitions and the update frequency of the partitions is relatively low. If we do not use incremental scanning to clean, then each time we need to load the files of all partitions of the entire table. Now I find that archival does not consider the instant that is said to be retained in the clean plan. Therefore, in some scenarios, archival will also archive this instant, causing the ability of clean incremental partition scanning fall back to full table scanning. It is hoped to introduce a configuration to enable archival to consider this situation, in order to ensure that incremental scans always take effect. ### Change Logs 1. Archival considers retaining the `the earliest retain instant` in the clean plan ### Impact Enhance the stability of cleaning service ### Risk level (write none, low medium or high below) low ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Change Logs and Impact were stated clearly - [x] Adequate tests were added if applicable - [x] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
