Hi, Hi, Ryan van Huuksloot. >Are you planning on having a thread to check for TTL? Or what is the plan >for TTL? >The quantity based would have a check when a new job is archived?
Just like the implementation in the POC[1], if we continue following the process where HistoryServer#start method periodically invokes HistoryServerArchiveFetcher#fetchArchives based on 'historyserver.archive.fs.refresh-interval' to check whether target files should be retained, what do you think about it ? Of course, I'm very open to hearing about other potentially better implementation approaches. Please let me know what's your opinion. Thank you. [1] https://github.com/apache/flink/pull/26902 Best, Yuepeng Pan On 2025/08/14 16:07:10 Ryan van Huuksloot wrote: > Thanks, sounds good. > > Are you planning on having a thread to check for TTL? Or what is the plan > for TTL? > The quantity based would have a check when a new job is archived? > > Ryan van Huuksloot > Staff Engineer, Infrastructure | Streaming Platform > [image: Shopify] > <https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> > > > On Thu, Aug 14, 2025 at 12:04 PM Yuepeng Pan <panyuep...@apache.org> wrote: > > > Hi, Ryan van Huuksloot. > > > > Thank you very much for your reply. > Question: Is the History Server then > > going to delete the files stored? > (i.e. we use GCS, would it delete the > > files there as well?) > Or is this strictly what is shown in the UI? > > > > > > > > > > Yes, this feature introduced in the FLIP is a super-set of the original > > feature that is controlled by 'historyserver.archive.retained-jobs'. > > > > So if I understand correctly, after the new feature is introduced, it > > would affect the retention period of remote distributed storage jobs > > history files as well, not only for what is shown in the UI. > > > > > > > > > > Best, > > Yuepeng Pan > > > > > > > > > > > > > > > > > > At 2025-08-14 23:34:54, "Ryan van Huuksloot" > > <ryan.vanhuuksl...@shopify.com.INVALID> wrote: > > >I took a look. Overall it would be nice to have more ways to configure the > > >History Server. > > > > > >Question: Is the History Server then going to delete the files stored? > > >(i.e. we use GCS, would it delete the files there as well?) > > >Or is this strictly what is shown in the UI? > > > > > >Ryan van Huuksloot > > >Staff Engineer, Infrastructure | Streaming Platform > > >[image: Shopify] > > ><https://www.shopify.com/?utm_medium=salessignatures&utm_source=hs_email> > > > > > > > > >On Thu, Aug 14, 2025 at 11:17 AM Yuepeng Pan <panyuep...@apache.org> > > wrote: > > > > > >> Bumping this thread. Thanks! > > >> > > >> Best, > > >> Yuepeng Pan > > >> > > >> On 2025/08/11 03:49:27 Yuepeng Pan wrote: > > >> > Hi community, > > >> > > > >> > > > >> > Currently, HistoryServer supports only a quantity-based job archive > > >> retention policy [1]. > > >> > This is insufficient for scenarios such as: > > >> > - Time-based retention (e.g., last X days). > > >> > - Combined rules (e.g., within 7 days AND ≤100 jobs). > > >> > > > >> > > > >> > To address these limitations, I’d like to start a discussion on > > FLIP-490 > > >> [2], > > >> > which proposes a more flexible job archive retention mechanism that > > >> supports time-based, quantity-based, and composite strategies (with > > AND/OR > > >> logic). > > >> > > > >> > > > >> > Looking forward to your feedback. > > >> > > > >> > > > >> > Best, > > >> > Yuepeng Pan > > >> > > > >> > > > >> > [1] > > >> > > https://github.com/apache/flink/blob/cae5fb4d3b6d9e0c10c3539ea4994fc1ad463b70/flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServer.java#L241 > > >> > [2] > > >> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=332499857 > > >> > > >
