DanielLeens commented on issue #10767:
URL: https://github.com/apache/seatunnel/issues/10767#issuecomment-6739656927

   Thanks for the measurements. Keeping compact summaries longer than detailed 
finished-job metrics is a reasonable first design direction, but it is not yet 
a change to defaults or retention configuration.
   
   Current `dev` still has three separate boundaries: finished-history maps use 
`history-job-expire-minutes`; running metrics use `engine_runningJobMetrics`; 
and `FileMapStore` explicitly skips persistence for the running-metrics map. 
Therefore the reported WAL growth must first be attributed to an exact 
map/store path rather than treated as evidence that finished-history retention 
caused it.
   
   Please add a small evidence table before proposing code: backend and map 
name, entry/byte counts before and after expiry, job mix, retention value, 
restart result, and the WAL files attributed to that map. Also specify the 
REST/CLI contract when detailed metrics have expired but a summary remains, 
including how callers distinguish an expired detail from an unavailable job.
   
   For the first implementation discussion, preserve the existing 
`history-job-expire-minutes` behavior and avoid adding batch/streaming knobs 
until this evidence identifies a material difference. WAL reclamation remains a 
separate concern; please keep it out of a retention-only PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to