Arpit Varshney created GOBBLIN-2026:
---------------------------------------
Summary: CLONE - Retention Job should fail on OOM
Key: GOBBLIN-2026
URL: https://issues.apache.org/jira/browse/GOBBLIN-2026
Project: Apache Gobblin
Issue Type: Improvement
Components: misc
Reporter: Arpit Varshney
Currently, while cleaning the log files, the Retention job goes into OOM and
fails when the no of log files is too many.
Retention job while fetching all the dataset versions, loads the file status
all at once into the memory, resulting in this issue.
Thus, the Retention job should avoid loading all data into memory, and use an
iterator-based approach. This will load only limited file status into memory
and making the retention job pipeline more robust to OOM errors
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
