fanxishu opened a new issue, #8386: URL: https://github.com/apache/seatunnel/issues/8386
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened Imap The history-job-expire-minutes parameter in the IMAP configuration for HDFS persistence is not functioning properly. As the number of running jobs increases, the memory usage rises, ultimately leading to memory overflow. ### SeaTunnel Version 2.3.8 ### SeaTunnel Config ```conf seatunnel.yaml seatunnel: engine: classloader-cache-mode: true history-job-expire-minutes: 1440 backup-count: 1 queue-type: blockingqueue print-execution-info-interval: 60 print-job-metrics-info-interval: 60 slot-service: dynamic-slot: false slot-num: 8 checkpoint: interval: 300000 timeout: 10000 storage: type: hdfs max-retained: 3 plugin-config: namespace: /tmp/asinainf/dataasync/checkpoint storage.type: hdfs fs.defaultFS: hdfs://10.11.11.19:9000 telemetry: metric: enabled: false map: engine*: map-store: enabled: true initial-mode: EAGER factory-class-name: org.apache.seatunnel.engine.server.persistence.FileMapStoreFactory properties: type: hdfs namespace: /tmp/asiainf/dataasync/imap clusterName: seatunnel storage.type: hdfs fs.defaultFS: hdfs://10.11.11.19:9000 hazelcast.yaml hazelcast: cluster-name: seatunnel network: rest-api: enabled: true endpoint-groups: CLUSTER_WRITE: enabled: true DATA: enabled: true join: tcp-ip: enabled: true member-list: - 10.11.11.19:5801 - 10.12.83.88:5801 port: auto-increment: true port: 5801 map: engine*: map-store: enabled: true initial-mode: EAGER factory-class-name: org.apache.seatunnel.engine.server.persistence.FileMapStoreFactory properties: type: hdfs namespace: /tmp/asiainf/dataasync/imap clusterName: seatunnel storage.type: hdfs fs.defaultFS: hdfs://10.11.11.19:9000 properties: hazelcast.invocation.max.retry.count: 20 hazelcast.tcp.join.port.try.count: 30 hazelcast.logging.type: log4j2 hazelcast.operation.generic.thread.count: 50 hazelcast.heartbeat.failuredetector.type: phi-accrual hazelcast.heartbeat.interval.seconds: 2 hazelcast.max.no.heartbeat.seconds: 180 hazelcast.heartbeat.phiaccrual.failuredetector.threshold: 10 hazelcast.heartbeat.phiaccrual.failuredetector.sample.size: 200 hazelcast.heartbeat.phiaccrual.failuredetector.min.std.dev.millis: 100 ``` ### Running Command ```shell 0569429165604872 15976352660192_job FAILED 2024-12-15 14:54:15.603 2024-12-15 14:56:30.729 919878581574696962 15955269670880_job FAILED 2024-12-13 17:09:04.73 2024-12-13 17:09:48.09 919877656927797250 15955241454304_job FAILED 2024-12-13 17:05:24.276 2024-12-13 17:06:38.813 919876389748867073 15955202781920_job FAILED 2024-12-13 17:00:22.169 2024-12-13 17:00:36.894 919876383730040833 15955202546272_job FAILED 2024-12-13 17:00:21.346 2024-12-13 17:00:36.478 919861655502848002 15954753130592_job FAILED 2024-12-13 16:01:49.214 2024-12-13 16:02:38.001 919861588020690945 15954750837088_job FAILED 2024-12-13 16:01:33.131 2024-12-13 16:01:46.554 919860338105516033 15954712834272_job FAILED 2024-12-13 15:56:35.917 2024-12-13 15:56:49.226 919859761241915394 15954695319776_job CANCELED 2024-12-13 15:54:21.778 2024-12-15 14:52:34.527 919859366633406465 15954683261280_job FAILED 2024-12-13 15:52:43.507 2024-12-13 15:52:57.412 919859116359286785 15954675637472_job FAILED 2024-12-13 15:51:43.93 2024-12-13 15:52:25.853 919788760994938882 15952528450656_job FAILED 2024-12-13 11:12:09.805 2024-12-26 16:06:28.615 919783836689104897 15952378274272_job FAILED 2024-12-13 11:07:48.926 2024-12-13 11:08:01.325 919786312528035842 15952453825376_job FAILED 2024-12-13 11:02:26.103 2024-12-13 11:02:39.723 919782737332666370 15952344712544_job FAILED 2024-12-13 10:48:13.646 2024-12-13 10:48:27.167 919782182703071233 15952327506144_job FAILED 2024-12-13 10:46:01.47 2024-12-13 10:46:14.142 919781852535848961 15952317728992_job FAILED 2024-12-13 10:44:42.682 2024-12-13 10:44:55.638 919780420550131714 15952274028768_job DOING_SAVEPOINT 2024-12-13 10:39:01.264 919779366185992193 15952241852896_job FAILED 2024-12-13 10:34:50.04 2024-12-13 10:35:02.175 919776731403911169 15952161448160_job SAVEPOINT_DONE 2024-12-13 10:24:21.764 2024-12-13 10:35:17.772 919774749586882562 15952100943328_job SAVEPOINT_DONE 2024-12-13 10:16:30.313 2024-12-13 10:20:29.282 919553839047114753 15945359302016_job FAILED 2024-12-12 19:38:39.922 2024-12-12 19:38:51.905 919553615964667906 15945352500096_job FAILED 2024-12-12 19:37:46.739 2024-12-12 19:37:59.258 919552356339679233 15945313951488_job FAILED 2024-12-12 19:32:46.427 2024-12-12 19:33:29.841 ``` ### Error Exception ```log 0569429165604872 15976352660192_job FAILED 2024-12-15 14:54:15.603 2024-12-15 14:56:30.729 919878581574696962 15955269670880_job FAILED 2024-12-13 17:09:04.73 2024-12-13 17:09:48.09 919877656927797250 15955241454304_job FAILED 2024-12-13 17:05:24.276 2024-12-13 17:06:38.813 919876389748867073 15955202781920_job FAILED 2024-12-13 17:00:22.169 2024-12-13 17:00:36.894 919876383730040833 15955202546272_job FAILED 2024-12-13 17:00:21.346 2024-12-13 17:00:36.478 919861655502848002 15954753130592_job FAILED 2024-12-13 16:01:49.214 2024-12-13 16:02:38.001 919861588020690945 15954750837088_job FAILED 2024-12-13 16:01:33.131 2024-12-13 16:01:46.554 919860338105516033 15954712834272_job FAILED 2024-12-13 15:56:35.917 2024-12-13 15:56:49.226 919859761241915394 15954695319776_job CANCELED 2024-12-13 15:54:21.778 2024-12-15 14:52:34.527 919859366633406465 15954683261280_job FAILED 2024-12-13 15:52:43.507 2024-12-13 15:52:57.412 919859116359286785 15954675637472_job FAILED 2024-12-13 15:51:43.93 2024-12-13 15:52:25.853 919788760994938882 15952528450656_job FAILED 2024-12-13 11:12:09.805 2024-12-26 16:06:28.615 919783836689104897 15952378274272_job FAILED 2024-12-13 11:07:48.926 2024-12-13 11:08:01.325 919786312528035842 15952453825376_job FAILED 2024-12-13 11:02:26.103 2024-12-13 11:02:39.723 919782737332666370 15952344712544_job FAILED 2024-12-13 10:48:13.646 2024-12-13 10:48:27.167 919782182703071233 15952327506144_job FAILED 2024-12-13 10:46:01.47 2024-12-13 10:46:14.142 919781852535848961 15952317728992_job FAILED 2024-12-13 10:44:42.682 2024-12-13 10:44:55.638 919780420550131714 15952274028768_job DOING_SAVEPOINT 2024-12-13 10:39:01.264 919779366185992193 15952241852896_job FAILED 2024-12-13 10:34:50.04 2024-12-13 10:35:02.175 919776731403911169 15952161448160_job SAVEPOINT_DONE 2024-12-13 10:24:21.764 2024-12-13 10:35:17.772 919774749586882562 15952100943328_job SAVEPOINT_DONE 2024-12-13 10:16:30.313 2024-12-13 10:20:29.282 919553839047114753 15945359302016_job FAILED 2024-12-12 19:38:39.922 2024-12-12 19:38:51.905 919553615964667906 15945352500096_job FAILED 2024-12-12 19:37:46.739 2024-12-12 19:37:59.258 919552356339679233 15945313951488_job FAILED 2024-12-12 19:32:46.427 2024-12-12 19:33:29.841 ``` ### Zeta or Flink or Spark Version _No response_ ### Java or Scala Version _No response_ ### Screenshots _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
