SEZ9 commented on issue #10329: URL: https://github.com/apache/seatunnel/issues/10329#issuecomment-5691585203
@albgen thanks for the follow-up measurements — the record-count framing is useful. `FileMapStore.loadAll()` re-reading and deserializing the entire WAL on every call explains why the startup peak tracks record count rather than raw bytes, and ~690 k records/day on `engine_checkpoint-id-map` at a 5 s checkpoint interval means the same "invisible until restart" failure mode you hit with `engine_runningJobMetrics` is just delayed, not avoided, once the metrics maps are excluded. On your point that `engine_checkpoint-id-map` may not need persistence at all because `CheckpointManager` overwrites the counter with `latestCheckpointId + 1` on restore: I'd like to pin that down before recommending exclusion as a general workaround. Could you: 1. Confirm the restore path you traced — specifically whether the overwrite happens for every job that is restored (including jobs whose last checkpoint was a savepoint/failed), or only in the normal completed-checkpoint case. If there is any path where the persisted counter is read before checkpoint storage is consulted, excluding the map would risk checkpoint-id reuse. 2. If you already tested a start with `engine_checkpoint-id-map` excluded from the map-store on 2.3.13, share whether the restored jobs continued with monotonically increasing checkpoint ids after restart. Independently of the exclusion question, the numbers you posted (13 GB for ~40 live entries, 2.45 M records for a counter map) reinforce that compaction/snapshotting of the file map-store WAL is the real fix — excluding individual maps only removes the biggest offenders, and any `engine*` map with a high update rate will eventually reproduce the same unbootable state. I'd like to keep this issue focused on that, and treat per-map exclusions as mitigations documented alongside it. Thanks also for narrowing the scope on `engine_finishedJobMetrics` and opening the docs PR — I'll look at it separately. <!-- streview-comment:1085 --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
