dybyte opened a new issue, #13013: URL: https://github.com/apache/seatunnel/issues/13013
### Search before asking - [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement. ### Description `engine_checkpoint_monitor` holds checkpoint overview/history for the REST API, CLI and telemetry. Nothing on the recovery path reads it, yet the documented `engine*` MapStore config persists it, and it is expensive to persist: - `HazelcastCheckpointOverviewStateStore.updateOverview` rewrites the whole `CheckpointOverview` on every trigger, every subtask ack and every completion, i.e. N + 2 writes per checkpoint (N = subtasks). - With the default history size (32), one value is about 1.5 KB. A job with 20 subtasks and a 10 s checkpoint interval appends about 280 MB/day to the append-only WAL, which is fully replayed on restart. Without persistence, a full cluster restart only loses checkpoint statistics. Running jobs fill them again on their next checkpoint. **Proposal:** treat it like `engine_runningJobMetrics` (#11244): in `FileMapStore.init`, always use `NoOpMapStorage` for `engine_checkpoint_monitor`. Both maps are the auxiliary state stores in `AuxiliaryStateStores`. An exact-name opt-in could be added later as a compatible extension if someone needs monitor data to survive a full restart; it is left out for now because enabling it brings the write amplification back. Behavior change: monitor data is no longer reloaded after a full cluster restart, whatever the map-store config. The optional `engine_checkpoint_monitor` override in the Kubernetes configuration docs becomes unnecessary and will be updated (en/zh), and the change will be noted in `incompatible-changes.md`. ### Usage Scenario Clusters running streaming jobs with IMap persistence enabled via `engine*`, where checkpoint monitor writes dominate WAL growth and restart time. ### Related issues #11244 (same approach for engine_runningJobMetrics), #12022, #12328, #10767 ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
