dybyte opened a new pull request, #13031:
URL: https://github.com/apache/seatunnel/pull/13031
Close #13017
### Purpose of this pull request
`engine_checkpoint-id-map` holds each pipeline's checkpoint ID counter. It
matches the `engine*` MapStore config, so every checkpoint trigger appended a
write to the IMap storage (#12022: 492 MB in 8 days with 40 streaming jobs).
Since #12387, restore rebuilds the counter from the latest completed checkpoint
in checkpoint storage, so `FileMapStore` now always uses `NoOpMapStorage` for
it. Docs (en/zh) are updated.
### Does this PR introduce _any_ user-facing change?
Yes. After a full cluster restart, restored jobs continue the checkpoint IDs
from their latest completed checkpoint; a pipeline without a completed
checkpoint starts again from 1. Recovery is not affected.
### How was this patch tested?
`FileMapStoreTest` (new restart test) and
`ClusterFaultToleranceIT#testStreamJobRestoreInAllNodeDown`, run locally. The
IT now checks that the first checkpoint after the restart continues the IDs; it
fails if the counter restore is removed.
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the
following files are updated:
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
