dybyte opened a new pull request, #11494:
URL: https://github.com/apache/seatunnel/pull/11494
### Purpose of this pull request
This PR reduces memory usage during file-based IMap WAL recovery.
Previously, WAL files and all deserialized mutations were retained in memory
and sorted. This could cause high heap usage as WAL history grows.
This PR:
- Reads WAL records incrementally.
- Retains only the latest mutation for each key.
- Removes the full mutation list and global sorting.
This reduces recovery memory usage from being proportional to the total
number of WAL mutations to the number of unique keys.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Added unit tests covering:
- Retaining only one latest mutation per key.
- Selecting the latest mutation regardless of WAL file order.
- Selective key loading.
- Recovery with an incomplete trailing WAL record.
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [ ] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the
following files are updated:
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
