allthingssecurity opened a new pull request, #28596: URL: https://github.com/apache/camel/pull/28596
# Description [CAMEL-26088](https://issues.apache.org/jira/browse/CAMEL-26088) `LogWriter` keeps the last `maxRecordCount` entries of the write-ahead log in `TransactionLog`, a circular buffer with a layer number per loop. After a roll-over, every `append` truncates or rewinds the file and writes the new entry, then writes again, after it, the entries of the previous layer that are still in the buffer. `WriteAheadResumeStrategy` appends an entry for each offset and marks it `PROCESSED` or `FAILED` from the callback of the strategy it wraps, so the state updates of the entries appended just before a roll-over arrive after it. Three things go wrong there: 1. `updateState(CachedEntryInfo, ...)` writes the entry at `entryInfo.getPosition()`, the position of its first append. Once a roll-over has written the entries again, that position holds another record when the records have different sizes (the key and value are the serialized offset key and value). The file can no longer be read: `LogReader` fails with `BufferTooSmallException ... 1667457792 bytes needed`, or reads a wrong record. 2. `TransactionLog.canUpdate` accepts an update from any older layer when its index is above the current index (the index of the next entry to add). An update arriving two or more layers late therefore marks the newer entry that now occupies the slot (a record whose own update is still in flight is marked `PROCESSED`) and writes it at the old record's position. It also rejects the entry at the current index, which the roll-over has not replaced yet: with capacity 3, the update of the second entry after the first append of the next layer is dropped, so the entry stays `NEW` in the log although it was processed. 3. When the new entry is shorter than the one it replaces, the file keeps the end of the previous content after the last entry, and `LogReader` fails on it (`BufferTooSmallException ... 814950 bytes needed`), without any update. So the log file holds wrong states or cannot be read at all. On main this does not yet reach a restart: `WriteAheadResumeStrategy.start()` creates the `LogWriter`, which truncates the file, before `loadCache()` reads it, so the records of the previous run are never replayed (see "Related" below). The fix is for the content of the file, which the replay reads once that is fixed. Changes: - `TransactionLog.canUpdate`: the same layer as before; the previous layer when its index is at or after the current index; never an older layer. The comment says why. - `TransactionLog.EntryContainer` keeps the current position of the entry in the file. `LogWriter.persist` sets it whenever it writes an entry (the new one and the ones of the previous layer written again), and `updateState` writes at that position. `TransactionLog.add` and `update` return the container (the class is package-private). - `append` truncates the file after the entries of the previous layer. - `append`, both `updateState` methods and `reset` take the writer's lock, the one the flush already takes. The update reads the position that `append` moves, and they can run on different threads (the append on the route's thread, the update in the callback of the wrapped strategy, which may complete on its own thread); before, nothing ordered them. - No option, doc or upgrade guide change: nothing for users to change. - Backport: camel-4.23.x, 4.22.x, 4.18.x and 4.14.x have the same `canUpdate` and `updateState` (read, not built). Tests: - New `LogWriterUpdateAfterRollOverPositionTest`, 4 tests through `LogWriter` and `LogReader` with capacity 3, all fail on main: - `updateOfPreviousLayerEntryAfterRollOverKeepsFileConsistent` (records `a`, `bb`, `ccc`, then a 16-character one, then `ccc` is marked `PROCESSED`): `BufferTooSmallException: ... 524288 bytes remaining, 1667457792 bytes needed`; - `updateOfTheNextEntryToReplaceAfterRollOverIsKept`: `expected: <{dd=NEW, bb=PROCESSED, cc=NEW}> but was: <{dd=NEW, bb=NEW, cc=NEW}>`; - `updateOfEntryTwoLayersBehindDoesNotTouchNewerEntry` (seven records, then a late update of the third): `expected: <{g=NEW, e=NEW, f=NEW}> but was: <{g=NEW, e=NEW, f=PROCESSED}>`; - `appendOfAShorterRecordAfterRollOverKeepsFileReadable` (no update at all): `BufferTooSmallException: ... 814950 bytes needed`. - `TransactionLogTest`: two assertions asserted the off-by-one (`canUpdate(1, 3, 0, 3)` and `canUpdate(1, 10, 0, 10)` false, an entry at the next index to add); they are now true, and two cases that must stay false were added (the last replaced index, and two layers behind). These two methods fail on main. - Unit tests of camel-wal with the fix: 27 run, 0 failures, including the existing roll-over and concurrent update tests (`LogWriterRollOverUpdateAsyncWithContentionTest`). - The formatter and import-sort plugins ran. I built offline with `-Djackson3-version=3.2.3` (3.2.4 is not in my local repository) and `-Dasm-version=9.11`. Related, not changed here: `WriteAheadResumeStrategy.start()` (`new LogWriter(...)`, which opens the file with `TRUNCATE_EXISTING`) runs before `loadCache()` (from `ResumableProcessor.doStart` and then the consumer's `ResumeStrategyHelper.resume`), so `loadCache()` reads a file with only the header and nothing of the previous run is replayed. I checked it with a test: three `NEW` records, 135 bytes before `start()`, 12 bytes after, nothing replayed; with the records written back after `start()`, all three are replayed. It is a separate defect with its own fix, so I will open a separate JIRA and PR for it. Not changed: `LogReader` gives each `PersistedLogEntry` the position of the file channel after its buffer was filled (`EntryInfo.createForPersisted(fileChannel.position())`), not the position of the entry, so `updateState(PersistedLogEntry, ...)` while the log is replayed at start looks like it writes at the wrong place too. I found it by reading and did not test it; I can open a JIRA for it. # Target - [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it): [CAMEL-26088](https://issues.apache.org/jira/browse/CAMEL-26088). # Apache Camel coding standards and style - [x] I checked that each commit in the pull request has a meaningful subject line and body. - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. (I built camel-wal and ran its tests, including the formatter and import-sort plugins. No generated file changes. I did not run the full root build.) # AI-assisted contributions - [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer. _Claude Code on behalf of allthingssecurity_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
