cheeeee opened a new pull request, #23446:
URL: https://github.com/apache/kafka/pull/23446
`readInto` fills the buffer to the end of the file, not the end of the
records it
was called on, so it returns preallocated space or bytes appended after the
instance was created. Bound it to `sizeInBytes() - position`.
This is the cause of KAFKA-13664. With `log.preallocate=true`,
`CoordinatorLoaderImpl` reads the segment's zero tail and the load dies:
```
Failed to load metadata from __consumer_offsets-0 with epoch 1 due to
Record size 0 is less than the minimum record overhead (14)
```
followed by `Unloading group metadata` for every group on the partition.
There is
no retry: `CoordinatorRuntime` moves the context to `FAILED`, so the
partition
stays unloaded until leadership moves again.
Three brokers, replication factor 3, the coordinator killed so a neighbour
loads
the partition from its own live preallocated replica. Same commit, one
change:
| | trunk | patched |
|------------------------------|-------|---------|
| Failed to load metadata | 4/4 | 0/4 |
| Finished loading of metadata | 0/4 | 4/4 |
On the released 4.3.1 the same scenario is 2/2 without the fix and 0/2 with
the
bound backported.
The recovery path is unaffected: `LogSegment.recover` reads batch headers
from the
channel rather than through `readInto`. Measured on an unclean restart with
`log.preallocate=true`, before and after the patch: identical, two
`CorruptRecordException` handled by recover, segment truncated to 1330002
bytes,
all 5000 records readable, broker up.
`TransactionStateManager` uses the same call pattern, so
`__transaction_state`
loading looks exposed as well; I have not reproduced that.
12 new cases in `FileRecordsTest` and 2 in `LogSegmentTest`; those plus
`:clients:checkstyleMain`, `:clients:checkstyleTest` and
`:storage:checkstyleTest`
pass on `c130b7382c`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
