FrankYang0529 commented on code in PR #18012:
URL: https://github.com/apache/kafka/pull/18012#discussion_r1908984798
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -253,17 +253,33 @@ public void append(long largestOffset,
// append the messages
long appendedBytes = log.append(records);
LOGGER.trace("Appended {} to {} at end offset {}", appendedBytes,
log.file(), largestOffset);
- // Update the in memory max timestamp and corresponding offset.
- if (largestTimestampMs > maxTimestampSoFar()) {
- maxTimestampAndOffsetSoFar = new
TimestampOffset(largestTimestampMs, shallowOffsetOfMaxTimestamp);
- }
- // append an entry to the index (if needed)
- if (bytesSinceLastIndexEntry > indexIntervalBytes) {
- offsetIndex().append(largestOffset, physicalPosition);
- timeIndex().maybeAppend(maxTimestampSoFar(),
shallowOffsetOfMaxTimestampSoFar());
- bytesSinceLastIndexEntry = 0;
+
+ long batchMaxTimestamp = RecordBatch.NO_TIMESTAMP;
+ long batchShallowOffsetOfMaxTimestamp = -1L;
+ // append an entry to the index at batches level (if needed)
+ for (RecordBatch batch : records.batches()) {
+ if (batch.maxTimestamp() > batchMaxTimestamp) {
+ batchMaxTimestamp = batch.maxTimestamp();
+ batchShallowOffsetOfMaxTimestamp = batch.lastOffset();
+ }
+
+ // Update the in memory max timestamp and corresponding offset.
+ if (batchMaxTimestamp > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new
TimestampOffset(batchMaxTimestamp, batchShallowOffsetOfMaxTimestamp);
+ }
+
+ if (bytesSinceLastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(batch.lastOffset(), physicalPosition);
+
+ // max timestamp may not be monotonic, so we need to check
it to avoid the time index append error
+ if (batchMaxTimestamp >= timeIndex().lastEntry().timestamp)
Review Comment:
Yes, like @chia7712 mentioned, if there is timestamp index in the file and
new timestamp is less than old one, `TimeIndex#maybeAppend` will throw error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
