chia7712 commented on code in PR #18012:
URL: https://github.com/apache/kafka/pull/18012#discussion_r1908095683
##########
storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java:
##########
@@ -253,17 +253,33 @@ public void append(long largestOffset,
// append the messages
long appendedBytes = log.append(records);
LOGGER.trace("Appended {} to {} at end offset {}", appendedBytes,
log.file(), largestOffset);
- // Update the in memory max timestamp and corresponding offset.
- if (largestTimestampMs > maxTimestampSoFar()) {
- maxTimestampAndOffsetSoFar = new
TimestampOffset(largestTimestampMs, shallowOffsetOfMaxTimestamp);
- }
- // append an entry to the index (if needed)
- if (bytesSinceLastIndexEntry > indexIntervalBytes) {
- offsetIndex().append(largestOffset, physicalPosition);
- timeIndex().maybeAppend(maxTimestampSoFar(),
shallowOffsetOfMaxTimestampSoFar());
- bytesSinceLastIndexEntry = 0;
+
+ long batchMaxTimestamp = RecordBatch.NO_TIMESTAMP;
+ long batchShallowOffsetOfMaxTimestamp = -1L;
+ // append an entry to the index at batches level (if needed)
+ for (RecordBatch batch : records.batches()) {
+ if (batch.maxTimestamp() > batchMaxTimestamp) {
+ batchMaxTimestamp = batch.maxTimestamp();
+ batchShallowOffsetOfMaxTimestamp = batch.lastOffset();
+ }
+
+ // Update the in memory max timestamp and corresponding offset.
+ if (batchMaxTimestamp > maxTimestampSoFar()) {
+ maxTimestampAndOffsetSoFar = new
TimestampOffset(batchMaxTimestamp, batchShallowOffsetOfMaxTimestamp);
+ }
+
+ if (bytesSinceLastIndexEntry > indexIntervalBytes) {
+ offsetIndex().append(batch.lastOffset(), physicalPosition);
+
+ // max timestamp may not be monotonic, so we need to check
it to avoid the time index append error
+ if (batchMaxTimestamp >= timeIndex().lastEntry().timestamp)
Review Comment:
I think @FrankYang0529 tried to avoid the exception of adding a "smaller"
timestamp. see
https://github.com/apache/kafka/blob/c28d9a348670f9efc599b481366da0af813aa8ae/storage/src/main/java/org/apache/kafka/storage/internals/log/TimeIndex.java#L199
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
