Hi Chen, thanks for this well written kip. It definitely addresses a core issue.
I have a few points to clarify. mb-1 : It is mentioned as 'The large format is only written after MetadataVersion IBP_4_4_IV1 is finalized' and 'New index files are written in 12-byte format. Existing 8-byte index files continue to be read correctly.'. But it is not clear how a 8 byte and 12 byte index files are detected by OffsetIndex, after the 4.4 is finalized. Because log dir may contain 8 byte and 12 byte index files simultaneously. Something like a filename suffix ? There is a mention of 5 arg constructor, but it controls the writing path, but no mechanism mentioned for the read path. mb-2 : RemoteIndexCache caches index files fetched from tiered storage. Is it possible that 12-byte index gets cached, but read back as legacy format ? mb-3 : Current default for log.index.size.max.bytes is 10mb and log.index.interval.bytes default is 4 kb. But to handle larger segments, I think this is not enough. Would it be better to mention some guidance or other default in the kip? For ex : guidance on value, for a 8 gb segment, needs this index cap to be around 24mb ? mb-4 : There is a change of type for SegmentPosition.relativePosition (raft) from int to long. But it is not mentioned why this needs a change? mb-5 : As per kip, BaseRecords.sizeInBytes() is not changed and keeps it as int. But when FileRecords size is > 2 gb, curious what should be returned by sizeInBytes ? And 'BaseRecords.sizeInBytes() remains int (449 callers.. ' is there really no impact ? For ex : LogSegment.java:256, we have int physicalPosition = log.sizeInBytes(). If this value is above 2gb, it's value can become wrong, pointing to wrong position ? May be there are other references like these. mb-6 : Also in LogSegment line 446, int startPosition = startOffsetAndSize.position; If this value is above 2gb, probably this gets changed to long ? and in line 460, int fetchSize = Math.min((int) (maxPositionOpt.get() - startPosition), adjustedMaxSize); these kinds of changes could be captured while implementing may be, Just thought, if they could be mentioned in kip, so we don't miss while implementing ? Thanks, Murali On Wed, May 6, 2026 at 12:35 PM Haifeng Chen <[email protected]> wrote: > Hi all, > > I'd like to propose KIP-1333: Support log segments larger than 2 GB. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1333%3A+Support+log+segments+larger+than+2+GB > > Please help take a look and look forward to your feedback. Thanks, > > Haifeng >
