Hi PoAn, This is truly a valuable KIP! It will be helpful for efficient real-time storage management. The previous comments and your thoughtful responses have already refined the proposal quite well.
I do have one minor scenario to bring to your attention. There could be a situation where there is a delay in remote log size updating. This implies that not only are there still segments present in local storage, but also the size information not precisely up-to-date. This is highly dependent on the update mechanism of the storage calculation in remote storage. Overall, great work on this KIP and looking forward to its further progress. Best, Jimmy Wang On 2025/08/05 15:06:33 PoAn Yang wrote: > Hi Kamal, Chia-Ping, and Satish, > > Thanks for the review and suggestions. > > Moved `IncludeRemoteInfo` out of `DescribableLogDirTopics`. > > chia_03: `PartitionSize != RemoteLogSize + OnlyLocalLogSize`. > The PartitionSize is all local log segments size. It includes part > of remote log segments which haven’t meet local retention. > > chia_04: I prefer to use `onlyLocalSize`, because it gets value > from UnifiedLog#onlyLocalLogSegmentSize. If we use `localSize`, > users may be think that `size = remoteSize + localSize`. > > chia_05: Change both `remoteSize` and `onlyLocalSize` to > optional long. > > 100: Updated the description of `RemoteLogSize`. > > 101: I prefer to align the naming order like > UnifiedLog#onlyLocalLogSegmentSize. > Updated the description of `onlyLocalSize`. > > Thanks, > PoAn > > > On Aug 5, 2025, at 1:29 PM, Satish Duggana <sa...@gmail.com> wrote: > > > > Hi PoAn, > > Thanks for the KIP, this is a valuable feature for operators to get > > better visibility into partition sizes across both local and remote > > storage. While RemoteLogSegmentMetadata provides segment-level details > > to help developers build custom utilities, it's beneficial to enhance > > the existing Kafka utilities to surface more operationally useful > > information. > > > > Overall proposal looks good to me. I have a couple of minor comments. > > > > 100. RemoteLogSize – It would be helpful to enhance the description > > with more detail. For example: > > "The size of the remote log segments for this partition, in bytes. > > Note that some of these segments may still be present in the broker’s > > local storage." > > > > 101. onlyLocalSize – I suggest renaming this to LocalOnlySize for > > better clarity. The description can also be made more explicit, such > > as: > > "The size of the log segments stored only in the broker’s local > > storage for this partition, in bytes. This excludes any data that has > > been offloaded to remote storage." > > > > ~Satish. > > > > On Mon, 4 Aug 2025 at 15:57, Chia-Ping Tsai <ch...@apache.org> wrote: > >> > >> hi PoAn > >> > >> chia_03: what is the difference between `OnlyLocalLogSize` and > >> `PartitionSize`? Am I correct in assuming that `PartitionSize` = > >> RemoteLogSize + OnlyLocalLogSize? > >> > >> chia_04: could you please consider renaming `onlyLocalSize` to `localSize` > >> for consistency? > >> > >> chia_05: should we use optional int as returned type for `remoteSize` and > >> `onlyLocalSize`? If not, could you please add comments to explain the use > >> of "-1"? > >> > >> Best, > >> Chia-Ping > >> > >> > >> On 2025/06/16 14:32:34 PoAn Yang wrote: > >>> Hi all, > >>> > >>> I would like to start a discussion thread about KIP-1187. > >>> > >>> Please take a look and feel free to share any thought. > >>> > >>> https://cwiki.apache.org/confluence/x/sYkhFg > >>> > >>> Thanks, > >>> PoAn > >
