[
https://issues.apache.org/jira/browse/KAFKA-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16368692#comment-16368692
]
ASF GitHub Bot commented on KAFKA-6569:
---------------------------------------
ambroff opened a new pull request #4586: KAFKA-6569 Reflection in OffsetIndex
and TimeIndex ctrs
URL: https://github.com/apache/kafka/pull/4586
We identified that we spend a lot of time in reflection when creating
OffsetIndex, TimeIndex, or other implementations of
AbstractIndex[K, V], because of the Logging mixin. When the broker is
bootstrapping it's just doing this in a tight loop, so this time adds
up.
This patch moves the logging to a companion objects, statically
initializing the logger.
*More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.*
*Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.*
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Reflection in OffsetIndex and TimeIndex construction
> ----------------------------------------------------
>
> Key: KAFKA-6569
> URL: https://issues.apache.org/jira/browse/KAFKA-6569
> Project: Kafka
> Issue Type: Bug
> Components: core
> Reporter: Kyle Ambroff-Kao
> Assignee: Kyle Ambroff-Kao
> Priority: Major
> Attachments: after.png, before.png
>
>
> kafka.log.AbstractIndex uses the Logging mixin to lazily initialize loggers
> for any concrete type that inherits from it. This works great, except that
> the LazyLogging trait uses reflection to compute the logger name.
> When you have hundreds of thousands of log segments to load on startup the
> extra cost adds up.
> I've attached flame graphs from broker startup on a broker that has about
> 12TB of log segments to load, and a second flame graph after changing
> AbstractIndex to statically initialize a logger.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
