Gavin Wang created KAFKA-21273:
----------------------------------
Summary: Improve refiner performance
Key: KAFKA-21273
URL: https://issues.apache.org/jira/browse/KAFKA-21273
Project: Kafka
Issue Type: Sub-task
Reporter: Gavin Wang
Assignee: Gavin Wang
We landed the first version of the refiner w/o any JMH results. If we need to
optimize for better performance a few ideas to explore (some already discussed
during code review):
* replace Java Streams API usage with “native” loops
* replace list+sort with heap
* cache more helper data structures (cf below)
* maybe integrate deeper with reconciler
About helper data structures: currently, the GC uses an in-memory cache to
store the intermediate assignment. We could extent this, to also store helper
data structures like “load index” which change incrementally for most cases,
and we should be able to re-used across refiner runs (atm, the refiner computes
all helpers from scratch on each run). Other candidates for caching is “funding
candidate order” (ie, in which order we plan to assign warmup tasks — it’s
mainly load based so keeping it across refiner runs and maintaining it
incrementally might also cut costs; for this case, a heap might become really
beneficial).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
