[
https://issues.apache.org/jira/browse/HELIX-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13871015#comment-13871015
]
Kanak Biscuitwala commented on HELIX-345:
-----------------------------------------
We came up with 4 approaches to compare:
1. The existing method of re-reading the entire cluster
2. Caching messages, purging them if names change
3. Cache all properties, updating them if the stats associated with the
property changes
4. Use message callbacks to mark paths as dirty, and only read those paths
I set up a sample cluster with 100 nodes, 100 resources, 100 partitions per
resource, and 3 replicas per partition.
1. The unchanged read stage can read the cluster in 2.5-5 s, with an average
around 3.5-4 s
2. Caching messages means that one pipeline read will take 4-5 s, but
subsequent reads will take about 0.5s
3. What I found here was that this stage was no faster than the baseline. The
crux of the problem is that reading stats is not a free operation. On average
this would take 2-2.5 s. Combined with reading the changed properties and the
added CPU time to manage the cache, the savings is minimal if it exists at all.
4. This approach suffers from the following problem: if the messages of a node
change, the current state must also be read. However, it also has the benefit
of only impacting reads at a single-node scope. What I realized when I was
testing for correctness here is that this single-node scope causes problems.
Imagine if we send transition messages, leading to message callbacks. These
messages are to multiple nodes, but we can only invalidate the cache for a
single node at a time. So if we queue the callbacks as we currently are, the
controller generates duplicate messages. If we run the pipeline in a new
thread, which we will, the result of this depends on who wins on a race
condition. There are a lot of safety considerations, which ultimately leads to
re-reading a lot of znodes.
Given this exploration, option 2 seems to be the safest and best-performing
strategy.
> Speed up the controller pipelines
> ---------------------------------
>
> Key: HELIX-345
> URL: https://issues.apache.org/jira/browse/HELIX-345
> Project: Apache Helix
> Issue Type: Bug
> Affects Versions: 0.6.2-incubating, 0.7.0-incubating
> Reporter: Kanak Biscuitwala
> Assignee: Kanak Biscuitwala
>
> ReadClusterDataStage can take some time. We should have techniques for
> speeding it up like parallelizing or caching.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
