Hang Qi created HELIX-596:
-----------------------------
Summary: Message throttling of controller behavior unexpectedly,
throttled messages still take the constraint quota
Key: HELIX-596
URL: https://issues.apache.org/jira/browse/HELIX-596
Project: Apache Helix
Issue Type: Bug
Components: helix-core
Affects Versions: 0.6.4
Reporter: Hang Qi
Fix For: master
We found a very strange behavior on message throttling of controller when there
is multiple constraints. Here is our setup ( we are using helix-0.6.4, only one
resource )
- constraint 1: per node constraint, we only allow 3 state transitions
happens on one node concurrently.
- constraint 2: per partition constraint, we define the state transition
priorities in the state model, and only allow one state transition happens on
one single partition concurrently.
We are using MasterSlave state model, suppose we have two nodes A, B, each has
8 partitions (p0-p7) respectively, and initially both A and B are shutdown, and
now we start them at the same time (say A is slightly earlier than B).
The expected behavior might be
- p0, p1, p2 on A starts from Offline -> Slave; p3, p4, p5 on B starts from
Offline -> Slave
But the real result is:
- p0, p1, p2 on A starts from Offline -> Slave, nothing happens on B
- until p0, p1, p2 all transited to Master state, p3, p4, p5 on A starts from
Offline -> Slave; p0, p1, p2 on B starts from Offline -> Slave
As step Offline -> Slave might take long time, this behavior result in very
long time to bring up these two nodes (long down time result in long catch up
time as well), though ideally we should not let both nodes down at the same
time.
Looked at the controller code, I like the stage and pipeline based
implementation, it is well design, very easy to understand and to reason about.
The logic of MessageThrottleStage#throttle,
- it goes through each messages selected by MessageSelectionStage,
- for each message, it goes through all selected matched constraints, and
decrease the quota of each constraints
- if any constraint's quota is less than 0, this message will be marked as
throttled.
I think there is something wrong here, the message will take the quota of
constraints even it is not going to be sent out (throttled). That explains our
case,
- all the messages have been generated by the beginning, (p0, A,
Offline->Slave), ... (p7, A, Offline->Slave), (p0, B, Offline->Slave), ...,
(p7, B, Offline->Slave)
- in the messageThrottleStage#throttle
- (p0, A, Offline->Slave), (p1, A, Offline->Slave), (p2, A, Offline->Slave)
are good, and constraint 1 on A reaches 0, constraint 2 on p0, p1, p2 reaches 0
as well
- (p3, A, Offline->Slave), ... (p7, A, Offline->Slave) throttled by
constraint 1 on A, also takes the quota of constraint 2 on those partitions as
well.
- (p0, B, Offline->Slave), ... (p7, B, Offline->Slave) throttled by
constraint 2
- thus only (p0, A, Offline->Slave), (p1, A, Oflline->Slave), (p2, A,
Offline->Slave) has been sent out by controller.
Does that make sense, or is there anything else you can think of to result in
this unexpected behavior? And is there any work around for it? One thing comes
into my mind is update constraint 2 to be only one state transition is allowed
of single partition on certain state transitions.
Thanks very much.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
