[
https://issues.apache.org/jira/browse/HELIX-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14554866#comment-14554866
]
ASF GitHub Bot commented on HELIX-596:
--------------------------------------
Github user asfgit closed the pull request at:
https://github.com/apache/helix/pull/28
> Message throttling of controller behavior unexpectedly, throttled messages
> still take the constraint quota
> ----------------------------------------------------------------------------------------------------------
>
> Key: HELIX-596
> URL: https://issues.apache.org/jira/browse/HELIX-596
> Project: Apache Helix
> Issue Type: Bug
> Components: helix-core
> Affects Versions: 0.6.4
> Reporter: Hang Qi
> Fix For: master
>
>
> We found a very strange behavior on message throttling of controller when
> there is multiple constraints. Here is our setup ( we are using helix-0.6.4,
> only one resource )
> - constraint 1: per node constraint, we only allow 3 state transitions
> happens on one node concurrently.
> - constraint 2: per partition constraint, we define the state transition
> priorities in the state model, and only allow one state transition happens on
> one single partition concurrently.
> We are using MasterSlave state model, suppose we have two nodes A, B, each
> has 8 partitions (p0-p7) respectively, and initially both A and B are
> shutdown, and now we start them at the same time (say A is slightly earlier
> than B).
> The expected behavior might be
> - p0, p1, p2 on A starts from Offline -> Slave; p3, p4, p5 on B starts from
> Offline -> Slave
> But the real result is:
> - p0, p1, p2 on A starts from Offline -> Slave, nothing happens on B
> - until p0, p1, p2 all transited to Master state, p3, p4, p5 on A starts
> from Offline -> Slave; p0, p1, p2 on B starts from Offline -> Slave
> As step Offline -> Slave might take long time, this behavior result in very
> long time to bring up these two nodes (long down time result in long catch up
> time as well), though ideally we should not let both nodes down at the same
> time.
> Looked at the controller code, I like the stage and pipeline based
> implementation, it is well design, very easy to understand and to reason
> about.
> The logic of MessageThrottleStage#throttle,
> - it goes through each messages selected by MessageSelectionStage,
> - for each message, it goes through all selected matched constraints, and
> decrease the quota of each constraints
> - if any constraint's quota is less than 0, this message will be marked
> as throttled.
>
> I think there is something wrong here, the message will take the quota of
> constraints even it is not going to be sent out (throttled). That explains
> our case,
> - all the messages have been generated by the beginning, (p0, A,
> Offline->Slave), ... (p7, A, Offline->Slave), (p0, B, Offline->Slave), ...,
> (p7, B, Offline->Slave)
> - in the messageThrottleStage#throttle
> - (p0, A, Offline->Slave), (p1, A, Offline->Slave), (p2, A,
> Offline->Slave) are good, and constraint 1 on A reaches 0, constraint 2 on
> p0, p1, p2 reaches 0 as well
> - (p3, A, Offline->Slave), ... (p7, A, Offline->Slave) throttled by
> constraint 1 on A, also takes the quota of constraint 2 on those partitions
> as well.
> - (p0, B, Offline->Slave), ... (p7, B, Offline->Slave) throttled by
> constraint 2
> - thus only (p0, A, Offline->Slave), (p1, A, Oflline->Slave), (p2, A,
> Offline->Slave) has been sent out by controller.
> Does that make sense, or is there anything else you can think of to result in
> this unexpected behavior? And is there any work around for it? One thing
> comes into my mind is update constraint 2 to be only one state transition is
> allowed of single partition on certain state transitions.
> Thanks very much.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
