[ https://issues.apache.org/jira/browse/PROTON-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580640#comment-13580640 ]
Ken Giusti edited comment on PROTON-200 at 2/18/13 2:55 PM: ------------------------------------------------------------ Welcome to my nightmare :) As you can see from the somewhat oldish patch I've included, I've been down this road a bit. You've all hit exactly what I discovered during my testing of the "fix" - that merely balancing the credit across all receivers does not address the issue. The fatal failure case occurs when the number of active receiver links is > the amount of credit granted. One of the features of Messenger is that it provides an easy-to-use level of abstraction from the gritty details of AMQP 1.0. One of those details is link management - by design, the application using messenger doesn't have to worry about managing the links. However, links require credit, and the application has control over that credit by indicating how many messages it can receive - this is the parameter to the Messenger::recv() method. The "disconnect" is that the application currently has no mechanisms for controlling the number of receive links when it is operating as a listener (ie. subscribes to a "~" address). By allowing unlimited links, the fixed sized credit pool (given by recv() parameter) can be exhausted by the external clients connecting in. These are the alternatives I can think of - a couple of which require API changes - I'd certainly like to hear of any better approaches, 'cause I don't like any of mine: 1) Have messenger limit the number of active receive links to <= the current amount of available credit. A problem with this approach is that the amount of credit is variable - it will drop as it is consumed. Once a link is permitted, there's no guarantee that the application will ever replenish credit to the point where all currently active links are satisfied. 2) Change the recv() api to allow an "unlimited" flag, rather than an actual credit count. To implement this, each link would be guaranteed a bit of credit. The problem, as Bozo points out - opens a door to DoS. 3) Keep the fixed pool of credit, fairly distributed amount all receive links. When the amount is credit is insufficient to cover all links, periodically revoke unused credit from those links that have it, and redistribute it to those that do not (The Robin Hood Approach). The difficulty here involves our heuristic for revoking the credit - I'm not sure if we can do this with 100% accuracy. Opinions? was (Author: kgiusti): Welcome to my nightmare :) As you can see from the somewhat oldish patch I've included, I've been down this road a bit. You've all hit exactly what I discovered during my testing of the "fix" - that merely balancing the credit across all receivers does not address the issue. The fatal failure case occurs when the number of active receiver links is > the amount of credit granted. One of the features of Messenger is that it provides an easy-to-use level of abstraction from the gritty details of AMQP 1.0. One of those details is link management - by design, the application using messenger doesn't have to worry about managing the links. However, links require credit, and the application has control over that credit by indicating how many messages it can receive - this is the parameter to the Messenger::recv() method. The "disconnect" is that the application currently has no mechanisms for controlling the number of receive links when it is operating as a listener (ie. subscribes to a "~" address). By allowing unlimited links, the fixed sized credit pool (given by recv() parameter) can be exhausted by the external clients connecting in. These are the alternatives I can think of - a couple of which require API changes - I'd certainly like to hear of any better approaches, 'cause I don't like any of mine: 1) Have messenger limit the number of active receive links to <= the current amount of available credit. A problem with this approach is that the amount of credit is variable - it will drop as it is consumed. Once a link is permitted, there's no guarantee that the application will every replenish credit to the point where all currently active links are satisfied. 2) Change the recv() api to allow an "unlimited" flag, rather than an actual credit count. To implement this, each link would be guaranteed a bit of credit. The problem, as Bozo points out - opens a door to DoS. 3) Keep the fixed pool of credit, fairly distributed amount all receive links. When the amount is credit is insufficient to cover all links, periodically revoke unused credit from those links that have it, and redistribute it to those that done (The Robin Hood Approach). The difficulty here involves our heuristic for revoking the credit - I'm not sure if we can do this with 100% accuracy. Opinions? > [Proton-c] Credit distribution by messenger is not balanced across all links > ---------------------------------------------------------------------------- > > Key: PROTON-200 > URL: https://issues.apache.org/jira/browse/PROTON-200 > Project: Qpid Proton > Issue Type: Bug > Components: proton-c > Affects Versions: 0.3 > Reporter: Ken Giusti > Assignee: Ken Giusti > Fix For: 0.4 > > Attachments: proton-200.patch > > > The method used to distribute credit to receiving links may lead to > starvation when the number of receiving links is > the available credit. > The distribution algorithm always starts with the same link - see > messenger.c::pn_messenger_flow() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira