[jira] [Commented] (PROTON-200) [Proton-c] Credit distribution by messenger is not balanced across all links
[ https://issues.apache.org/jira/browse/PROTON-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13838055#comment-13838055 ] ASF subversion and git services commented on PROTON-200: Commit 1547534 from [~kgiusti] in branch 'proton/trunk' [ https://svn.apache.org/r1547534 ] PROTON-200: Java implementation of credit scheduler [Proton-c] Credit distribution by messenger is not balanced across all links Key: PROTON-200 URL: https://issues.apache.org/jira/browse/PROTON-200 Project: Qpid Proton Issue Type: Bug Components: proton-j Affects Versions: 0.3 Reporter: Ken Giusti Assignee: Ken Giusti Fix For: 0.6 Attachments: proton-200.patch, upstream-credit.patch The method used to distribute credit to receiving links may lead to starvation when the number of receiving links is the available credit. The distribution algorithm always starts with the same link - see messenger.c::pn_messenger_flow() -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (PROTON-200) [Proton-c] Credit distribution by messenger is not balanced across all links
[ https://issues.apache.org/jira/browse/PROTON-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13796547#comment-13796547 ] Keith Wall commented on PROTON-200: --- Yesterday's commit (https://svn.apache.org/r1532452) broke an internal Jenkins build as the change makes use of a Python 2.5 language feature (try/except/finally in the same block - support for which was introduced by Python 2.5). We'd understood that we wanted to remain Python 2.4 compatible. We are using python distribution python-2.4.3-24.el5.x86_64 and the box is running Red Hat Enterprise Linux Server release 5.3 (Tikanga). Traceback (most recent call last): File ./tests/python/proton-test, line 596, in ? m = __import__(name, None, None, [dummy]) File /home/.jenkins/jobs/Trunk-Proton-C/workspace/proton/tests/python/proton_tests/__init__.py, line 23, in ? import proton_tests.messenger File /home/.jenkins/jobs/Trunk-Proton-C/workspace/proton/tests/python/proton_tests/messenger.py, line 106 finally: [Proton-c] Credit distribution by messenger is not balanced across all links Key: PROTON-200 URL: https://issues.apache.org/jira/browse/PROTON-200 Project: Qpid Proton Issue Type: Bug Components: proton-c Affects Versions: 0.3 Reporter: Ken Giusti Assignee: Ken Giusti Attachments: proton-200.patch, upstream-credit.patch The method used to distribute credit to receiving links may lead to starvation when the number of receiving links is the available credit. The distribution algorithm always starts with the same link - see messenger.c::pn_messenger_flow() -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (PROTON-200) [Proton-c] Credit distribution by messenger is not balanced across all links
[ https://issues.apache.org/jira/browse/PROTON-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13796674#comment-13796674 ] ASF subversion and git services commented on PROTON-200: Commit 1532733 from r...@apache.org in branch 'proton/trunk' [ https://svn.apache.org/r1532733 ] PROTON-200: removed usage of try except finally with [Proton-c] Credit distribution by messenger is not balanced across all links Key: PROTON-200 URL: https://issues.apache.org/jira/browse/PROTON-200 Project: Qpid Proton Issue Type: Bug Components: proton-c Affects Versions: 0.3 Reporter: Ken Giusti Assignee: Ken Giusti Attachments: proton-200.patch, upstream-credit.patch The method used to distribute credit to receiving links may lead to starvation when the number of receiving links is the available credit. The distribution algorithm always starts with the same link - see messenger.c::pn_messenger_flow() -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (PROTON-200) [Proton-c] Credit distribution by messenger is not balanced across all links
[ https://issues.apache.org/jira/browse/PROTON-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13796708#comment-13796708 ] Rafael H. Schloming commented on PROTON-200: Thanks for catching this, I've removed the offending try/finally. [Proton-c] Credit distribution by messenger is not balanced across all links Key: PROTON-200 URL: https://issues.apache.org/jira/browse/PROTON-200 Project: Qpid Proton Issue Type: Bug Components: proton-c Affects Versions: 0.3 Reporter: Ken Giusti Assignee: Ken Giusti Attachments: proton-200.patch, upstream-credit.patch The method used to distribute credit to receiving links may lead to starvation when the number of receiving links is the available credit. The distribution algorithm always starts with the same link - see messenger.c::pn_messenger_flow() -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (PROTON-200) [Proton-c] Credit distribution by messenger is not balanced across all links
[ https://issues.apache.org/jira/browse/PROTON-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13795407#comment-13795407 ] ASF subversion and git services commented on PROTON-200: Commit 1532452 from r...@apache.org in branch 'proton/trunk' [ https://svn.apache.org/r1532452 ] PROTON-200: modified messenger's credit distribution algorithm to cope with credit scarce scenarios [Proton-c] Credit distribution by messenger is not balanced across all links Key: PROTON-200 URL: https://issues.apache.org/jira/browse/PROTON-200 Project: Qpid Proton Issue Type: Bug Components: proton-c Affects Versions: 0.3 Reporter: Ken Giusti Assignee: Ken Giusti Attachments: proton-200.patch, upstream-credit.patch The method used to distribute credit to receiving links may lead to starvation when the number of receiving links is the available credit. The distribution algorithm always starts with the same link - see messenger.c::pn_messenger_flow() -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (PROTON-200) [Proton-c] Credit distribution by messenger is not balanced across all links
[ https://issues.apache.org/jira/browse/PROTON-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13781780#comment-13781780 ] ASF subversion and git services commented on PROTON-200: Commit 1527535 from r...@apache.org in branch 'proton/trunk' [ https://svn.apache.org/r1527535 ] Modified the engine to track how much credit is drained vs used. This should simplify PROTON-200. [Proton-c] Credit distribution by messenger is not balanced across all links Key: PROTON-200 URL: https://issues.apache.org/jira/browse/PROTON-200 Project: Qpid Proton Issue Type: Bug Components: proton-c Affects Versions: 0.3 Reporter: Ken Giusti Assignee: Ken Giusti Attachments: proton-200.patch, upstream-credit.patch The method used to distribute credit to receiving links may lead to starvation when the number of receiving links is the available credit. The distribution algorithm always starts with the same link - see messenger.c::pn_messenger_flow() -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (PROTON-200) [Proton-c] Credit distribution by messenger is not balanced across all links
[ https://issues.apache.org/jira/browse/PROTON-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13580631#comment-13580631 ] Rafael H. Schloming commented on PROTON-200: Also, given the above breakdown of work, I would only make the first chunk of work a 0.4 blocker. [Proton-c] Credit distribution by messenger is not balanced across all links Key: PROTON-200 URL: https://issues.apache.org/jira/browse/PROTON-200 Project: Qpid Proton Issue Type: Bug Components: proton-c Affects Versions: 0.3 Reporter: Ken Giusti Assignee: Ken Giusti Fix For: 0.4 Attachments: proton-200.patch The method used to distribute credit to receiving links may lead to starvation when the number of receiving links is the available credit. The distribution algorithm always starts with the same link - see messenger.c::pn_messenger_flow() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PROTON-200) [Proton-c] Credit distribution by messenger is not balanced across all links
[ https://issues.apache.org/jira/browse/PROTON-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13580640#comment-13580640 ] Ken Giusti commented on PROTON-200: --- Welcome to my nightmare :) As you can see from the somewhat oldish patch I've included, I've been down this road a bit. You've all hit exactly what I discovered during my testing of the fix - that merely balancing the credit across all receivers does not address the issue. The fatal failure case occurs when the number of active receiver links is the amount of credit granted. One of the features of Messenger is that it provides an easy-to-use level of abstraction from the gritty details of AMQP 1.0. One of those details is link management - by design, the application using messenger doesn't have to worry about managing the links. However, links require credit, and the application has control over that credit by indicating how many messages it can receive - this is the parameter to the Messenger::recv() method. The disconnect is that the application currently has no mechanisms for controlling the number of receive links when it is operating as a listener (ie. subscribes to a ~ address). By allowing unlimited links, the fixed sized credit pool (given by recv() parameter) can be exhausted by the external clients connecting in. These are the alternatives I can think of - a couple of which require API changes - I'd certainly like to hear of any better approaches, 'cause I don't like any of mine: 1) Have messenger limit the number of active receive links to = the current amount of available credit. A problem with this approach is that the amount of credit is variable - it will drop as it is consumed. Once a link is permitted, there's no guarantee that the application will every replenish credit to the point where all currently active links are satisfied. 2) Change the recv() api to allow an unlimited flag, rather than an actual credit count. To implement this, each link would be guaranteed a bit of credit. The problem, as Bozo points out - opens a door to DoS. 3) Keep the fixed pool of credit, fairly distributed amount all receive links. When the amount is credit is insufficient to cover all links, periodically revoke unused credit from those links that have it, and redistribute it to those that done (The Robin Hood Approach). The difficulty here involves our heuristic for revoking the credit - I'm not sure if we can do this with 100% accuracy. Opinions? [Proton-c] Credit distribution by messenger is not balanced across all links Key: PROTON-200 URL: https://issues.apache.org/jira/browse/PROTON-200 Project: Qpid Proton Issue Type: Bug Components: proton-c Affects Versions: 0.3 Reporter: Ken Giusti Assignee: Ken Giusti Fix For: 0.4 Attachments: proton-200.patch The method used to distribute credit to receiving links may lead to starvation when the number of receiving links is the available credit. The distribution algorithm always starts with the same link - see messenger.c::pn_messenger_flow() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PROTON-200) [Proton-c] Credit distribution by messenger is not balanced across all links
[ https://issues.apache.org/jira/browse/PROTON-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13580643#comment-13580643 ] Bozo Dragojevic commented on PROTON-200: DoS is way better than not being able to connect the client no 2 :) More so for a 0.4 release. A way to do that is linked in my first comment above. apologize for external link to the patch, I can include it inline if that's more appropriate. [Proton-c] Credit distribution by messenger is not balanced across all links Key: PROTON-200 URL: https://issues.apache.org/jira/browse/PROTON-200 Project: Qpid Proton Issue Type: Bug Components: proton-c Affects Versions: 0.3 Reporter: Ken Giusti Assignee: Ken Giusti Fix For: 0.4 Attachments: proton-200.patch The method used to distribute credit to receiving links may lead to starvation when the number of receiving links is the available credit. The distribution algorithm always starts with the same link - see messenger.c::pn_messenger_flow() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PROTON-200) [Proton-c] Credit distribution by messenger is not balanced across all links
[ https://issues.apache.org/jira/browse/PROTON-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13580670#comment-13580670 ] Rafael H. Schloming commented on PROTON-200: I'd say first priority for 0.4 is the -1 fix. My sense is re-balancing properly is going to be a 0.5 thing, however if we get the -1 fix in and have extra time and it turns out to be low risk, I'm not opposed to rebalancing for 0.4. On the DoS topic, while I agree there are a number of potential DoSes here, I don't think the -1 patch really makes them any worse. In some ways the DoS issue is orthogonal. Currently we could be DoSed anyways by simply holding connections open without involving links at all. I think to address the DoS issue we need a more general inactivity timer. With that in place it would not only address a more general class of DoS, but in this case it would cause the links held by that client to close and their credit to be reclaimed (i.e. doing a primitive destructive sort of rebalancing). [Proton-c] Credit distribution by messenger is not balanced across all links Key: PROTON-200 URL: https://issues.apache.org/jira/browse/PROTON-200 Project: Qpid Proton Issue Type: Bug Components: proton-c Affects Versions: 0.3 Reporter: Ken Giusti Assignee: Ken Giusti Fix For: 0.4 Attachments: proton-200.patch The method used to distribute credit to receiving links may lead to starvation when the number of receiving links is the available credit. The distribution algorithm always starts with the same link - see messenger.c::pn_messenger_flow() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PROTON-200) [Proton-c] Credit distribution by messenger is not balanced across all links
[ https://issues.apache.org/jira/browse/PROTON-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13580762#comment-13580762 ] Ken Giusti commented on PROTON-200: --- Reviewboad for proposed fix: https://reviews.apache.org/r/9503/ [Proton-c] Credit distribution by messenger is not balanced across all links Key: PROTON-200 URL: https://issues.apache.org/jira/browse/PROTON-200 Project: Qpid Proton Issue Type: Bug Components: proton-c Affects Versions: 0.3 Reporter: Ken Giusti Assignee: Ken Giusti Fix For: 0.4 Attachments: proton-200.patch, upstream-credit.patch The method used to distribute credit to receiving links may lead to starvation when the number of receiving links is the available credit. The distribution algorithm always starts with the same link - see messenger.c::pn_messenger_flow() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PROTON-200) [Proton-c] Credit distribution by messenger is not balanced across all links
[ https://issues.apache.org/jira/browse/PROTON-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13580317#comment-13580317 ] michael goulish commented on PROTON-200: I think this bug is a release blocker. An implication, that is not immediately obvious, is that if you have one receiver with two senders -- one of the senders will hang until the other receiver calls 'stop'. This makes it impossible to set up any topology except the simplest possible -- one sender, one receiver. [Proton-c] Credit distribution by messenger is not balanced across all links Key: PROTON-200 URL: https://issues.apache.org/jira/browse/PROTON-200 Project: Qpid Proton Issue Type: Bug Components: proton-c Affects Versions: 0.3 Reporter: Ken Giusti Assignee: Ken Giusti Fix For: 0.4 The method used to distribute credit to receiving links may lead to starvation when the number of receiving links is the available credit. The distribution algorithm always starts with the same link - see messenger.c::pn_messenger_flow() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PROTON-200) [Proton-c] Credit distribution by messenger is not balanced across all links
[ https://issues.apache.org/jira/browse/PROTON-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13580326#comment-13580326 ] Bozo Dragojevic commented on PROTON-200: For us we could not do anything except 1:1 until we fixed this. There is one other implication that is spread over time: 1) messenger (say a broker) hands out all it's credit (using whatever algorithm) to set of current peers, 2) these peers choose to not send messages to the broker (consider services registered with a broker, waiting for requests) 3) then a future peer (a client trying to send a request message via the broker to the service) will not be able to send, since the broker has handed out all of it's credit. My solution was to severely limit the credit given out to any link so that messenger always has some credit on hand. Not pretty, but it at least allows us to use proton in a hub-and-spokes configuration. The fix seems only a band-aid, tho. Even if each link gets just one credit, and assuming the code uses a constant value for the pn_messenger_recv() call, it's easy to DOS such a broker by just connecting enough clients that do nothing. [Proton-c] Credit distribution by messenger is not balanced across all links Key: PROTON-200 URL: https://issues.apache.org/jira/browse/PROTON-200 Project: Qpid Proton Issue Type: Bug Components: proton-c Affects Versions: 0.3 Reporter: Ken Giusti Assignee: Ken Giusti Fix For: 0.4 The method used to distribute credit to receiving links may lead to starvation when the number of receiving links is the available credit. The distribution algorithm always starts with the same link - see messenger.c::pn_messenger_flow() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PROTON-200) [Proton-c] Credit distribution by messenger is not balanced across all links
[ https://issues.apache.org/jira/browse/PROTON-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13576473#comment-13576473 ] Bozo Dragojevic commented on PROTON-200: My observation was that messenger was willing to hand out all of it's credit to the first link. So when the second client connects, say few seconds later, messenger has no credit left to hand out. This is my attempt at mitigating the issue. https://github.com/ttlj/qpid-proton/tree/upstream-credit [Proton-c] Credit distribution by messenger is not balanced across all links Key: PROTON-200 URL: https://issues.apache.org/jira/browse/PROTON-200 Project: Qpid Proton Issue Type: Bug Components: proton-c Affects Versions: 0.3 Reporter: Ken Giusti Assignee: Ken Giusti Fix For: 0.5 The method used to distribute credit to receiving links may lead to starvation when the number of receiving links is the available credit. The distribution algorithm always starts with the same link - see messenger.c::pn_messenger_flow() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
