Re: messenger store and links
we don't have that commit yet. last merge from trunk was PROTON-440: fix from hiram git-svn-id: https://svn.apache.org/repos/asf/qpid/proton/trunk@1531508 13f79535-47bb-0310-9956-ffa450edef68 maybe this trace of an unsuspecting client (recv) connecting to a broker in 'hosed' state can illustrate. what happened before is this: broker was started. a publisher and a slow subscriber are started, they publish approx 2000 messages. when publisher is done it disconnects gracefully and the slow subscriber (living in a daemon thread) just dies. Broker is left alone with approx 800 messages inside messenger's store. broker is unaware the messages exist, thinks all is well (until keepalive timeout kicks in). $ PN_TRACE_FRM=1 ./recv amqp://127.0.0.1:8194 2>&1 | head -200 Connected to 127.0.0.1:8194 -> SASL [0x7fd051814600:0] -> @sasl-init(65) [mechanism=:ANONYMOUS, initial-response=b""] <- SASL [0x7fd051814600:0] <- @sasl-mechanisms(64) [sasl-server-mechanisms=@PN_SYMBOL[:ANONYMOUS]] [0x7fd051814600:0] <- @sasl-outcome(68) [code=0] <- AMQP [0x7fd05180e600:0] <- @open(16) [container-id="8FC1DAF7-1783-41D7-A892-9CDC45BCAEA7"] -> AMQP [0x7fd05180e600:0] -> @open(16) [container-id="CEF14E1A-4D53-44AE-A513-F82C489AEB63", hostname="127.0.0.1"] [0x7fd05180e600:0] -> @begin(17) [next-outgoing-id=0, incoming-window=2147483647, outgoing-window=0] [0x7fd05180e600:0] -> @attach(18) [name="receiver-xxx", handle=0, role=true, snd-settle-mode=2, rcv-settle-mode=0, source=@source(40) [durable=0, timeout=0, dynamic=false], target=@target(41) [durable=0, timeout=0, dynamic=false], initial-delivery-count=0] [0x7fd05180e600:0] -> @flow(19) [incoming-window=2147483647, next-outgoing-id=0, outgoing-window=0, handle=0, delivery-count=0, link-credit=1024, drain=false] [0x7fd05180e600:0] <- @begin(17) [remote-channel=0, next-outgoing-id=0, incoming-window=2147483647, outgoing-window=1] [0x7fd05180e600:0] <- @attach(18) [name="receiver-xxx", handle=0, role=false, snd-settle-mode=2, rcv-settle-mode=0, initial-delivery-count=0] [0x7fd05180e600:0] <- @transfer(20) [handle=0, delivery-id=0, delivery-tag=b"w\x00\x00\x00\x00\x00\x00\x00", message-format=0, settled=true, more=false] (551) "\x00Sp\xd0\x00\x00\x00\x0b\x00\x00\x00\x05BP\x04@BR\x00\x00Ss\xd0\x00\x00\x00z\x00\x00\x00\x0d@@\xa1+amqp://d80a10f2-5e82-4fa4-a3d1-03ff188bbffa@\xa1+amqp://8FC1DAF7-1783-41D7-A892-9CDC45BCAEA7@@@\x83\x00\x00\x00\x00\x00\x00\x00\x00\x83\x00\x00\x00\x00\x00\x00\x00\x00@R\x00@\x00Sw\xd0\x00\x00\x01\x8a\x00\x00\x00\x06q\x00\x00\x00\x04q\x00\x00\x00\x08`\x00\x01\xa1\x0d"... (truncated) [0x7fd05180e600:0] <- @transfer(20) [handle=0, delivery-id=1, delivery-tag=b"x\x00\x00\x00\x00\x00\x00\x00", message-format=0, settled=true, more=false] (551) "\x00Sp\xd0\x00\x00\x00\x0b\x00\x00\x00\x05BP\x04@BR\x00\x00Ss\xd0\x00\x00\x00z\x00\x00\x00\x0d@@\xa1+amqp://d80a10f2-5e82-4fa4-a3d1-03ff188bbffa@\xa1+amqp://8FC1DAF7-1783-41D7-A892-9CDC45BCAEA7@@@\x83\x00\x00\x00\x00\x00\x00\x00\x00\x83\x00\x00\x00\x00\x00\x00\x00\x00@R\x00@\x00Sw\xd0\x00\x00\x01\x8a\x00\x00\x00\x06q\x00\x00\x00\x04q\x00\x00\x00\x08`\x00\x01\xa1\x0d"... (truncated) [0x7fd05180e600:0] <- @transfer(20) [handle=0, delivery-id=2, delivery-tag=b"y\x00\x00\x00\x00\x00\x00\x00", message-format=0, settled=true, more=false] (551) "\x00Sp\xd0\x00\x00\x00\x0b\x00\x00\x00\x05BP\x04@BR\x00\x00Ss\xd0\x00\x00\x00z\x00\x00\x00\x0d@@\xa1+amqp://d80a10f2-5e82-4fa4-a3d1-03ff188bbffa@\xa1+amqp://8FC1DAF7-1783-41D7-A892-9CDC45BCAEA7@@@\x83\x00\x00\x00\x00\x00\x00\x00\x00\x83\x00\x00\x00\x00\x00\x00\x00\x00@R\x00@\x00Sw\xd0\x00\x00\x01\x8a\x00\x00\x00\x06q\x00\x00\x00\x04q\x00\x00\x00\x08`\x00\x01\xa1\x0d"... (truncated) ... lots more Note the two addresses in the transfer frames. The 'to' address is lowercase and distinct from either container-id in the open frames. This is the address of the already gone slow subscriber. The 'reply_to' address is set and is from the broker, which is consistent with not having the change from PROTON-278 Below is the slice of the broker trace of the same thing. There is the extra printf from pn_messenger_endpoints before it calls pni_pump_out Accepted from localhost:53480 <- SASL [0x7fbbc9846200:0] <- @sasl-init(65) [mechanism=:ANONYMOUS, initial-response=b""] [0x7fbbc9846200:0] -> @sasl-mechanisms(64) [sasl-server-mechanisms=@PN_SYMBOL[:ANONYMOUS]] [0x7fbbc9846200:0] -> @sasl-outcome(68) [code=0] -> SASL -> AMQP [0x7fbbc9840c00:0] -> @open(16) [container-id="8FC1DAF7-1783-41D7-A892-9CDC45BCAEA7"] <- AMQP [0x7fbbc9840c00:0] <- @open(16) [container-id="CEF14E1A-4D53-44AE-A513-F82C489AEB63", hostname="127.0.0.1"] [0x7fbbc9840c00:0] <- @begin(17) [next-outgoing-id=0, incoming-window=2147483647, outgoing-window=0] [0x7fbbc9840c00:0] <- @attach(18) [name="receiver-xxx", handle=0, role=true, snd-settle-mode=2, rcv-settle-mode=0, source=@source(40) [durable=0, timeout=0, dynamic=false], target=@target(41) [durable=0, timeout=
Re: messenger store and links
If this is with trunk I'm guessing you might be noticing a change in reply-to behaviour from a recent fix: https://issues.apache.org/jira/browse/PROTON-278 As you mention, previously an unset reply-to would get automatically filled in with the messenger's name. This is no longer the case. That behaviour was unintentional as there are times when you legitimately want the reply-to to be left unset. The intended behaviour was to expand "~" at the beginning of an address to the messenger's name. That is now how trunk behaves, so if you set your reply-to's to "~" then your problem might go away, although your question is still an interesting one as I believe if you wished you could intentionally set up competing receivers using explicit non-uuid addresses that collide. --Rafael On Thu, Oct 24, 2013 at 2:49 PM, Bozo Dragojevic wrote: > All messengers are created with default constructor (uuid-based names). > The 'broker' messenger does a pn_messenger_subscribe("amqp:/**/~ > 0.0.0.0:8194") > All messages are constructed with address amqp://127.0.0.1:8194 and leave > reply_to unset (so it's set to amqp://$uuid) > > Broker does application-level routing of messages > a publisher sends a special 'register' message > replies are constructed using stored 'reply_to' address from incoming > message > forwarded messages are constructed using stored 'reply_to' address from > incoming 'registration' messages > > Messenger routing facility is not used in any way. > All Messengers are running in async mode (broker and client library share > the same 'event loop code'). > We're using outgoing trackers, mostly for the 'buffered' check > All incoming messages are accepted as soon as they are processed. > All outgoing messages are settled as soon as they are not buffered anymore > > maybe it'd be possible to simulate the situation by commenting out the > pni_pump_out() in pn_messenger_put(), that, or at least checking if sender > link address really has anything to do with > the address calculated in pn_messenger_put() > > Bozzo > > > On 24. 10. 13 20:25, Rafael Schloming wrote: > >> Can you post the exact addresses and routing configuration you're using >> and >> which direction messages are flowing? I'd like to try to simulate this >> using the example send/recv scripts. >> >> My guess is that the issue may not be so much related to whether the >> addresses are NULL or not but whether there are multiple receivers >> competing for the same messages. >> >> --Rafael >> >> >> On Thu, Oct 24, 2013 at 11:52 AM, Bozo Dragojevic** >> wrote: >> >> Hi! >>> >>> Chasing down a weird behavior... >>> >>> looking at messengers pni_pump_out() and how it's used from >>> pn_messenger_endpoints() >>> >>> link = pn_link_head(conn, PN_LOCAL_ACTIVE | PN_REMOTE_ACTIVE); >>> while (link) { >>> if (pn_link_is_sender(link)) { >>> pni_pump_out(messenger, pn_terminus_get_address(pn_ >>> link_target(link)), >>> >>> link); >>> >>> >>> is it really fair to assume that target address is always expected to be >>> non NULL? >>> >>> >>> I've added a bit of debug code to pn_messenger_endpoints() so it reads: >>> >>>link = pn_link_head(conn, PN_LOCAL_ACTIVE | PN_REMOTE_ACTIVE); >>>while (link) { >>> if (pn_link_is_sender(link)) { >>>static int addrnull, addrok; >>>const char *address = pn_terminus_get_address(pn_** >>> >>> link_target(link)); >>>if (!address) { >>> addrnull++; >>>} else { >>> addrok++; >>>} >>>fprintf(stderr, "links with null address: %d links with ok address >>> %d\n", >>>addrnull, addrok); >>>pni_pump_out(messenger, address, link); >>> >>> >>> and I never see 'addrok' change from 0 >>> >>> >>> when pni_pump_out is called with address==NULL: >>> >>> int pni_pump_out(pn_messenger_t *messenger, const char *address, >>> pn_link_t >>> *sender) >>> { >>>pni_entry_t *entry = pni_store_get(messenger->outgoing, address); >>> >>>if (!entry) return 0; >>> >>> pni_store_get cheerfuly returns first message on the list >>> >>> >>> end effect is that random clients start receiving messages not directed >>> at >>> them. >>> >>> >>> For some inexplicable reason is mostly works out while there are just two >>> clients >>> connected to the messenger and we're not pushing it really hard. Still >>> trying to come >>> up with a simple test-case. >>> >>> Can anyone shed some light how the addressing on the link level is >>> supposed to work in mesenger? >>> >>> Bozzo >>> >>> >
Re: messenger store and links
All messengers are created with default constructor (uuid-based names). The 'broker' messenger does a pn_messenger_subscribe("amqp://~0.0.0.0:8194") All messages are constructed with address amqp://127.0.0.1:8194 and leave reply_to unset (so it's set to amqp://$uuid) Broker does application-level routing of messages a publisher sends a special 'register' message replies are constructed using stored 'reply_to' address from incoming message forwarded messages are constructed using stored 'reply_to' address from incoming 'registration' messages Messenger routing facility is not used in any way. All Messengers are running in async mode (broker and client library share the same 'event loop code'). We're using outgoing trackers, mostly for the 'buffered' check All incoming messages are accepted as soon as they are processed. All outgoing messages are settled as soon as they are not buffered anymore maybe it'd be possible to simulate the situation by commenting out the pni_pump_out() in pn_messenger_put(), that, or at least checking if sender link address really has anything to do with the address calculated in pn_messenger_put() Bozzo On 24. 10. 13 20:25, Rafael Schloming wrote: Can you post the exact addresses and routing configuration you're using and which direction messages are flowing? I'd like to try to simulate this using the example send/recv scripts. My guess is that the issue may not be so much related to whether the addresses are NULL or not but whether there are multiple receivers competing for the same messages. --Rafael On Thu, Oct 24, 2013 at 11:52 AM, Bozo Dragojevicwrote: Hi! Chasing down a weird behavior... looking at messengers pni_pump_out() and how it's used from pn_messenger_endpoints() link = pn_link_head(conn, PN_LOCAL_ACTIVE | PN_REMOTE_ACTIVE); while (link) { if (pn_link_is_sender(link)) { pni_pump_out(messenger, pn_terminus_get_address(pn_**link_target(link)), link); is it really fair to assume that target address is always expected to be non NULL? I've added a bit of debug code to pn_messenger_endpoints() so it reads: link = pn_link_head(conn, PN_LOCAL_ACTIVE | PN_REMOTE_ACTIVE); while (link) { if (pn_link_is_sender(link)) { static int addrnull, addrok; const char *address = pn_terminus_get_address(pn_** link_target(link)); if (!address) { addrnull++; } else { addrok++; } fprintf(stderr, "links with null address: %d links with ok address %d\n", addrnull, addrok); pni_pump_out(messenger, address, link); and I never see 'addrok' change from 0 when pni_pump_out is called with address==NULL: int pni_pump_out(pn_messenger_t *messenger, const char *address, pn_link_t *sender) { pni_entry_t *entry = pni_store_get(messenger->**outgoing, address); if (!entry) return 0; pni_store_get cheerfuly returns first message on the list end effect is that random clients start receiving messages not directed at them. For some inexplicable reason is mostly works out while there are just two clients connected to the messenger and we're not pushing it really hard. Still trying to come up with a simple test-case. Can anyone shed some light how the addressing on the link level is supposed to work in mesenger? Bozzo
Re: messenger store and links
Can you post the exact addresses and routing configuration you're using and which direction messages are flowing? I'd like to try to simulate this using the example send/recv scripts. My guess is that the issue may not be so much related to whether the addresses are NULL or not but whether there are multiple receivers competing for the same messages. --Rafael On Thu, Oct 24, 2013 at 11:52 AM, Bozo Dragojevic wrote: > Hi! > > Chasing down a weird behavior... > > looking at messengers pni_pump_out() and how it's used from > pn_messenger_endpoints() > >link = pn_link_head(conn, PN_LOCAL_ACTIVE | PN_REMOTE_ACTIVE); >while (link) { > if (pn_link_is_sender(link)) { >pni_pump_out(messenger, > pn_terminus_get_address(pn_**link_target(link)), > link); > > > is it really fair to assume that target address is always expected to be > non NULL? > > > I've added a bit of debug code to pn_messenger_endpoints() so it reads: > > link = pn_link_head(conn, PN_LOCAL_ACTIVE | PN_REMOTE_ACTIVE); > while (link) { > if (pn_link_is_sender(link)) { > static int addrnull, addrok; > const char *address = pn_terminus_get_address(pn_** > link_target(link)); > if (!address) { > addrnull++; > } else { > addrok++; > } > fprintf(stderr, "links with null address: %d links with ok address > %d\n", > addrnull, addrok); > pni_pump_out(messenger, address, link); > > > and I never see 'addrok' change from 0 > > > when pni_pump_out is called with address==NULL: > > int pni_pump_out(pn_messenger_t *messenger, const char *address, pn_link_t > *sender) > { > pni_entry_t *entry = pni_store_get(messenger->**outgoing, address); > if (!entry) return 0; > > pni_store_get cheerfuly returns first message on the list > > > end effect is that random clients start receiving messages not directed at > them. > > > For some inexplicable reason is mostly works out while there are just two > clients > connected to the messenger and we're not pushing it really hard. Still > trying to come > up with a simple test-case. > > Can anyone shed some light how the addressing on the link level is > supposed to work in mesenger? > > Bozzo >