Re: Proton-c Null Messages
On Thu, 2015-06-11 at 14:45 +0100, Gordon Sim wrote: > On 06/11/2015 01:54 PM, aconway wrote: > > On Thu, 2015-06-11 at 13:40 +0100, Gordon Sim wrote: > > > If a name field is populated with an empty string, that to me is > > > the > > > same as not supplying a name. An empty string is a legal > > > encoding, > > > but > > > in my view it does not supply a value at all. (It is not like say > > > 0 > > > which may be the default but is clearly a value in its own > > > right). > > > > > > > It is exactly like 0, a perfectly legal value that is often abused > > to > > mean something special. > > It is treated as meaning 'empty' i.e. 'there is nothing here'. I > don't > consider that abuse myself. > If a string is a sequence of chars, the > empty string is not a sequence of chars, there are no chars specified > in > sequence, therefore it clearly is 'special'. The distinction between > an > empty string and null is an artificial one and in my view anything > for > that relies on that difference to convey something logically > significant > is poorly designed. > 0 does not mean "there is no number here", even though numbers are for counting things and 0 means there are no things. Similarly "" does not mean "there is no string here" it means there is a string and it is empty. "Abuse" was a poor choice of word - it is fine to assign special meanings to 0, negative numbers, "" etc. but you have to *say* that in the API or spec. The spec could have said "a delivery tag is a non -empty binary" and I would agree with you, but it just says "a delivery tag is a binary." I also agree that "" is a terrible choice of delivery tag and we shouldn't ever *use* it (precisely because other people may not make the distinction) but for interop purposes I think we have to accept it. Be strict with what you send forgiving with what you receive. > In any case there is no harm in handling an empty string as a valid > delivery tag (as long as it is unique). I have no objection to that. > > The only part I personally consider important is that proton-c > doesn't > crash. Agreed.
Re: Proton-c Null Messages
On 06/11/2015 08:47 PM, logty wrote: I used valgrind and the segfault isn't directly proton's fault, the problem is we are getting back null messages from pn_messenger_get, and proton seg-faults when I call pn_message_get_content_type on that message. Ok, thanks for the information!
Re: Proton-c Null Messages
I used valgrind and the segfault isn't directly proton's fault, the problem is we are getting back null messages from pn_messenger_get, and proton seg-faults when I call pn_message_get_content_type on that message. -- View this message in context: http://qpid.2158936.n2.nabble.com/Proton-c-Null-Messages-tp7625967p7626329.html Sent from the Apache Qpid Proton mailing list archive at Nabble.com.
Re: Proton-c Null Messages
On 06/09/2015 07:54 PM, Gordon Sim wrote: On 06/09/2015 06:40 PM, logty wrote: When I run the client I get: [0x5351db0]:0 <- @transfer(20) [handle=0, delivery-id=0, delivery-tag=b"", message-format=0, settled=true, more=true] (16363) "\x00Sp\xc0\x07\x05B..." My guess would be that it is the delivery tag being null (or empty, can't tell which) that is the problem. From the spec: "This field MUST be specified for the first transfer of a multi-transfer message and can only be omitted for continuation transfers." [section 2.7.5] So I think that whatever is sending that frame has a bug. Proton-c has a bug too of course, since it shouldn't segfault but should close the connection with a framing-error or similar. Can you get a backtrace for the segfault? (I've been unable to reproduce so far).
Re: Proton-c Null Messages
On 06/11/2015 01:54 PM, aconway wrote: On Thu, 2015-06-11 at 13:40 +0100, Gordon Sim wrote: If a name field is populated with an empty string, that to me is the same as not supplying a name. An empty string is a legal encoding, but in my view it does not supply a value at all. (It is not like say 0 which may be the default but is clearly a value in its own right). It is exactly like 0, a perfectly legal value that is often abused to mean something special. It is treated as meaning 'empty' i.e. 'there is nothing here'. I don't consider that abuse myself. If a string is a sequence of chars, the empty string is not a sequence of chars, there are no chars specified in sequence, therefore it clearly is 'special'. The distinction between an empty string and null is an artificial one and in my view anything for that relies on that difference to convey something logically significant is poorly designed. "" is a legal string literal in every language I know of. It can be used as a key in a map or hash table. It can be compared with other strings. There is no string operation in any language I know that will throw NotAString if you apply it to "". Those are fair points and I withdraw the point I made about not being a 'value', since that has a specific technical meaning and as you point out the empty string would clearly qualify. However I still believe that logically an empty string and null are two different ways of not supplying any information. I don't consider either as acceptable if an item of data is stated as being required. However, from the practical point of view... This is very practical. Interoperability is about agreeing on a type system. A "type" defines a range of legal values. The AMQP type system includes empty string and 0-length binaries as legal values for those types. We absolutely cannot treat any legal value in an exceptional way unless that is clearly mandated by the spec. The wording in the spec could certainly be more precise on the point. To me it is still clear, but we will not reach agreement on that. In any case there is no harm in handling an empty string as a valid delivery tag (as long as it is unique). I have no objection to that. The only part I personally consider important is that proton-c doesn't crash.
Re: Proton-c Null Messages
On Thu, 2015-06-11 at 13:40 +0100, Gordon Sim wrote: > On 06/11/2015 01:11 PM, aconway wrote: > > I disagree. An empty string is a perfectly legal value for a > > string. If > > the spec wants to assign special meaning to particular values of a > > property that needs to be stated. Of course, like you, I personally > > would not use an empty string as an identifier but as an > > implementor of > > an inter-operable spec I think we have to take the large view: > > *any* > > legal value of a parameter has to be considered equal unless the > > spec > > clearly states otherwise. > > If a name field is populated with an empty string, that to me is the > same as not supplying a name. An empty string is a legal encoding, > but > in my view it does not supply a value at all. (It is not like say 0 > which may be the default but is clearly a value in its own right). > It is exactly like 0, a perfectly legal value that is often abused to mean something special. "" is a legal string literal in every language I know of. It can be used as a key in a map or hash table. It can be compared with other strings. There is no string operation in any language I know that will throw NotAString if you apply it to "". > However, from the practical point of view... This is very practical. Interoperability is about agreeing on a type system. A "type" defines a range of legal values. The AMQP type system includes empty string and 0-length binaries as legal values for those types. We absolutely cannot treat any legal value in an exceptional way unless that is clearly mandated by the spec. > [...] > > Quote me the spec, this is a mater of law not opinion ;) > > I suspect that the sending of an empty string for a multi-frame > message > is entirely unintentional on the part of Apollo. I suspect it is a > bug > in Apollo or in the proton-j version/path it uses. That should be > confirmed and an appropriate JIRA raised That's fine for Apollo but irrelevant for proton. The first law of interoperability is "be strict with what you send, be forgiving with what you receive." To me that means that we should never *send* an empty delivery tag, but we should accept one unless the spec clearly states that it is illegal for anyone to ever send one. I see no such clear statement. . > > Proton-c should also not crash on receiving an empty (or null) > delivery > id. Beyond that I'm not overly concerned how it handles the empty > string > case.
Re: Proton-c Null Messages
On 06/11/2015 01:11 PM, aconway wrote: I disagree. An empty string is a perfectly legal value for a string. If the spec wants to assign special meaning to particular values of a property that needs to be stated. Of course, like you, I personally would not use an empty string as an identifier but as an implementor of an inter-operable spec I think we have to take the large view: *any* legal value of a parameter has to be considered equal unless the spec clearly states otherwise. If a name field is populated with an empty string, that to me is the same as not supplying a name. An empty string is a legal encoding, but in my view it does not supply a value at all. (It is not like say 0 which may be the default but is clearly a value in its own right). However, from the practical point of view... [...] Quote me the spec, this is a mater of law not opinion ;) I suspect that the sending of an empty string for a multi-frame message is entirely unintentional on the part of Apollo. I suspect it is a bug in Apollo or in the proton-j version/path it uses. That should be confirmed and an appropriate JIRA raised. Proton-c should also not crash on receiving an empty (or null) delivery id. Beyond that I'm not overly concerned how it handles the empty string case.
Re: Proton-c Null Messages
On Wed, 2015-06-10 at 16:32 +0100, Gordon Sim wrote: > On 06/10/2015 04:01 PM, aconway wrote: > > On Tue, 2015-06-09 at 19:54 +0100, Gordon Sim wrote: > > > On 06/09/2015 06:40 PM, logty wrote: > > > > When I run the client I get: > > > > > > > > [0x5351db0]:0 <- @transfer(20) [handle=0, delivery-id=0, > > > > delivery > > > > -tag=b"", > > > > message-format=0, settled=true, more=true] (16363) > > > > "\x00Sp\xc0\x07\x05B..." > > > > > > My guess would be that it is the delivery tag being null (or > > > empty, > > > can't tell which) that is the problem. From the spec: > > > > > > "This field MUST be specified for the first transfer of > > > a multi-transfer message and can only be omitted for > > > continuation transfers." [section 2.7.5] > > > > > > So I think that whatever is sending that frame has a bug. Proton > > > -c > > > has a > > > bug too of course, since it shouldn't segfault but should close > > > the > > > connection with a framing-error or similar. > > > > It says the field must be specified, it does not say it must not be > > an > > empty binary value. Is the field really missing or is proton > > choking on > > a 0-length delivery tag? > > I'm not sure the distinction between null and an empty value is very > useful here. The intent is that the delivery is clearly identified. I > > would argue that a 'zero byte identifier' doesn't meet the spirit of > the > law there. I disagree. An empty string is a perfectly legal value for a string. If the spec wants to assign special meaning to particular values of a property that needs to be stated. Of course, like you, I personally would not use an empty string as an identifier but as an implementor of an inter-operable spec I think we have to take the large view: *any* legal value of a parameter has to be considered equal unless the spec clearly states otherwise. > > It shouldn't, which might explain why rabbit is OK with > > it. > > I don't think RabbitMQ is ever seeing that frame. I believe that > frame > is emitted by ApolloMQ to the receiving client. Rabbit, Apollo, whoever. If somebody is using the empty string as a delivery tag and the spec does not clearly state "you must never use the empty string as a delivery tag" then we should accept it. > I agree that proton should not choke on a zero byte delivery tag (or > indeed on a non-existent delivery tag). But I do think it's a bug to > send such a frame. Quote me the spec, this is a mater of law not opinion ;)
Re: Proton-c Null Messages
On 06/10/2015 07:26 PM, logty wrote: The odd thing is the sender is specifying the delivery-id, here is the output of PN_TRACE_FRM=1 on the sender side: [0x86b6580]:0 -> @transfer(20) [handle=0, delivery-id=0, delivery-tag=b"\x00\x00\x00\x00\x00\x00\x00\x00", message-format=0, settled=true, more=false] (5243441) "\x00Sp\xd0\x00\x00\x00\x0b\x00\x..." The delivery tage used by the broker when sending the message out to a consuming client is not (or need not) be the same as the one used by the publishing client. This is the output from one of the tests, but I can replicate it just using example programs and inputing large amounts of data, yet even with small amounts of data the delivery-tag is still not being set when sent to a reciever. Here is the sender: [0x17471b0]:0 -> @transfer(20) [handle=0, delivery-id=3, delivery-tag=b"\x03\x00\x00\x00\x00\x00\x00\x00", message-format=0, settled=false, more=false] (109) "\x00Sp\xd0\x00\x00\x00\x0b\x00\x00\x00\x05BP\x04@BR\x00\x00Ss\xd0\x00\x00\x00L\x00\x00\x00\x0d@@\xa1$amqp://127.0.0.1:61613/topic://event\xa1\x04Test\x83\x00\x00\x00\x00\x00\x00\x00\x00\x83\x00\x00\x00\x00\x00\x00\x00\x00@R\x00@\x00Sw\xa1\x01a" Here is the reciever: [0x1de3ac0]:0 <- @transfer(20) [handle=0, delivery-id=5, delivery-tag=b"0", This is ok, that is a 1 byte tage with the ascii value '0' I believe. However the original trace commented on had: delivery-tag=b"", which is a zero byte value. message-format=0] (122) "\x00Sp\xc0\x07\x05BP\x04@BC\x00Ss\xc0G\x0c@@\xa1$amqp://127.0.0.1:61613/topic://event\xa1\x04Test\x83\x00\x00\x00\x00\x00\x00\x00\x00\x83\x00\x00\x00\x00\x00\x00\x00\x00@C\x00Sw\xa1\x01a\x00Sx\xc1\x17\x02\xa3\x06origin\xa1\x0cmybroker-43c" -- View this message in context: http://qpid.2158936.n2.nabble.com/Proton-c-Null-Messages-tp7625967p7626224.html Sent from the Apache Qpid Proton mailing list archive at Nabble.com.
Re: Proton-c Null Messages
The odd thing is the sender is specifying the delivery-id, here is the output of PN_TRACE_FRM=1 on the sender side: [0x86b6580]:0 -> @transfer(20) [handle=0, delivery-id=0, delivery-tag=b"\x00\x00\x00\x00\x00\x00\x00\x00", message-format=0, settled=true, more=false] (5243441) "\x00Sp\xd0\x00\x00\x00\x0b\x00\x..." This is the output from one of the tests, but I can replicate it just using example programs and inputing large amounts of data, yet even with small amounts of data the delivery-tag is still not being set when sent to a reciever. Here is the sender: [0x17471b0]:0 -> @transfer(20) [handle=0, delivery-id=3, delivery-tag=b"\x03\x00\x00\x00\x00\x00\x00\x00", message-format=0, settled=false, more=false] (109) "\x00Sp\xd0\x00\x00\x00\x0b\x00\x00\x00\x05BP\x04@BR\x00\x00Ss\xd0\x00\x00\x00L\x00\x00\x00\x0d@@\xa1$amqp://127.0.0.1:61613/topic://event\xa1\x04Test\x83\x00\x00\x00\x00\x00\x00\x00\x00\x83\x00\x00\x00\x00\x00\x00\x00\x00@R\x00@\x00Sw\xa1\x01a" Here is the reciever: [0x1de3ac0]:0 <- @transfer(20) [handle=0, delivery-id=5, delivery-tag=b"0", message-format=0] (122) "\x00Sp\xc0\x07\x05BP\x04@BC\x00Ss\xc0G\x0c@@\xa1$amqp://127.0.0.1:61613/topic://event\xa1\x04Test\x83\x00\x00\x00\x00\x00\x00\x00\x00\x83\x00\x00\x00\x00\x00\x00\x00\x00@C\x00Sw\xa1\x01a\x00Sx\xc1\x17\x02\xa3\x06origin\xa1\x0cmybroker-43c" -- View this message in context: http://qpid.2158936.n2.nabble.com/Proton-c-Null-Messages-tp7625967p7626224.html Sent from the Apache Qpid Proton mailing list archive at Nabble.com.
Re: Proton-c Null Messages
On 06/10/2015 04:01 PM, aconway wrote: On Tue, 2015-06-09 at 19:54 +0100, Gordon Sim wrote: On 06/09/2015 06:40 PM, logty wrote: When I run the client I get: [0x5351db0]:0 <- @transfer(20) [handle=0, delivery-id=0, delivery -tag=b"", message-format=0, settled=true, more=true] (16363) "\x00Sp\xc0\x07\x05B..." My guess would be that it is the delivery tag being null (or empty, can't tell which) that is the problem. From the spec: "This field MUST be specified for the first transfer of a multi-transfer message and can only be omitted for continuation transfers." [section 2.7.5] So I think that whatever is sending that frame has a bug. Proton-c has a bug too of course, since it shouldn't segfault but should close the connection with a framing-error or similar. It says the field must be specified, it does not say it must not be an empty binary value. Is the field really missing or is proton choking on a 0-length delivery tag? I'm not sure the distinction between null and an empty value is very useful here. The intent is that the delivery is clearly identified. I would argue that a 'zero byte identifier' doesn't meet the spirit of the law there. It shouldn't, which might explain why rabbit is OK with it. I don't think RabbitMQ is ever seeing that frame. I believe that frame is emitted by ApolloMQ to the receiving client. I agree that proton should not choke on a zero byte delivery tag (or indeed on a non-existent delivery tag). But I do think it's a bug to send such a frame. (I should say of course that this is all still somewhat speculative, based only on a snippet of protocol trace and this thread. I haven't actually reproduced myself to verify the bad behavior in ApolloMQ or that the crash in proton is caused by the delivery tag value).
Re: Proton-c Null Messages
On Tue, 2015-06-09 at 19:54 +0100, Gordon Sim wrote: > On 06/09/2015 06:40 PM, logty wrote: > > When I run the client I get: > > > > [0x5351db0]:0 <- @transfer(20) [handle=0, delivery-id=0, delivery > > -tag=b"", > > message-format=0, settled=true, more=true] (16363) > > "\x00Sp\xc0\x07\x05B..." > > My guess would be that it is the delivery tag being null (or empty, > can't tell which) that is the problem. From the spec: > > "This field MUST be specified for the first transfer of > a multi-transfer message and can only be omitted for > continuation transfers." [section 2.7.5] > > So I think that whatever is sending that frame has a bug. Proton-c > has a > bug too of course, since it shouldn't segfault but should close the > connection with a framing-error or similar. It says the field must be specified, it does not say it must not be an empty binary value. Is the field really missing or is proton choking on a 0-length delivery tag? It shouldn't, which might explain why rabbit is OK with it. > > And then the segfault occurs when transfering a 5 MB message, and > > it is only > > coming through as this 16 KB message. >
Re: Proton-c Null Messages
On 06/10/2015 01:26 PM, dylan25 wrote: We're using Apache Apollo. Is there the possibility that Apollo has a server-side configuration setting that limits message sizes? Even if it did, that wouldn't explain why it would send out a frame with no delivery tag. It sounds like it may be a bug in Apollo to me (or perhaps in the proton-j version used by Apollo).
Re: Proton-c Null Messages
Ah, okay, you are right, I checked out the source code and it does do that automatically. I also ran PN_TRACE_FRM=1 on the sender and found that the delivery-tag is set correctly on the way out, yet it is empty on the other side. I am using proton 0.9.1, and using an Apache Apollo broker. The messaging works fine using a RabbitMQ broker. -- View this message in context: http://qpid.2158936.n2.nabble.com/Proton-c-Null-Messages-tp7625967p7626164.html Sent from the Apache Qpid Proton mailing list archive at Nabble.com.
Re: Proton-c Null Messages
We're using Apache Apollo. Is there the possibility that Apollo has a server-side configuration setting that limits message sizes? -- View this message in context: http://qpid.2158936.n2.nabble.com/Proton-c-Null-Messages-tp7625967p7626159.html Sent from the Apache Qpid Proton mailing list archive at Nabble.com.
Re: Proton-c Null Messages
On 06/09/2015 09:36 PM, logty wrote: Can you give an example of how I would set the delivery tag? You don't set the tag when using messenger, that should be done for you. What version of proton are you using? Are you using a broker or similar? Or just sending direct between two processes using proton-c?
Re: Proton-c Null Messages
Can you give an example of how I would set the delivery tag? I have been using a messenger to send the messages, but I have been having a tough time finding the documentation on how to set the tag. -- View this message in context: http://qpid.2158936.n2.nabble.com/Proton-c-Null-Messages-tp7625967p7626141.html Sent from the Apache Qpid Proton mailing list archive at Nabble.com.
Re: Proton-c Null Messages
On 06/09/2015 06:40 PM, logty wrote: When I run the client I get: [0x5351db0]:0 <- @transfer(20) [handle=0, delivery-id=0, delivery-tag=b"", message-format=0, settled=true, more=true] (16363) "\x00Sp\xc0\x07\x05B..." My guess would be that it is the delivery tag being null (or empty, can't tell which) that is the problem. From the spec: "This field MUST be specified for the first transfer of a multi-transfer message and can only be omitted for continuation transfers." [section 2.7.5] So I think that whatever is sending that frame has a bug. Proton-c has a bug too of course, since it shouldn't segfault but should close the connection with a framing-error or similar. And then the segfault occurs when transfering a 5 MB message, and it is only coming through as this 16 KB message.
Re: Proton-c Null Messages
When I run the client I get: [0x5351db0]:0 <- @transfer(20) [handle=0, delivery-id=0, delivery-tag=b"", message-format=0, settled=true, more=true] (16363) "\x00Sp\xc0\x07\x05B..." And then the segfault occurs when transfering a 5 MB message, and it is only coming through as this 16 KB message. -- View this message in context: http://qpid.2158936.n2.nabble.com/Proton-c-Null-Messages-tp7625967p7626133.html Sent from the Apache Qpid Proton mailing list archive at Nabble.com.
Re: Proton-c Null Messages
On Mon, Jun 08, 2015 at 01:47:03PM -0700, logty wrote: > The strange thing is pn_messenger_get is returning 0, I have code to throw an > error if it isn't 0. Also, the large messages work fine on RabbitMQ but not > on Apache Apollo. What's the output when you run your client with: $ PN_TRACE_FRM=1 [your client cmdline] ? Do you see the message being received on the client side? -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpxSErwfJzge.pgp Description: PGP signature
Re: Proton-c Null Messages
The strange thing is pn_messenger_get is returning 0, I have code to throw an error if it isn't 0. Also, the large messages work fine on RabbitMQ but not on Apache Apollo. -- View this message in context: http://qpid.2158936.n2.nabble.com/Proton-c-Null-Messages-tp7625967p7626075.html Sent from the Apache Qpid Proton mailing list archive at Nabble.com.
Re: Proton-c Null Messages
On Mon, Jun 08, 2015 at 05:56:14AM -0700, logty wrote: > Hi all, > > I just had a quick question about proton-c returning null messages. I am > calling pn_messenger_get on my message and on my messenger and getting back > messages that are null. When I call: > > pn_bytes_t b = pn_data_get_bytes(pn_message_body(message)); > > calls to b.start and b.size create segfaults. Any ideas as to why this might > be happening? At the moment I am running a test with large messages (several > megabytes) being sent to the server. > > Thanks for the help! Are you *certain* you're getting messages back? What's the return value on the call to pn_messenger_get? -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpq1f55F5Ztp.pgp Description: PGP signature
Re: Proton-c Null Messages
After some research it appears to be a problem with messages over a certain size aren't being correctly sent/recieved and are instead being picked up by the messenger as null, although I don't know how to fix this, any thoughts? -- View this message in context: http://qpid.2158936.n2.nabble.com/Proton-c-Null-Messages-tp7625967p7626057.html Sent from the Apache Qpid Proton mailing list archive at Nabble.com.