[jira] [Commented] (PROTON-784) Settled Deliveries keep piling up, and do not seem to get freed from heap
[
https://issues.apache.org/jira/browse/PROTON-784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14264501#comment-14264501
]
Rafael H. Schloming commented on PROTON-784:
Any chance you can come up with a minimal reproducer in terms of the proton
API? I have no clue how messaging is using proton internally, so a minimal
reproducer in terms of the messaging API doesn't tell me a lot.
Settled Deliveries keep piling up, and do not seem to get freed from heap
-
Key: PROTON-784
URL: https://issues.apache.org/jira/browse/PROTON-784
Project: Qpid Proton
Issue Type: Bug
Components: proton-c
Affects Versions: 0.9
Reporter: German Shepherd (PrE)
Fix For: 0.9
When running the ProtonC 0.8, this issue was not present.
Then, in ProtonC 0.9 (Git trunk of 12/19/2014), there are changes in a way
the deliveries are handled (engine.c).
Correct me if I'm wrong, but given the behavior in 0.8, and keeping the same
application side approach to ProtonC, while running 0.9, I would expect the
Deliveries to be eventually freed from heap. Or is there a new plan on how to
approach this ?
I'm calling pn_messenger_settle with PN_CUMMULATIVE for both input and output
trackers, and the delivery-local.settled gets set to true (verified).
Here is the dump of Deliveries, after there is one message received:
{{tag=, local.type=36, remote.type=0, local.settled=1, remote.settled=1,
updated=0, current=0, writable=0, readable=0, work=0}}
Then my application code sends a message to broker. The previous, now fully
settled Delivery, is not freed - here is the dump (there are now two
Deliveries on heap):
{{tag=, local.type=36, remote.type=0, local.settled=1, remote.settled=1,
updated=0, current=0, writable=0, readable=0, work=0}}
{{tag=\x00\x00\x00\x00\x00\x00\x00\x00, local.type=36, remote.type=36,
local.settled=1, remote.settled=1, updated=0, current=0, writable=0,
readable=0, work=0}}
Continuing, by received one more message and also sending one to broker,
there are now four Deliveries on the heap.
{{tag=, local.type=36, remote.type=0, local.settled=1, remote.settled=1,
updated=0, current=0, writable=0, readable=0, work=0}}
{{tag=\x00\x00\x00\x00\x00\x00\x00\x00, local.type=36, remote.type=36,
local.settled=1, remote.settled=1, updated=0, current=0, writable=0,
readable=0, work=0}}
{{tag=, local.type=36, remote.type=0, local.settled=1, remote.settled=1,
updated=0, current=0, writable=0, readable=0, work=0}}
{{tag=\x01\x00\x00\x00\x00\x00\x00\x00, local.type=36, remote.type=36,
local.settled=1, remote.settled=1, updated=0, current=0, writable=0,
readable=0, work=0}}
And so on, until it eventually runs out of heap.
I have done an extensive debug session, looking at the refcounts, behavior
inside the pn_delivery_finalize(), places this gets called from, etc.
One thing that hits the eye is the missing pn_full_settle() function, which
is now replaced somehow differently by other code. Also, the tpwork-ers used
to decref the Disposions, this is also modified for 0.9 (compared to 0.8).
I'm not sure (yet) what is going on in there, it also might just be some
error of mine (so I'm not immediately describing this as a bug, and I might
provide more information as the situation unravels).
At this moment, I would appreciate some hints on this topic, and also a
review of the particular functionality (dispositon of Deliveries) in ProtonC
0.9.
Thanks a lot !
(minor edit - text formatting - on 12/21/2014)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PROTON-784) Settled Deliveries keep piling up, and do not seem to get freed from heap
[
https://issues.apache.org/jira/browse/PROTON-784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14264322#comment-14264322
]
German Shepherd (PrE) commented on PROTON-784:
--
I'm back at work. And - so I got to compile the 0.9 with the Rafael's patch
(BTW. thanks for that !):
it is stable and it does work well, the deliveries are removed from heap (or
put onto the list for reuse) now.
Nice.
I will let it work over night, sending and receiving messages, to see the
longer-term outcome.
Note. I'm testing the plain ProtonC (the Messenger), not the Qpid additions to
it,
so I can't comment on whether it fixed the situation Chuck has reported.
Settled Deliveries keep piling up, and do not seem to get freed from heap
-
Key: PROTON-784
URL: https://issues.apache.org/jira/browse/PROTON-784
Project: Qpid Proton
Issue Type: Bug
Components: proton-c
Affects Versions: 0.9
Reporter: German Shepherd (PrE)
Fix For: 0.9
When running the ProtonC 0.8, this issue was not present.
Then, in ProtonC 0.9 (Git trunk of 12/19/2014), there are changes in a way
the deliveries are handled (engine.c).
Correct me if I'm wrong, but given the behavior in 0.8, and keeping the same
application side approach to ProtonC, while running 0.9, I would expect the
Deliveries to be eventually freed from heap. Or is there a new plan on how to
approach this ?
I'm calling pn_messenger_settle with PN_CUMMULATIVE for both input and output
trackers, and the delivery-local.settled gets set to true (verified).
Here is the dump of Deliveries, after there is one message received:
{{tag=, local.type=36, remote.type=0, local.settled=1, remote.settled=1,
updated=0, current=0, writable=0, readable=0, work=0}}
Then my application code sends a message to broker. The previous, now fully
settled Delivery, is not freed - here is the dump (there are now two
Deliveries on heap):
{{tag=, local.type=36, remote.type=0, local.settled=1, remote.settled=1,
updated=0, current=0, writable=0, readable=0, work=0}}
{{tag=\x00\x00\x00\x00\x00\x00\x00\x00, local.type=36, remote.type=36,
local.settled=1, remote.settled=1, updated=0, current=0, writable=0,
readable=0, work=0}}
Continuing, by received one more message and also sending one to broker,
there are now four Deliveries on the heap.
{{tag=, local.type=36, remote.type=0, local.settled=1, remote.settled=1,
updated=0, current=0, writable=0, readable=0, work=0}}
{{tag=\x00\x00\x00\x00\x00\x00\x00\x00, local.type=36, remote.type=36,
local.settled=1, remote.settled=1, updated=0, current=0, writable=0,
readable=0, work=0}}
{{tag=, local.type=36, remote.type=0, local.settled=1, remote.settled=1,
updated=0, current=0, writable=0, readable=0, work=0}}
{{tag=\x01\x00\x00\x00\x00\x00\x00\x00, local.type=36, remote.type=36,
local.settled=1, remote.settled=1, updated=0, current=0, writable=0,
readable=0, work=0}}
And so on, until it eventually runs out of heap.
I have done an extensive debug session, looking at the refcounts, behavior
inside the pn_delivery_finalize(), places this gets called from, etc.
One thing that hits the eye is the missing pn_full_settle() function, which
is now replaced somehow differently by other code. Also, the tpwork-ers used
to decref the Disposions, this is also modified for 0.9 (compared to 0.8).
I'm not sure (yet) what is going on in there, it also might just be some
error of mine (so I'm not immediately describing this as a bug, and I might
provide more information as the situation unravels).
At this moment, I would appreciate some hints on this topic, and also a
review of the particular functionality (dispositon of Deliveries) in ProtonC
0.9.
Thanks a lot !
(minor edit - text formatting - on 12/21/2014)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PROTON-784) Settled Deliveries keep piling up, and do not seem to get freed from heap
[
https://issues.apache.org/jira/browse/PROTON-784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265176#comment-14265176
]
Rafael H. Schloming commented on PROTON-784:
I'm a bit confused as to how the fix could be platform specific. Are you sure
the issue was there on linux prior to my fix?
Settled Deliveries keep piling up, and do not seem to get freed from heap
-
Key: PROTON-784
URL: https://issues.apache.org/jira/browse/PROTON-784
Project: Qpid Proton
Issue Type: Bug
Components: proton-c
Affects Versions: 0.9
Reporter: German Shepherd (PrE)
Fix For: 0.9
When running the ProtonC 0.8, this issue was not present.
Then, in ProtonC 0.9 (Git trunk of 12/19/2014), there are changes in a way
the deliveries are handled (engine.c).
Correct me if I'm wrong, but given the behavior in 0.8, and keeping the same
application side approach to ProtonC, while running 0.9, I would expect the
Deliveries to be eventually freed from heap. Or is there a new plan on how to
approach this ?
I'm calling pn_messenger_settle with PN_CUMMULATIVE for both input and output
trackers, and the delivery-local.settled gets set to true (verified).
Here is the dump of Deliveries, after there is one message received:
{{tag=, local.type=36, remote.type=0, local.settled=1, remote.settled=1,
updated=0, current=0, writable=0, readable=0, work=0}}
Then my application code sends a message to broker. The previous, now fully
settled Delivery, is not freed - here is the dump (there are now two
Deliveries on heap):
{{tag=, local.type=36, remote.type=0, local.settled=1, remote.settled=1,
updated=0, current=0, writable=0, readable=0, work=0}}
{{tag=\x00\x00\x00\x00\x00\x00\x00\x00, local.type=36, remote.type=36,
local.settled=1, remote.settled=1, updated=0, current=0, writable=0,
readable=0, work=0}}
Continuing, by received one more message and also sending one to broker,
there are now four Deliveries on the heap.
{{tag=, local.type=36, remote.type=0, local.settled=1, remote.settled=1,
updated=0, current=0, writable=0, readable=0, work=0}}
{{tag=\x00\x00\x00\x00\x00\x00\x00\x00, local.type=36, remote.type=36,
local.settled=1, remote.settled=1, updated=0, current=0, writable=0,
readable=0, work=0}}
{{tag=, local.type=36, remote.type=0, local.settled=1, remote.settled=1,
updated=0, current=0, writable=0, readable=0, work=0}}
{{tag=\x01\x00\x00\x00\x00\x00\x00\x00, local.type=36, remote.type=36,
local.settled=1, remote.settled=1, updated=0, current=0, writable=0,
readable=0, work=0}}
And so on, until it eventually runs out of heap.
I have done an extensive debug session, looking at the refcounts, behavior
inside the pn_delivery_finalize(), places this gets called from, etc.
One thing that hits the eye is the missing pn_full_settle() function, which
is now replaced somehow differently by other code. Also, the tpwork-ers used
to decref the Disposions, this is also modified for 0.9 (compared to 0.8).
I'm not sure (yet) what is going on in there, it also might just be some
error of mine (so I'm not immediately describing this as a bug, and I might
provide more information as the situation unravels).
At this moment, I would appreciate some hints on this topic, and also a
review of the particular functionality (dispositon of Deliveries) in ProtonC
0.9.
Thanks a lot !
(minor edit - text formatting - on 12/21/2014)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PROTON-784) Settled Deliveries keep piling up, and do not seem to get freed from heap
[
https://issues.apache.org/jira/browse/PROTON-784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265271#comment-14265271
]
Chuck Rolke commented on PROTON-784:
I'm sure there is an issue. It may not be this issue.
Settled Deliveries keep piling up, and do not seem to get freed from heap
-
Key: PROTON-784
URL: https://issues.apache.org/jira/browse/PROTON-784
Project: Qpid Proton
Issue Type: Bug
Components: proton-c
Affects Versions: 0.9
Reporter: German Shepherd (PrE)
Fix For: 0.9
When running the ProtonC 0.8, this issue was not present.
Then, in ProtonC 0.9 (Git trunk of 12/19/2014), there are changes in a way
the deliveries are handled (engine.c).
Correct me if I'm wrong, but given the behavior in 0.8, and keeping the same
application side approach to ProtonC, while running 0.9, I would expect the
Deliveries to be eventually freed from heap. Or is there a new plan on how to
approach this ?
I'm calling pn_messenger_settle with PN_CUMMULATIVE for both input and output
trackers, and the delivery-local.settled gets set to true (verified).
Here is the dump of Deliveries, after there is one message received:
{{tag=, local.type=36, remote.type=0, local.settled=1, remote.settled=1,
updated=0, current=0, writable=0, readable=0, work=0}}
Then my application code sends a message to broker. The previous, now fully
settled Delivery, is not freed - here is the dump (there are now two
Deliveries on heap):
{{tag=, local.type=36, remote.type=0, local.settled=1, remote.settled=1,
updated=0, current=0, writable=0, readable=0, work=0}}
{{tag=\x00\x00\x00\x00\x00\x00\x00\x00, local.type=36, remote.type=36,
local.settled=1, remote.settled=1, updated=0, current=0, writable=0,
readable=0, work=0}}
Continuing, by received one more message and also sending one to broker,
there are now four Deliveries on the heap.
{{tag=, local.type=36, remote.type=0, local.settled=1, remote.settled=1,
updated=0, current=0, writable=0, readable=0, work=0}}
{{tag=\x00\x00\x00\x00\x00\x00\x00\x00, local.type=36, remote.type=36,
local.settled=1, remote.settled=1, updated=0, current=0, writable=0,
readable=0, work=0}}
{{tag=, local.type=36, remote.type=0, local.settled=1, remote.settled=1,
updated=0, current=0, writable=0, readable=0, work=0}}
{{tag=\x01\x00\x00\x00\x00\x00\x00\x00, local.type=36, remote.type=36,
local.settled=1, remote.settled=1, updated=0, current=0, writable=0,
readable=0, work=0}}
And so on, until it eventually runs out of heap.
I have done an extensive debug session, looking at the refcounts, behavior
inside the pn_delivery_finalize(), places this gets called from, etc.
One thing that hits the eye is the missing pn_full_settle() function, which
is now replaced somehow differently by other code. Also, the tpwork-ers used
to decref the Disposions, this is also modified for 0.9 (compared to 0.8).
I'm not sure (yet) what is going on in there, it also might just be some
error of mine (so I'm not immediately describing this as a bug, and I might
provide more information as the situation unravels).
At this moment, I would appreciate some hints on this topic, and also a
review of the particular functionality (dispositon of Deliveries) in ProtonC
0.9.
Thanks a lot !
(minor edit - text formatting - on 12/21/2014)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (PROTON-784) Settled Deliveries keep piling up, and do not seem to get freed from heap
[
https://issues.apache.org/jira/browse/PROTON-784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265090#comment-14265090
]
Chuck Rolke commented on PROTON-784:
On Linux this issue appears resolved. On Windows it's still an issue and I'm
investigating further from my end.
-Cheers
Settled Deliveries keep piling up, and do not seem to get freed from heap
-
Key: PROTON-784
URL: https://issues.apache.org/jira/browse/PROTON-784
Project: Qpid Proton
Issue Type: Bug
Components: proton-c
Affects Versions: 0.9
Reporter: German Shepherd (PrE)
Fix For: 0.9
When running the ProtonC 0.8, this issue was not present.
Then, in ProtonC 0.9 (Git trunk of 12/19/2014), there are changes in a way
the deliveries are handled (engine.c).
Correct me if I'm wrong, but given the behavior in 0.8, and keeping the same
application side approach to ProtonC, while running 0.9, I would expect the
Deliveries to be eventually freed from heap. Or is there a new plan on how to
approach this ?
I'm calling pn_messenger_settle with PN_CUMMULATIVE for both input and output
trackers, and the delivery-local.settled gets set to true (verified).
Here is the dump of Deliveries, after there is one message received:
{{tag=, local.type=36, remote.type=0, local.settled=1, remote.settled=1,
updated=0, current=0, writable=0, readable=0, work=0}}
Then my application code sends a message to broker. The previous, now fully
settled Delivery, is not freed - here is the dump (there are now two
Deliveries on heap):
{{tag=, local.type=36, remote.type=0, local.settled=1, remote.settled=1,
updated=0, current=0, writable=0, readable=0, work=0}}
{{tag=\x00\x00\x00\x00\x00\x00\x00\x00, local.type=36, remote.type=36,
local.settled=1, remote.settled=1, updated=0, current=0, writable=0,
readable=0, work=0}}
Continuing, by received one more message and also sending one to broker,
there are now four Deliveries on the heap.
{{tag=, local.type=36, remote.type=0, local.settled=1, remote.settled=1,
updated=0, current=0, writable=0, readable=0, work=0}}
{{tag=\x00\x00\x00\x00\x00\x00\x00\x00, local.type=36, remote.type=36,
local.settled=1, remote.settled=1, updated=0, current=0, writable=0,
readable=0, work=0}}
{{tag=, local.type=36, remote.type=0, local.settled=1, remote.settled=1,
updated=0, current=0, writable=0, readable=0, work=0}}
{{tag=\x01\x00\x00\x00\x00\x00\x00\x00, local.type=36, remote.type=36,
local.settled=1, remote.settled=1, updated=0, current=0, writable=0,
readable=0, work=0}}
And so on, until it eventually runs out of heap.
I have done an extensive debug session, looking at the refcounts, behavior
inside the pn_delivery_finalize(), places this gets called from, etc.
One thing that hits the eye is the missing pn_full_settle() function, which
is now replaced somehow differently by other code. Also, the tpwork-ers used
to decref the Disposions, this is also modified for 0.9 (compared to 0.8).
I'm not sure (yet) what is going on in there, it also might just be some
error of mine (so I'm not immediately describing this as a bug, and I might
provide more information as the situation unravels).
At this moment, I would appreciate some hints on this topic, and also a
review of the particular functionality (dispositon of Deliveries) in ProtonC
0.9.
Thanks a lot !
(minor edit - text formatting - on 12/21/2014)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (PROTON-791) c-reactor-tests fails because it listens to an IPv4 socket but connects to an IPv6 socket
Andrew Stitcher created PROTON-791:
--
Summary: c-reactor-tests fails because it listens to an IPv4
socket but connects to an IPv6 socket
Key: PROTON-791
URL: https://issues.apache.org/jira/browse/PROTON-791
Project: Qpid Proton
Issue Type: Bug
Components: proton-c
Reporter: Andrew Stitcher
Assignee: Rafael H. Schloming
strace of relevant bits of failing run.
{noformat}
...
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
bind(3, {sa_family=AF_INET, sin_port=htons(5672),
sin_addr=inet_addr(0.0.0.0)}, 16) = 0
listen(3, 50) = 0
...
socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 4
fcntl(4, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(4, F_SETFL, O_RDWR|O_NONBLOCK)= 0
setsockopt(4, SOL_TCP, TCP_NODELAY, [1], 4) = 0
connect(4, {sa_family=AF_INET6, sin6_port=htons(5672), inet_pton(AF_INET6,
::1, sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 EINPROGRESS
(Operation now in progress)
poll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN|POLLOUT}], 2, 1000) = 1
([{fd=4, revents=POLLIN|POLLOUT|POLLERR|POLLHUP}])
recvfrom(4, 0x1b19200, 16384, 0, 0, 0) = -1 ECONNREFUSED (Connection refused)
...
sendto(4, AMQP\3\1\0\0\0\0\0!\2\1\0\0\0SA\320\0\0\0\21\0\0\0\2\243\tAN...,
224, MSG_NOSIGNAL, NULL, 0) = -1 EPIPE (Broken pipe)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
