Re: proton cpp container.stop not fully thread safe
I can reproduce and it looks more like a proactor problem than the C++ container.stop(). I will investigate further and create an appropriate JIRA. Thank you for the simple test case. Cliff On Wed, Apr 3, 2019 at 4:17 AM Rabih M wrote: > > Hello, > > Concerning this point, i still have the same crash even after modifying the > code to be thread safe. > I attached the corrected code. > > If i add a sleep of 50 ms before stopping the container everything works > correctly. > > I think there is a concurrency problem somewhere in the proton code. Do you > confirm? > > Best regards, > Rabih > > On Wed, Mar 27, 2019 at 5:31 PM Rabih M wrote: >> >> Hello, >> >> I am using proton cpp version 0.27.0. >> I have a random crash when calling stop on the container after closing the >> connection. >> Please find the attached the code and the stack. >> >> In the doc, it is said that the container object is thread safe but >> apparently there is flaw... >> >> Can you please advise? >> >> Do you need me to create a Jira issue? >> >> >> Best regards, >> >> Rabih > > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: python proton reconnect broken on master?
On 03/04/2019 11:08 pm, Andrew Stitcher wrote: On Wed, 2019-04-03 at 22:58 +0100, Gordon Sim wrote: ... I think the issue is that the loss of the socket no longer results in a transport_close event? (or that the socket.error is raised and causes the loop to exit before processing that event?) Hmm, I think that in the case of a connect call failing there never was a transport to generate the transport_close event in the first place - the python code whch fails the connect doesn't generate that event. It does with 0.26.0. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: python proton reconnect broken on master?
On 03/04/2019 11:05 pm, Andrew Stitcher wrote: On Wed, 2019-04-03 at 22:58 +0100, Gordon Sim wrote: On 03/04/2019 10:50 pm, Andrew Stitcher wrote: On Wed, 2019-04-03 at 22:02 +0100, Gordon Sim wrote: It looks like the reconnect capability in the python client is broken on master. Is this a known issue? Any chance of some more details or a reproducer? Just run any of the examples either against a port with no broker, or kill the broker after client has connected. On 0.26.0 the client will keep trying to reconnect. On master it exits immediately with a socket.error. Really? the previous behaviour was to retry without bound? I'm not surprised I didn't replicate that behaviour - for the simplest examples it seems wrong. Certainly not what I would expect. It was debated quite a bit many years ago. It has been that way for a long, long time. As far as I can tell it doesn't work at all anymore, even if explicitly requested i.e. it is not simply a change in the default behaviour. How was the retry previously achieved? It handles the transport closed event. There is a Connector class in _reactor.py that controls it. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: python proton reconnect broken on master?
On Wed, 2019-04-03 at 22:58 +0100, Gordon Sim wrote: > ... > I think the issue is that the loss of the socket no longer results in > a > transport_close event? (or that the socket.error is raised and > causes > the loop to exit before processing that event?) Hmm, I think that in the case of a connect call failing there never was a transport to generate the transport_close event in the first place - the python code whch fails the connect doesn't generate that event. A - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: python proton reconnect broken on master?
On Wed, 2019-04-03 at 22:58 +0100, Gordon Sim wrote: > On 03/04/2019 10:50 pm, Andrew Stitcher wrote: > > On Wed, 2019-04-03 at 22:02 +0100, Gordon Sim wrote: > > > It looks like the reconnect capability in the python client is > > > broken > > > on > > > master. Is this a known issue? > > > > Any chance of some more details or a reproducer? > > Just run any of the examples either against a port with no broker, > or > kill the broker after client has connected. On 0.26.0 the client > will > keep trying to reconnect. On master it exits immediately with a > socket.error. Really? the previous behaviour was to retry without bound? I'm not surprised I didn't replicate that behaviour - for the simplest examples it seems wrong. Certainly not what I would expect. How was the retry previously achieved? A - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: Qpid proton cpp public API
On Wed, 2019-04-03 at 15:19 +0200, Rabih M wrote: > Hello, > > I am using the latest version of proton cpp 0.27.0. > > I have a suggestion for the signature of the messaging handler's > methods: > In my opinion, giving the parameters as references is error prone. > Because > it gives the user the illusion that he can keep a reference on the > proton > object in his handler and that proton is managing the life cycle of > this > object which is not the case. In other words, proton is giving a > reference > to a temporary variable. > > I understand that you would like to optimise the copy but now with > C++ 11 > we can move the objects. > > What do you think? I think you are mistaken! Passing a reference - merely signals that there will always be a value here never a null - It doesn't (in my experience) signal anything about the lifetime of that reference. Usually things passed into a call are only guaranteed to last for the duration of the call itself! In any event you can't keep a reference in any case irrespective of the passing convention. It seems you think that the following should be safe: class Foo { proton::connection* connection; ... void on_connection_blah(connection& c) { connection = } } Huh? However the following is safe the lifetime of connection will be extended until the instance of Foo itself is destroyed - if not there is a bug: class Foo { proton::connection connection; ... void on_connection_blah(connection& c) { connection = c; } } There would be no advantage to making the signatures void on_connection_blah(connection) which would force a copy for every callback. And the signature void on_connection_blah(connection&&) is just wrong as we are not passing the ownership of the connection to the callback. Andrew - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: python proton reconnect broken on master?
On 03/04/2019 10:50 pm, Andrew Stitcher wrote: On Wed, 2019-04-03 at 22:02 +0100, Gordon Sim wrote: It looks like the reconnect capability in the python client is broken on master. Is this a known issue? Any chance of some more details or a reproducer? Just run any of the examples either against a port with no broker, or kill the broker after client has connected. On 0.26.0 the client will keep trying to reconnect. On master it exits immediately with a socket.error. The current python bindings *should* work the same (by design) as the previous implementation. I think the issue is that the loss of the socket no longer results in a transport_close event? (or that the socket.error is raised and causes the loop to exit before processing that event?) - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: python proton reconnect broken on master?
On Wed, 2019-04-03 at 22:02 +0100, Gordon Sim wrote: > It looks like the reconnect capability in the python client is broken > on > master. Is this a known issue? Any chance of some more details or a reproducer? The current python bindings *should* work the same (by design) as the previous implementation. Andrew - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
python proton reconnect broken on master?
It looks like the reconnect capability in the python client is broken on master. Is this a known issue? - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: [VOTE] Release Apache Qpid JMS 0.41.0
On 4/3/19 1:17 PM, Robbie Gemmell wrote: Hi folks, I have put together a spin for a 0.41.0 Qpid JMS client release, please give it a test out and vote accordingly. The source and binary archives can be grabbed from: https://dist.apache.org/repos/dist/dev/qpid/jms/0.41.0-rc1/ The maven artifacts are also staged for now at: https://repository.apache.org/content/repositories/orgapacheqpid-1171 The JIRAs assigned are: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12314524=12344648 Regards, Robbie P.S. If you want to test it out using maven (e.g with the examples src, or your own things), you can temporarily add this to your poms to access the staging repo: staging https://repository.apache.org/content/repositories/orgapacheqpid-1171 The dependency for the client itself would then be: org.apache.qpid qpid-jms-client 0.41.0 - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org +1 * Validated signatures and checksums * Verified archives contained license and notice files * Checked source files for license headers using 'mvn apache-rat:check' * Built from source and ran all tests. * Built ActiveMQ 5 master using staged bits and ran all AMQP tests * Built ActiveMQ Artemis master using staged bits and ran the AMQP integration tests. -- Tim Bish - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: [VOTE] Release Qpid Broker-J 7.1.2
+1 built and ran tests (0-9-1 and 1.0) ad-hoc testing via the web console On Wed, 3 Apr 2019 at 16:15, Robbie Gemmell wrote: > On Sun, 31 Mar 2019 at 14:36, Oleksandr Rudyy wrote: > > > > Hi, > > > > I built release artefacts for Qpid Broker-J version 7.1.2 RC1. > > Please, give them a test out and vote accordingly. > > > > The source and binary archives can be found at: > > https://dist.apache.org/repos/dist/dev/qpid/broker-j/7.1.2-rc1/ > > > > The maven artifacts are also staged at: > > https://repository.apache.org/content/repositories/orgapacheqpid-1170 > > > > The new version incorporates improvement for priority queues allowing to > > change message priorities. You can find the full list of JIRAs included > > into the release here: > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310520=12345042 > > > > Kind Regards, > > Alex > > > > P.S. For testing of maven broker staging repo artefacts, please add into > to > > your project pom the staging repo as below: > > > > > > > > staging > > > > https://repository.apache.org/content/repositories/orgapacheqpid-1170 > > > > > > > > > +1 > > I checked things over as follows: > - Verified all the signature and checksum files. > - Ran "mvn apache-rat:check" to verify headers in the source archive. > - Checked for LICENCE and NOTICE files being present in the archives. > - Ran the build and tests with "mvn clean verify -DskipITs=false". > - Started broker from binary archive, used the web console to create a > queue. > - Ran the Qpid JMS master HelloWorld example against the broker. > > Robbie > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > >
[VOTE] Release Apache Qpid JMS 0.41.0
Hi folks, I have put together a spin for a 0.41.0 Qpid JMS client release, please give it a test out and vote accordingly. The source and binary archives can be grabbed from: https://dist.apache.org/repos/dist/dev/qpid/jms/0.41.0-rc1/ The maven artifacts are also staged for now at: https://repository.apache.org/content/repositories/orgapacheqpid-1171 The JIRAs assigned are: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12314524=12344648 Regards, Robbie P.S. If you want to test it out using maven (e.g with the examples src, or your own things), you can temporarily add this to your poms to access the staging repo: staging https://repository.apache.org/content/repositories/orgapacheqpid-1171 The dependency for the client itself would then be: org.apache.qpid qpid-jms-client 0.41.0 - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: [VOTE] Release Qpid Broker-J 7.1.2
On Sun, 31 Mar 2019 at 14:36, Oleksandr Rudyy wrote: > > Hi, > > I built release artefacts for Qpid Broker-J version 7.1.2 RC1. > Please, give them a test out and vote accordingly. > > The source and binary archives can be found at: > https://dist.apache.org/repos/dist/dev/qpid/broker-j/7.1.2-rc1/ > > The maven artifacts are also staged at: > https://repository.apache.org/content/repositories/orgapacheqpid-1170 > > The new version incorporates improvement for priority queues allowing to > change message priorities. You can find the full list of JIRAs included > into the release here: > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310520=12345042 > > Kind Regards, > Alex > > P.S. For testing of maven broker staging repo artefacts, please add into to > your project pom the staging repo as below: > > > > staging > > https://repository.apache.org/content/repositories/orgapacheqpid-1170 > > > +1 I checked things over as follows: - Verified all the signature and checksum files. - Ran "mvn apache-rat:check" to verify headers in the source archive. - Checked for LICENCE and NOTICE files being present in the archives. - Ran the build and tests with "mvn clean verify -DskipITs=false". - Started broker from binary archive, used the web console to create a queue. - Ran the Qpid JMS master HelloWorld example against the broker. Robbie - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: [External] Re: Force (C++) broker to drop connection and/or reset session
On 01/04/2019 22:28, Gordon Sim wrote: On 29/03/2019 12:26 pm, Toralf Lund wrote: Another one related to issues I've mentioned in other recent posts: I'm doing some debugging related to undesired side effects in one of our applications after it gets a "Failed to connect (reconnect disabled)" error while sending to or receiving from the C++ broker. In then "manually" reopens the connection after a slight delay. As parts of the reconnect logic, I also tend to get "'session-busy: Session detached by peer'". This is possibly triggered by the connect itself, but it could also come from other operations done just after connecting. This would be a whole lot easier if I could provoke the errors into occurring, so as to speak. I guess the "failed to connect" error means the broker has dropped the connection, but is there a way I can force it to do that (without restarting)? And how about the "Session detached by peer"? It's not clear to me when exactly I get that, but I guess it's related to sessions that existed before the connection was closed. Which (according to information I got here earlier) Connection::open() might try to recreate, but perhaps the operation fails? Can I somehow manipulate the state so I can find out more about the behaviour? You can specify the name for the session when you create it. So one approach might be to connect your client (the one you want to test), then kill -STOP that client, then start a test client that just opens a session with the same name (e.g. see attached), then kill -CONT the client which should then failover (assuming heartbeats were on) and hit the session-busy error. Thanks. That helps quite I bit. The errors I get with this experiment are consistent with what I'm seeing on the real installation - which I guess suggests the "Session detached by peer" occurs because the session already exists and is attached to another active connection (as seen from the broker.) In the test case, it's quite definitely Connection::open() itself that raises the exception. Furthermore, it looks like it's quite all right to use the connection anyway, but the session has to be set up again from scratch. - Toralf - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: [External] Re: Force (C++) broker to drop connection and/or reset session
On 29/03/2019 15:50, Chuck Rolke wrote: An easy thing to try would be to turn on protocol logging. In qpidd.conf file or on the command line add --log-enable trace+:Protocol This would expose what led up to the session being rejected. The protocol logs will show if the broker is spontaneously closing the session or if the client(s) are violating AMQP rules in some way. Yeah, maybe I'll try that. I'm a bit worried about the amount of log data, though, seeing as the error doesn't occur that frequently. Dropping connections or resetting sessions from within the broker would be really hard. Testing has focused on having cruel clients who close connections and sessions at inopportune moments to test that the broker survives. OK. - Toralf -Chuck - Original Message - From: "Toralf Lund" To: users@qpid.apache.org Sent: Friday, March 29, 2019 8:26:21 AM Subject: Force (C++) broker to drop connection and/or reset session Another one related to issues I've mentioned in other recent posts: I'm doing some debugging related to undesired side effects in one of our applications after it gets a "Failed to connect (reconnect disabled)" error while sending to or receiving from the C++ broker. In then "manually" reopens the connection after a slight delay. As parts of the reconnect logic, I also tend to get "'session-busy: Session detached by peer'". This is possibly triggered by the connect itself, but it could also come from other operations done just after connecting. This would be a whole lot easier if I could provoke the errors into occurring, so as to speak. I guess the "failed to connect" error means the broker has dropped the connection, but is there a way I can force it to do that (without restarting)? And how about the "Session detached by peer"? It's not clear to me when exactly I get that, but I guess it's related to sessions that existed before the connection was closed. Which (according to information I got here earlier) Connection::open() might try to recreate, but perhaps the operation fails? Can I somehow manipulate the state so I can find out more about the behaviour? Thanks. - Toralf - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: proton cpp container.stop not fully thread safe
Hello, Concerning this point, i still have the same crash even after modifying the code to be thread safe. I attached the corrected code. If i add a sleep of 50 ms before stopping the container everything works correctly. I think there is a concurrency problem somewhere in the proton code. Do you confirm? Best regards, Rabih On Wed, Mar 27, 2019 at 5:31 PM Rabih M wrote: > Hello, > > I am using proton cpp version 0.27.0. > I have a random crash when calling stop on the container after closing the > connection. > Please find the attached the code and the stack. > > In the doc, it is said that the container object is thread safe but > apparently there is flaw... > > Can you please advise? > > Do you need me to create a Jira issue? > > > Best regards, > > Rabih > #include #include #include #include #include class handler : public proton::messaging_handler { public: handler(const std::string& u) : m_url(u) {} proton::connection m_con; std::promise m_startProm; std::promise m_closeProm; private: void on_connection_open(proton::connection& c)override{ m_con = c; m_startProm.set_value(); } void on_transport_close(proton::transport&)override { m_closeProm.set_value(); } std::string m_url; }; int main () { { try { std::string url("//127.0.0.1:5672"); handler h(url); proton::container c(h); c.auto_stop(false); std::thread clientThread([&]() { try { c.run(); } catch (const std::exception& e) { std::cerr << "Broker threw exception: " << e.what() << std::endl; }}); c.connect(url); h.m_startProm.get_future().get(); h.m_con.work_queue().add([&]() { h.m_con.close(); }); h.m_closeProm.get_future().get(); c.stop(); if (clientThread.joinable()) { clientThread.join(); } } catch (std::exception& e) { std::cerr << e.what() << std::endl; } return 0; } - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: Proton cpp crashes on pn_class_decref
Hello, I adapted the code like we said in the last meeting and it worked without crashes. So the rule that i deduce is that "we cannot copy any proton objet outside of the handler's class". I attached the corrected code for future reference. Thanks, Rabih On Wed, Mar 27, 2019 at 5:19 PM Rabih M wrote: > Hello, > > > > I am using proton cpp version 0.27.0. > > We found a random failure in our code in proton sometime ago. > > I was able to reproduce using simple code against a java broker. Please > find attached the code and the stack. > > The frequency is lower with the simplified code but we are still > encountering the crash before reaching 60 000 repetitions. > > > > Murex will be using Proton and we need to ensure we have no such failure > before using the code in production. > > > > Can you please advise? > > Do you need me to create a Jira issue? > > > > Best regards, > > Rabih > #include #include #include #include #include class handler : public proton::messaging_handler { public: handler(const std::string& u) : m_url(u) {} proton::connection m_con; std::promise m_startProm; std::promise m_closeProm; private: void on_connection_open(proton::connection& c)override{ m_con = c; m_startProm.set_value(); } void on_connection_close(proton::connection&)override{ m_closeProm.set_value(); } std::string m_url; }; int main () { { try { std::string url("//127.0.0.1:5672"); handler h(url); proton::container c(h); std::thread clientThread([&]() { try { c.run(); } catch (const std::exception& e) { std::cerr << "Broker threw exception: " << e.what() << std::endl; }}); c.connect(url); h.m_startProm.get_future().get(); h.m_con.work_queue().add([&]() { h.m_con.close(); }); h.m_closeProm.get_future().get(); if (clientThread.joinable()) { clientThread.join(); } } catch (std::exception& e) { std::cerr << e.what() << std::endl; } return 0; } - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: Re: "Session detached by peer" when trying to re-establish lost connection
On 03/04/2019 09:32, Gordon Sim wrote: On 03/04/2019 7:32 am, Toralf Lund wrote: The broker also has a message saying "Connection ... timed out: closing". This occurs 4-5 seconds after the session-busy error, according to the timestamps. So for some reason, the broker is deciding the connection is dead 4-5 seconds after the client has reconnected... what caused the client to reconnect? Was there a timed out notice there as well? The direct reason for the reconnect is that "connection::isOpen()" returns false on a connection that was successfully opened some time in the past. The connection status is checked periodically. The "reconnect" option is not set on the connection. Prior to getting a "false" isOpen(), an exception with message "Failed to connect (reconnect disabled)" is caught, typically while waiting for incoming messages. session::nextReceiver() is used for that purpose. The broker log does not seem to have any messages related to the original issue. - Toralf - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: [External] Re: "Session detached by peer" when trying to re-establish lost connection
On 03/04/2019 7:32 am, Toralf Lund wrote: The broker also has a message saying "Connection ... timed out: closing". This occurs 4-5 seconds after the session-busy error, according to the timestamps. So for some reason, the broker is deciding the connection is dead 4-5 seconds after the client has reconnected... what caused the client to reconnect? Was there a timed out notice there as well? - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: Re: "Session detached by peer" when trying to re-establish lost connection
On 01/04/2019 22:33, Gordon Sim wrote: On 01/04/2019 12:54 pm, Toralf Lund wrote: On 28/03/2019 15:56, Toralf Lund wrote: Hi, I have another question related to my recent post about trying to re-establish lost connections. Recall that my system is set up without automatic reconnect, but may do a "manual reconnect" via qpid::Messaging connection(url); ... if(!connection.isOpen()) { connection.open(); } and that this is sometimes triggered by an "internal disconnect" accompanied by an exception with message "Failed to connect (reconnect disabled)". Now, I just found that some of the reconnects are accompanied by a "Session detached by peer" exception. I'm struggling a bit to find out what exactly triggers this; the problems occur once every few hours at a remote site, and I can't just start or stop or update the software at any old time, so it's a bit hard to debug... I just noticed that the broker log has something like 2019-04-01 11:11:42 [Broker] error Channel exception: session-busy: Session already attached: anonymous.c9a1f4cf-7427-4e77-8870-e118b099c6da (/work/build/rpmbuild/BUILD/msat-qpid-cpp-1.36/msat-qpid-cpp-1.36.0/src/qpid/broker/SessionManager.cpp:55) 2019-04-01 11:11:46 [Protocol] error Connection qpid.10.31.193.215:5672-10.31.192.61:35954 timed out: closing for most or all the reconnects. The first time-stamp seems to correspond to the connection open and "Session detached by peer" exception. Perhaps this means the broker still thinks the old connection is open, even though the client considers it as lost? And maybe the attempt to re-establish/re-use the old session with the new connection fails as a result of that? Correct. Does the broker log the loss of the connection, e.g. a timed out notice? If so how does the time of that compare to the session-busy error and to the client detecting the failed connection? The broker also has a message saying "Connection ... timed out: closing". This occurs 4-5 seconds after the session-busy error, according to the timestamps. - Toralf - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org