[ISSUE][proton-j] session begin frame handler throws NPE
Hello Qpid developers!! We are using proton-j Reactor APIs to and are hitting a NullPointerException – which is a TODO in the proton-j codebase . Truly appreciate if someone can help complete the TODO. Link to the line: https://github.com/apache/qpid-proton-j/blob/master/proton-j/src/main/java/org/apache/qpid/proton/engine/impl/TransportImpl.java#L1160 Here’s the scenario when our customers are hitting this: · - configure the listener - to delay respond to session.begin on an AmqpConnection for ex: by 5 secs · - then client (or another peer) sends session.begin to that listener · - client doesn't receive any response even after 1 sec (as delay of 5 secs was configured) · - then the client decides to close the session · - then, after few sec's service(listener) responds back with session.begin for that closed session · · In this case TransportImpl.handleBegin (frameparser implementation) – bails out with NPE. · · This is truly not such an uncommon scenario! Many a times, when our Service nodes are busy (high CPU or have large request queue length) – our clients are experiencing this. Filed a JIRA for this: https://issues.apache.org/jira/browse/PROTON-1939 Thanks a lot for the help so far! Sreeram
Building Qpid broker 7.0.6 Java version with OpenJDK 1.8.0_162_x64 fails in BDB Link Store Plug-in tests
Hello Qpid developers, I am trying to build the Qpid broker version 7.0.6 Java version, and it is failing the BDB Link Store Plug-in tests. I am building on an Intel 64-bit system running Ubuntu Linux 16.04 with OpenJDK 1.8.0_162_x64. I have searched for the errors that I am seeing but can't find anything. We are transitioning to OpenJDK and so I would like to be able to use it to build and run the Qpid broker. Can you provide any advice on how to build this successfully? Thank you and regards, Brian O'Shea When I build with the Oracle JDK of the same version it builds successfully and all tests pass. Here is the error that I am getting: Running org.apache.qpid.server.protocol.v1_0.store.bdb.BDBLinkStoreTestTests run: 5, Failures: 0, Errors: 5, Skipped: 0, Time elapsed: 0.625 sec <<< FAILURE! - in org.apache.qpid.server.protocol.v1_0.store.bdb.BDBLinkStoreTesttestOpenAndLoad(org.apache.qpid.server.protocol.v1_0.store.bdb.BDBLinkStoreTest) Time elapsed: 0.331 sec <<< ERROR!java.lang.IllegalStateException: Could not access Zing management bean. Make sure -XX:+UseZingMXBeans was specified. at org.apache.qpid.server.protocol.v1_0.store.bdb.BDBLinkStoreTest.createLinkStore(BDBLinkStoreTest.java:68)Caused by: java.lang.ClassNotFoundException: com.azul.zing.management.ManagementFactory at org.apache.qpid.server.protocol.v1_0.store.bdb.BDBLinkStoreTest.createLinkStore(BDBLinkStoreTest.java:68)testSaveLink(org.apache.qpid.server.protocol.v1_0.store.bdb.BDBLinkStoreTest) Time elapsed: 0.006 sec <<< ERROR!java.lang.IllegalStateException: Could not access Zing management bean. Make sure -XX:+UseZingMXBeans was specified. at org.apache.qpid.server.protocol.v1_0.store.bdb.BDBLinkStoreTest.createLinkStore(BDBLinkStoreTest.java:68)Caused by: java.lang.ClassNotFoundException: com.azul.zing.management.ManagementFactory at org.apache.qpid.server.protocol.v1_0.store.bdb.BDBLinkStoreTest.createLinkStore(BDBLinkStoreTest.java:68)testClose(org.apache.qpid.server.protocol.v1_0.store.bdb.BDBLinkStoreTest) Time elapsed: 0.004 sec <<< ERROR!java.lang.IllegalStateException: Could not access Zing management bean. Make sure -XX:+UseZingMXBeans was specified. at org.apache.qpid.server.protocol.v1_0.store.bdb.BDBLinkStoreTest.createLinkStore(BDBLinkStoreTest.java:68)Caused by: java.lang.ClassNotFoundException: com.azul.zing.management.ManagementFactory at org.apache.qpid.server.protocol.v1_0.store.bdb.BDBLinkStoreTest.createLinkStore(BDBLinkStoreTest.java:68)testDeleteLink(org.apache.qpid.server.protocol.v1_0.store.bdb.BDBLinkStoreTest) Time elapsed: 0.01 sec <<< ERROR!java.lang.IllegalStateException: Could not access Zing management bean. Make sure -XX:+UseZingMXBeans was specified. at org.apache.qpid.server.protocol.v1_0.store.bdb.BDBLinkStoreTest.createLinkStore(BDBLinkStoreTest.java:68)Caused by: java.lang.ClassNotFoundException: com.azul.zing.management.ManagementFactory at org.apache.qpid.server.protocol.v1_0.store.bdb.BDBLinkStoreTest.createLinkStore(BDBLinkStoreTest.java:68)testDelete(org.apache.qpid.server.protocol.v1_0.store.bdb.BDBLinkStoreTest) Time elapsed: 0.007 sec <<< ERROR!java.lang.IllegalStateException: Could not access Zing management bean. Make sure -XX:+UseZingMXBeans was specified. at org.apache.qpid.server.protocol.v1_0.store.bdb.BDBLinkStoreTest.createLinkStore(BDBLinkStoreTest.java:68)Caused by: java.lang.ClassNotFoundException: com.azul.zing.management.ManagementFactory at org.apache.qpid.server.protocol.v1_0.store.bdb.BDBLinkStoreTest.createLinkStore(BDBLinkStoreTest.java:68)Results :Tests in error: BDBLinkStoreTest>QpidTestCase.run:165->LinkStoreTestCase.setUp:55->createLinkStore:68 » IllegalState BDBLinkStoreTest>QpidTestCase.run:165->LinkStoreTestCase.setUp:55->createLinkStore:68 » IllegalState BDBLinkStoreTest>QpidTestCase.run:165->LinkStoreTestCase.setUp:55->createLinkStore:68 » IllegalState BDBLinkStoreTest>QpidTestCase.run:165->LinkStoreTestCase.setUp:55->createLinkStore:68 » IllegalState BDBLinkStoreTest>QpidTestCase.run:165->LinkStoreTestCase.setUp:55->createLinkStore:68 » IllegalStateTests run: 5, Failures: 0, Errors: 5, Skipped: 0[INFO] [INFO] Reactor Summary:[INFO] [INFO] Apache Qpid Broker-J Parent SUCCESS [ 1.945 s][INFO] Apache Qpid Broker-J Code Generation ... SUCCESS [ 1.593 s][INFO] Apache Qpid Broker-J Test Utilities SUCCESS [ 0.270 s][INFO] Apache Qpid Broker-J Core .. SUCCESS [ 51.180 s][INFO] Apache Qpid Broker-J LogBack Logging Plug-in ... SUCCESS [ 4.200 s][INFO] Apache Qpid Broker-J Access Control Plug-in SUCCESS [ 3.223 s][INFO] Apache Qpid Broker-J AMQP 0-10 Protocol Plug-in SUCCESS [ 4.107 s][INFO] Apache Qpid Broker-J AMQP 0-8 Protocol Plug-in . SUCCESS [ 4.030 s][INFO] Apache Qpid Broker-J AMQP 1-0 Protocol Plug-in .
Re: Visual Studio Compliation Error: Any ideas/help Welcomed!
On Mon, Sep 24, 2018 at 4:36 PM Alan Conway wrote: > > > On Mon, Sep 24, 2018 at 9:48 AM Chuck Rolke wrote: > >> This was discussed in https://issues.apache.org/jira/browse/QPID-7926 >> >> I'm not sure how far the resolution made it into the upstream source. >> > > Looks like that never got merged, seems like an oversight. > Correction, it was merged on Apr 10 but it has not yet been released. It will be in the next release. The fix was this commit: 9f823d4df8 QPID-7926: [c++ broker] Windows build error "cannot convert from 'int' to 'qpid::sys::PODMutex'"
Re: qpid-proton-cpp decoder.cpp - possible bug?
On Mon, Sep 24, 2018 at 9:42 AM Nathan wrote:
> Hi,
>
> I applied the bug fix below into the code for 0.24.0
>
> It fixes that issue, but appears to have a side effect of causing open ssl
> to disconnect too soon - when the fix below is in place we get "no more
> data" exceptions being thrown from decoder.cpp around line 75, but when the
> fix below is removed the ssl session continues no problem.
>
> This is the throw:
>
> decoder::pre_get() {
> if (!next()) throw conversion_error("no more data");
>
> That doesnt happen when the fix below is removed.
>
> Wasnt sure if perhaps I need something else from 0.25 to fix that, or if
> the fix below introduced this new issue?
>
That doesn't ring any bells, possibly this is a regression that we missed
in testing. Can you open a JIRA with the details of how to reproduce it?
I'll get on it ASAP.
Thanks,
Alan.
> Thanks
> Nathan
>
>
>
> From: Alan Conway [mailto:acon...@redhat.com]
> Sent: 07 September 2018 16:10
> To: users@qpid.apache.org[mailto:users@qpid.apache.org]
> Cc: n...@gmx.es[mailto:n...@gmx.es]
> Subject: Re: qpid-proton-cpp decoder.cpp - possible bug?
>
> I think this is a bug I fixed recently, it should be in the latest release
> proton-0.25. If you still have a problem please raise a JIRA.
>
> Here's the fix - NULL is somewhat special but it is a valid scalar type,
> so I added it to type_id_is_scalar()
>
>
> https://github.com/alanconway/qpid-proton/commit/9e8edc17#diff-d6a2b218a8187976430ae388c2a9b176[https://github.com/alanconway/qpid-proton/commit/9e8edc17#diff-d6a2b218a8187976430ae388c2a9b176]
>
>
>
> On Thu, Sep 6, 2018 at 7:55 AM, mailto:n...@gmx.es]> wrote:
> Hi,
>
> Around line 180, decoder.cpp has the following:
>
>
> decoder& decoder::operator>>(scalar& x) {
> internal::state_guard sg(*this);
> type_id got = pre_get();
>
> if (!type_id_is_scalar(got))
> throw conversion_error("expected scalar, found "+type_name(got));
>
> x.set(pn_data_get_atom(pn_object()));
> sg.cancel(); // No error, no rewind
> return *this;
> }
>
>
> When our client code is talking to a 3rd party system's broker, we find
> that the scalar in argument "x" is coming through from the other party as
> type NULL in some cases - which causes the throw in the above method to get
> triggered and then disconnects us from the broker.
>
> However, if we comment out the if/throw from the code in decoder.cpp,
> everything then appears to be workding 100% correctly.
>
> Is there a correct way to do this without removing the throw? - it seems
> that if the check is there it must have a purpose, so by removing the check
> I am probably opening myself up to some other issues further down the line?
>
> Thanks
> N
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
> For additional commands, e-mail: users-h...@qpid.apache.org
>
>
Re: Visual Studio Compliation Error: Any ideas/help Welcomed!
On Mon, Sep 24, 2018 at 9:48 AM Chuck Rolke wrote: > This was discussed in https://issues.apache.org/jira/browse/QPID-7926 > > I'm not sure how far the resolution made it into the upstream source. > Looks like that never got merged, seems like an oversight. > - Original Message - > > From: "Paul A. Flores" > > To: users@qpid.apache.org > > Sent: Tuesday, September 11, 2018 5:50:55 PM > > Subject: Visual Studio Compliation Error: Any ideas/help Welcomed! > > > > Hi, > > > > > > I am getting the following compilation error and I am a bit baffled how > to > > resolve!? Any ideas are welcomed! > > > > > > Thanks in advance! > > > > > > Paul > > > > > > > > --- > > > > > > c:\qpid-cpp-1.38.0\src\qpid\log\logger.cpp(48): error C2440: > 'initializing': > > cannot convert from 'int' to 'qpid::sys::PODMutex' > > > > c:\qpid-cpp-1.38.0\src\qpid\log\logger.cpp(48): note: No constructor > could > > take the source type, or constructor overload resolution was ambiguous > > > > > > > > > > > > This communication (including any attachments) may contain information > that > > is proprietary, confidential or exempt from disclosure. If you are not > the > > intended recipient, please note that further dissemination, distribution, > > use or copying of this communication is strictly prohibited. Anyone who > > received this message in error should notify the sender immediately by > > telephone or by return email and delete it from his or her computer. > > > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > >
Re: Visual Studio Compliation Error: Any ideas/help Welcomed!
This was discussed in https://issues.apache.org/jira/browse/QPID-7926 I'm not sure how far the resolution made it into the upstream source. - Original Message - > From: "Paul A. Flores" > To: users@qpid.apache.org > Sent: Tuesday, September 11, 2018 5:50:55 PM > Subject: Visual Studio Compliation Error: Any ideas/help Welcomed! > > Hi, > > > I am getting the following compilation error and I am a bit baffled how to > resolve!? Any ideas are welcomed! > > > Thanks in advance! > > > Paul > > > > --- > > > c:\qpid-cpp-1.38.0\src\qpid\log\logger.cpp(48): error C2440: 'initializing': > cannot convert from 'int' to 'qpid::sys::PODMutex' > > c:\qpid-cpp-1.38.0\src\qpid\log\logger.cpp(48): note: No constructor could > take the source type, or constructor overload resolution was ambiguous > > > > > > This communication (including any attachments) may contain information that > is proprietary, confidential or exempt from disclosure. If you are not the > intended recipient, please note that further dissemination, distribution, > use or copying of this communication is strictly prohibited. Anyone who > received this message in error should notify the sender immediately by > telephone or by return email and delete it from his or her computer. > - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: qpid-proton-cpp decoder.cpp - possible bug?
Hi,
I applied the bug fix below into the code for 0.24.0
It fixes that issue, but appears to have a side effect of causing open ssl to
disconnect too soon - when the fix below is in place we get "no more data"
exceptions being thrown from decoder.cpp around line 75, but when the fix below
is removed the ssl session continues no problem.
This is the throw:
decoder::pre_get() {
if (!next()) throw conversion_error("no more data");
That doesnt happen when the fix below is removed.
Wasnt sure if perhaps I need something else from 0.25 to fix that, or if the
fix below introduced this new issue?
Thanks
Nathan
From: Alan Conway [mailto:acon...@redhat.com]
Sent: 07 September 2018 16:10
To: users@qpid.apache.org[mailto:users@qpid.apache.org]
Cc: n...@gmx.es[mailto:n...@gmx.es]
Subject: Re: qpid-proton-cpp decoder.cpp - possible bug?
I think this is a bug I fixed recently, it should be in the latest release
proton-0.25. If you still have a problem please raise a JIRA.
Here's the fix - NULL is somewhat special but it is a valid scalar type, so I
added it to type_id_is_scalar()
https://github.com/alanconway/qpid-proton/commit/9e8edc17#diff-d6a2b218a8187976430ae388c2a9b176[https://github.com/alanconway/qpid-proton/commit/9e8edc17#diff-d6a2b218a8187976430ae388c2a9b176]
On Thu, Sep 6, 2018 at 7:55 AM, mailto:n...@gmx.es]> wrote:
Hi,
Around line 180, decoder.cpp has the following:
decoder& decoder::operator>>(scalar& x) {
internal::state_guard sg(*this);
type_id got = pre_get();
if (!type_id_is_scalar(got))
throw conversion_error("expected scalar, found "+type_name(got));
x.set(pn_data_get_atom(pn_object()));
sg.cancel(); // No error, no rewind
return *this;
}
When our client code is talking to a 3rd party system's broker, we find that
the scalar in argument "x" is coming through from the other party as type NULL
in some cases - which causes the throw in the above method to get triggered and
then disconnects us from the broker.
However, if we comment out the if/throw from the code in decoder.cpp,
everything then appears to be workding 100% correctly.
Is there a correct way to do this without removing the throw? - it seems that
if the check is there it must have a purpose, so by removing the check I am
probably opening myself up to some other issues further down the line?
Thanks
N
-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
Re: [Dispatch-Router] max message size
Glad to know it is working and thanks for testing.
On Mon, Sep 24, 2018 at 8:21 AM VERMEULEN Olivier <
olivier.vermeu...@murex.com> wrote:
> Hello,
>
> We tested against the master and it is now working fine.
>
> Thanks,
> Olivier
>
> -Original Message-
> From: Ganesh Murthy
> Sent: jeudi 20 septembre 2018 17:59
> To: users@qpid.apache.org
> Subject: Re: [Dispatch-Router] max message size
>
> Here is the sequence of events we gleaned from the log file (btw, it looks
> like you sent your router config file but we did not get it, it was
> stripped)
>
> The Router has two Java brokers attached to it and autolinks on the same
> address to those two brokers
>
> 1. A message over 10kb is sent and the router forwards it to Broker 1 over
> the autolink. The message exceeds the max message size set by the broker
> and so the broker detaches the autolink and closes the session. The
> autolink from Router to Broker 1 is now destroyed. The only remaining
> autolink is to Broker2 2. A message smaller than 10kb is sent and the
> router sends it on the autolink to Broker 2. (all good here) 3. Once again,
> a message over 10kb is sent, and that is forwarded to Broker
> 2 and Broker 2 rejects that messages because it is too big and detaches
> the link and closes the session. Now the autolinks on Broker 2 are
> destroyed as well. Now, since both autolinks are gone, any further messages
> sent to that address 'myQueue' have nowhere to go and hence the final send
> hangs (the router does not issue a flow to the sender because there are no
> consumers for the message)
>
> On the master branch, we added an enhancement -
> https://issues.apache.org/jira/browse/DISPATCH-1103 - which would retry
> and re-establish failed autolinks. This new feature will fix your issue.
> Unfortunately, this new feature will only be available in Qpid Dispatch
> 1.4.0 which is scheduled to go out mid to late October.
>
> Is it possible for you to get Dispatch running with the latest master
> branch code and see if this problem goes away?
>
> Thanks.
>
> On Thu, Sep 20, 2018 at 10:22 AM VERMEULEN Olivier <
> olivier.vermeu...@murex.com> wrote:
>
> > Here are the files.
> >
> > Thanks,
> > Olivier
> >
> > -Original Message-
> > From: Ganesh Murthy
> > Sent: jeudi 20 septembre 2018 15:19
> > To: users@qpid.apache.org
> > Subject: Re: [Dispatch-Router] max message size
> >
> > Can you please enable trace logging on the router and share the log file?
> >
> > To enable trace logging add the following to your router configuration
> > file.
> >
> > log {
> > module: DEFAULT
> > enable: trace+
> > output: qdrouterd.log
> > }
> >
> > Also please share your router config file if possible and which broker
> > you are using to test.
> >
> >
> > On Thu, Sep 20, 2018 at 8:47 AM VERMEULEN Olivier <
> > olivier.vermeu...@murex.com> wrote:
> >
> > > Hello,
> > >
> > > We did a test with 1 dispatch-router and 2 brokers.
> > > On the brokers we configured a maximum message size of 10KB.
> > > The use case is the following:
> > >
> > > * We send a message above 10 KB: we receive a "failure at remote"
> > > exception from the dispatch-router
> > > * We send a message under 10KB: send is successful
> > > * We send a message above 10 KB again: we receive a "failure at
> > > remote" exception from the dispatch-router
> > > * We send a message under 10KB again: the send hangs...
> > >
> > > This was reproduced with the latest version of the dispatch-router.
> > > Do you have any idea what is going on and is it a known issue?
> > >
> > > Thanks,
> > > Olivier
> > >
> > > ***
> > >
> > > This e-mail contains information for the intended recipient only. It
> > > may contain proprietary material or confidential information. If you
> > > are not the intended recipient you are not authorised to distribute,
> > > copy or use this e-mail or any attachment to it. Murex cannot
> > > guarantee that it is virus free and accepts no responsibility for
> > > any loss or damage arising from its use. If you have received this
> > > e-mail in error please notify immediately the sender and delete the
> > > original email received, any attachments and all copies from your
> system.
> > >
> > ***
> >
> > This e-mail contains information for the intended recipient only. It
> > may contain proprietary material or confidential information. If you
> > are not the intended recipient you are not authorised to distribute,
> > copy or use this e-mail or any attachment to it. Murex cannot
> > guarantee that it is virus free and accepts no responsibility for any
> > loss or damage arising from its use. If you have received this e-mail
> > in error please notify immediately the sender and delete the original
> > email received, any attachments and all copies from your system.
> >
> > -
> > To unsubscribe, e-mail:
RE: [Dispatch-Router] max message size
Hello,
We tested against the master and it is now working fine.
Thanks,
Olivier
-Original Message-
From: Ganesh Murthy
Sent: jeudi 20 septembre 2018 17:59
To: users@qpid.apache.org
Subject: Re: [Dispatch-Router] max message size
Here is the sequence of events we gleaned from the log file (btw, it looks like
you sent your router config file but we did not get it, it was
stripped)
The Router has two Java brokers attached to it and autolinks on the same
address to those two brokers
1. A message over 10kb is sent and the router forwards it to Broker 1 over the
autolink. The message exceeds the max message size set by the broker and so the
broker detaches the autolink and closes the session. The autolink from Router
to Broker 1 is now destroyed. The only remaining autolink is to Broker2 2. A
message smaller than 10kb is sent and the router sends it on the autolink to
Broker 2. (all good here) 3. Once again, a message over 10kb is sent, and that
is forwarded to Broker
2 and Broker 2 rejects that messages because it is too big and detaches the
link and closes the session. Now the autolinks on Broker 2 are destroyed as
well. Now, since both autolinks are gone, any further messages sent to that
address 'myQueue' have nowhere to go and hence the final send hangs (the router
does not issue a flow to the sender because there are no consumers for the
message)
On the master branch, we added an enhancement -
https://issues.apache.org/jira/browse/DISPATCH-1103 - which would retry and
re-establish failed autolinks. This new feature will fix your issue.
Unfortunately, this new feature will only be available in Qpid Dispatch
1.4.0 which is scheduled to go out mid to late October.
Is it possible for you to get Dispatch running with the latest master branch
code and see if this problem goes away?
Thanks.
On Thu, Sep 20, 2018 at 10:22 AM VERMEULEN Olivier <
olivier.vermeu...@murex.com> wrote:
> Here are the files.
>
> Thanks,
> Olivier
>
> -Original Message-
> From: Ganesh Murthy
> Sent: jeudi 20 septembre 2018 15:19
> To: users@qpid.apache.org
> Subject: Re: [Dispatch-Router] max message size
>
> Can you please enable trace logging on the router and share the log file?
>
> To enable trace logging add the following to your router configuration
> file.
>
> log {
> module: DEFAULT
> enable: trace+
> output: qdrouterd.log
> }
>
> Also please share your router config file if possible and which broker
> you are using to test.
>
>
> On Thu, Sep 20, 2018 at 8:47 AM VERMEULEN Olivier <
> olivier.vermeu...@murex.com> wrote:
>
> > Hello,
> >
> > We did a test with 1 dispatch-router and 2 brokers.
> > On the brokers we configured a maximum message size of 10KB.
> > The use case is the following:
> >
> > * We send a message above 10 KB: we receive a "failure at remote"
> > exception from the dispatch-router
> > * We send a message under 10KB: send is successful
> > * We send a message above 10 KB again: we receive a "failure at
> > remote" exception from the dispatch-router
> > * We send a message under 10KB again: the send hangs...
> >
> > This was reproduced with the latest version of the dispatch-router.
> > Do you have any idea what is going on and is it a known issue?
> >
> > Thanks,
> > Olivier
> >
> > ***
> >
> > This e-mail contains information for the intended recipient only. It
> > may contain proprietary material or confidential information. If you
> > are not the intended recipient you are not authorised to distribute,
> > copy or use this e-mail or any attachment to it. Murex cannot
> > guarantee that it is virus free and accepts no responsibility for
> > any loss or damage arising from its use. If you have received this
> > e-mail in error please notify immediately the sender and delete the
> > original email received, any attachments and all copies from your system.
> >
> ***
>
> This e-mail contains information for the intended recipient only. It
> may contain proprietary material or confidential information. If you
> are not the intended recipient you are not authorised to distribute,
> copy or use this e-mail or any attachment to it. Murex cannot
> guarantee that it is virus free and accepts no responsibility for any
> loss or damage arising from its use. If you have received this e-mail
> in error please notify immediately the sender and delete the original
> email received, any attachments and all copies from your system.
>
> -
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For
> additional commands, e-mail: users-h...@qpid.apache.org
***
This e-mail contains information for the intended recipient only. It may
contain proprietary material or confidential information. If you are not the
intended recipient you are not authorised to distribute, copy or use this
RE: [Broker-J] JDBC message store performance
Hello Rob, I think there is something wrong with your patch, it only contains the git-diff manual... Olivier -Original Message- From: Rob Godfrey Sent: mercredi 19 septembre 2018 09:31 To: users@qpid.apache.org Subject: Re: [Broker-J] JDBC message store performance I'm on vacation at the moment, but if I get a chance I'll at least try to test with Derby. -- Rob On Wed, 19 Sep 2018 at 09:14, VERMEULEN Olivier wrote: > Thanks a lot Rob, I'll try to test that next week. > > Olivier > > -Original Message- > From: Rob Godfrey > Sent: mercredi 19 septembre 2018 01:23 > To: users@qpid.apache.org > Subject: Re: [Broker-J] JDBC message store performance > > As an alternative approach, how about the (totally untested) patch > I've attached to QPID-8242 [1] > > Rather than trying to consolidate the commits to delete the queue > entry and the message content it instead attempts to delete the > message content asynchronously and batch up all message contents > awaiting deletion. From a performance point of view, on the broker it > takes the message content deletion off the connection thread, and if > the database is being the bottleneck it should start batching up the > deletions of content. > > -- Rob > > [1] > https://issues.apache.org/jira/secure/attachment/12940318/QPID-8242.pa > tch > > On Mon, 17 Sep 2018 at 15:22, Rob Godfrey wrote: > > > The reference counting is done in AbstractServerMessageImpl.java. > > In general instances of ServerMessage should not be passed around, > > rather a MessageReference (obtained by calling newReference(..) on > > the message object. Then if the message is no longer required in > > that context, > > release() can be called on the reference. Within the context of a > > queue this is normally encapsulated by the notion of the QueueEntry > > (which holds the relevant MessageReference). When delete() is > > called on the QueueEntry (delete being defined in the > > MessageInstance interface), this releases the reference, which > > decrements the reference count, and if the count has gone to zero > > the message calls the > store to delete itself. > > > > I think trying to change this logic would be a bit of a nightmare > > (understatement). I think a better alternative in terms of reducing > > the number of commits in the JDBC store might instead be for the > > remove() method on StoredJDBCMessage to (rather than immediately > > commit the message > > delete) schedule the message removal to be picked up by the next > > commit that the store is asked to perform. This would make the > > behaviour more like the BDB store (where we schedule the commit but > > don't actually force the sync to disk on message deletion). > > > > -- Rob > > > > On Mon, 17 Sep 2018 at 14:55, VERMEULEN Olivier < > > olivier.vermeu...@murex.com> wrote: > > > >> Ok then I missed something. > >> Whan/Where is the reference-counting you were talking about in your > >> first mail happening? > >> > >> Olivier > >> > >> -Original Message- > >> From: Rob Godfrey > >> Sent: lundi 17 septembre 2018 14:05 > >> To: users@qpid.apache.org > >> Subject: Re: [Broker-J] JDBC message store performance > >> > >> Hi Olivier, > >> > >> the approach you are attempting will not work for the reasons I > >> described previously. If the message has (for instance) been > >> placed in two durable subscription queues (because there are two > >> durable > >> subscriptions) then as soon as the message is consumed from the > >> first queue it would be removed from the store, leading to problems > >> when the second consumer tries to read the message. > >> > >> -- Rob > >> > >> On Mon, 17 Sep 2018 at 11:39, VERMEULEN Olivier < > >> olivier.vermeu...@murex.com> > >> wrote: > >> > >> > Hello Rob, > >> > > >> > Thanks for the answer. > >> > I started looking at the code to see if there is something I can > >> > do about these 2 commits. > >> > But before going any further I'd like your input on the below, to > >> > see if what I'm trying to do could work or if I'm missing > >> > something (which I'm surely are) > >> > > >> > https://github.com/apache/qpid-broker-j/compare/master...overmeul > >> > en > >> > :fe > >> > ature/jdbc-message-store-commits > >> > > >> > Regards, > >> > Olivier > >> > > >> > -Original Message- > >> > From: Rob Godfrey > >> > Sent: vendredi 14 septembre 2018 16:06 > >> > To: users@qpid.apache.org > >> > Cc: AYOUBI Ali > >> > Subject: Re: [Broker-J] JDBC message store performance > >> > > >> > On Fri, 14 Sep 2018 at 15:30, VERMEULEN Olivier < > >> > olivier.vermeu...@murex.com> > >> > wrote: > >> > > >> > > Hello, > >> > > > >> > > We ran a performance test with a bunch of brokers and an Oracle > >> > > database to store the messages. > >> > > We noticed that the database was a bit overloaded with commits. > >> > > Looking at the logs we saw that sending a message was > >> > > triggering > >> > > 1 commit for 3 operations
