the image.png you sent could not be delivered by the infra for some reason.
It's rare an occasion where I affirm I am 100% sure about something. I usually always keep a margin for the changing my opinion, or just being wrong.. But on this case I am 10000% sure you will hit the bug I fixed. The issue is on creating new sessions. I added some instructions on how you could reproduce the issue on my last comment: https://issues.apache.org/jira/browse/ARTEMIS-3848 Also, as part of my day job (I work for Red Hat), I have a JIRA where I also describe the issue: https://issues.redhat.com/browse/ENTMQBR-6683 I would just go straight to the upgrade to be honest.. you could do some acceptance test as it's usually standard before you accept any new software into your production system. On Tue, Jun 7, 2022 at 5:47 PM Ben Warrick <horseatingwe...@gmail.com> wrote: > @Clebert: Thanks for finding this. Here is some more evidence for you. > This is only one monkey on my back right now, but I do plan on upgrading. > I'll get you some feedback for sure. If I can manage, I want to try > reproducing this in a lower environment. > > [image: image.png] > > On Tue, Jun 7, 2022 at 5:07 PM Clebert Suconic <clebert.suco...@gmail.com> > wrote: > >> @Ben Warrick: BTW When I said "you are likely" hitting the bug, I was >> just using polite words... >> >> >> I am 100% sure you are hitting that condition by creating one session >> per message. You should definitely look to upgrade or cherry-pick the >> fix to your own branch. >> >> On Tue, Jun 7, 2022 at 3:14 PM Clebert Suconic >> <clebert.suco...@gmail.com> wrote: >> > >> > The release candidate : >> > >> > >> > https://dist.apache.org/repos/dist/dev/activemq/activemq-artemis/2.23.0 >> > >> > I would appreciate your feedback here or even better on the dev list at >> the voting thread. >> > >> > On Tue, Jun 7, 2022 at 12:52 PM Clebert Suconic < >> clebert.suco...@gmail.com> wrote: >> >> >> >> right, but since I'm just releasing it would be great to hear his >> >> feedback on top of the release candidate itself. >> >> >> >> On Tue, Jun 7, 2022 at 12:38 PM Justin Bertram <jbert...@apache.org> >> wrote: >> >> > >> >> > A snapshot [1] is always a possibility as well. >> >> > >> >> > >> >> > Justin >> >> > >> >> > [1] >> >> > >> https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-artemis/2.23.0-SNAPSHOT/ >> >> > >> >> > On Tue, Jun 7, 2022 at 11:34 AM Clebert Suconic < >> clebert.suco...@gmail.com> >> >> > wrote: >> >> > >> >> > > I am preparing artemis 2.23.0 at the moment... >> >> > > >> >> > > The way a release works in apache, you send it for vote and it >> will be >> >> > > officially released after 72 hours of a passing vote. but you >> should >> >> > > have a candidate release today that you can try it out. Based on >> the >> >> > > current diffs it should be fairly safe for you to upgrade. >> >> > > >> >> > > 2.14.0 would require JDK 1.8 minimal. while artemis 2.23 will >> require >> >> > > JDK 11 minimal. >> >> > > >> >> > > >> >> > > I will send you a link to the Candidate release as soon as it's >> ready >> >> > > on this thread. >> >> > > >> >> > > On Mon, Jun 6, 2022 at 10:25 PM Ben Warrick < >> horseatingwe...@gmail.com> >> >> > > wrote: >> >> > > > >> >> > > > Hi Clebert, >> >> > > > >> >> > > > Yeah, I can use whatever I want after I prove it out in lower >> >> > > environments. >> >> > > > >> >> > > > Thanks, >> >> > > > >> >> > > > Ben >> >> > > > >> >> > > > On Mon, Jun 6, 2022 at 10:06 PM Clebert Suconic < >> >> > > clebert.suco...@gmail.com> >> >> > > > wrote: >> >> > > > >> >> > > > > Can you use jdk 11? >> >> > > > > >> >> > > > > On Mon, Jun 6, 2022 at 8:32 PM Clebert Suconic < >> >> > > clebert.suco...@gmail.com> >> >> > > > > wrote: >> >> > > > > >> >> > > > > > You are likely hitting the bug I fixed at: >> >> > > > > > >> >> > > > > > https://github.com/apache/activemq-artemis/pull/4098 >> >> > > > > > >> >> > > > > > I introduced a Thread Local Leakage. I thought by setting >> >> > > > > > threadLocal.set(null) I would be clearing it... and in fact >> I just >> >> > > > > > updated the threadLocal variable with null. >> >> > > > > > >> >> > > > > > That ThreadLocal is created for every new session. >> >> > > > > > >> >> > > > > > >> >> > > > > > I was going to start a release today but I couldn't start it >> today as >> >> > > > > > I'm a bit sick today. I should be able to start a release >> tomorrow >> >> > > and >> >> > > > > > then you could try it out. >> >> > > > > > >> >> > > > > > >> >> > > > > > your other option would be to cherry-pick the change and >> build the >> >> > > > > > version yourself so you can run it in production. >> >> > > > > > >> >> > > > > > On Fri, Jun 3, 2022 at 4:27 PM Francesco Nigro < >> nigro....@gmail.com> >> >> > > > > > wrote: >> >> > > > > > > >> >> > > > > > > Some more info about it here >> >> > > https://netty.io/wiki/tcp-fast-open.html >> >> > > > > > > >> >> > > > > > > Given that Netty can makes use of it and Artemis can as >> well: if >> >> > > it's >> >> > > > > not >> >> > > > > > > present in the doc it means we need to expose it in the >> acceptor >> >> > > > > > > configuration (for epoll) and...we love contributions :) >> >> > > > > > > But I suggest first to give it a shot first in a test env >> to see >> >> > > if it >> >> > > > > > > solves your problem! >> >> > > > > > > >> >> > > > > > > Il ven 3 giu 2022, 22:21 Ben Warrick < >> horseatingwe...@gmail.com> >> >> > > ha >> >> > > > > > scritto: >> >> > > > > > > >> >> > > > > > > > Hi Francesco, >> >> > > > > > > > >> >> > > > > > > > I don't find anything in the Artemis docs about >> TCP_FAST_OPEN. It >> >> > > > > must >> >> > > > > > not >> >> > > > > > > > be an exposed configuration on Artemis? Where do I find >> more on >> >> > > this? >> >> > > > > > > > >> >> > > > > > > > Thanks, >> >> > > > > > > > >> >> > > > > > > > Ben >> >> > > > > > > > >> >> > > > > > > > On Fri, Jun 3, 2022 at 4:08 PM Francesco Nigro < >> >> > > nigro....@gmail.com> >> >> > > > > > > > wrote: >> >> > > > > > > > >> >> > > > > > > > > Are coming, not commenting: typo introduced by the the >> phone, >> >> > > > > sorry! >> >> > > > > > > > > >> >> > > > > > > > > Il ven 3 giu 2022, 22:07 Francesco Nigro < >> nigro....@gmail.com> >> >> > > ha >> >> > > > > > > > scritto: >> >> > > > > > > > > >> >> > > > > > > > > > Hi Ben, >> >> > > > > > > > > > >> >> > > > > > > > > > You can use TCP_FAST_OPEN on Artemis acceptors in >> case >> >> > > > > connections >> >> > > > > > are >> >> > > > > > > > > > commenting from the came clients (I don't remember >> if is an >> >> > > > > exposes >> >> > > > > > > > Netty >> >> > > > > > > > > > configuration) or...submit a PR to let Artemis >> separate >> >> > > > > (multiple) >> >> > > > > > > > > threads >> >> > > > > > > > > > to accept connections but, as you said, this is just >> an >> >> > > > > > anti-pattern >> >> > > > > > > > and >> >> > > > > > > > > in >> >> > > > > > > > > > order to reduce the client code you can think about >> using a >> >> > > > > > transparent >> >> > > > > > > > > > connection pooling mechanism, instead. >> >> > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > > Il ven 3 giu 2022, 21:55 Ben Warrick < >> >> > > horseatingwe...@gmail.com> >> >> > > > > > ha >> >> > > > > > > > > > scritto: >> >> > > > > > > > > > >> >> > > > > > > > > >> I'm running Artemis 2.14, and one of the systems >> publishing >> >> > > to >> >> > > > > it >> >> > > > > > > > > creates >> >> > > > > > > > > >> a >> >> > > > > > > > > >> lot of separate tasks -- from its perspective. I >> can see on >> >> > > > > > Artemis >> >> > > > > > > > that >> >> > > > > > > > > >> the Total connection count vs Total messages >> acknowledged is >> >> > > > > > pretty >> >> > > > > > > > > close, >> >> > > > > > > > > >> so it's apparent that the client system is making >> use of the >> >> > > > > > > > > anti-pattern >> >> > > > > > > > > >> of creating a new connection for each message it >> publishes. >> >> > > I >> >> > > > > > don't >> >> > > > > > > > have >> >> > > > > > > > > >> access to the client system implementation, but is >> there >> >> > > > > anything >> >> > > > > > I >> >> > > > > > > > can >> >> > > > > > > > > do >> >> > > > > > > > > >> on the Artemis end to make it more efficient? >> Latency with >> >> > > this >> >> > > > > > system >> >> > > > > > > > > is >> >> > > > > > > > > >> becoming an issue. Also every 30 minutes or so >> there is a >> >> > > spike >> >> > > > > > in CPU >> >> > > > > > > > > >> usage (~50%) which I assume is garbage collection. >> During >> >> > > that >> >> > > > > > spike >> >> > > > > > > > > >> message publishing slows down a lot. >> >> > > > > > > > > >> >> >> > > > > > > > > >> Thanks, >> >> > > > > > > > > >> >> >> > > > > > > > > >> Ben >> >> > > > > > > > > >> >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > -- >> >> > > > > > Clebert Suconic >> >> > > > > > >> >> > > > > -- >> >> > > > > Clebert Suconic >> >> > > > > >> >> > > >> >> > > >> >> > > >> >> > > -- >> >> > > Clebert Suconic >> >> > > >> >> > > >> >> >> >> >> >> >> >> -- >> >> Clebert Suconic >> > >> > -- >> > Clebert Suconic >> >> >> >> -- >> Clebert Suconic >> > -- Clebert Suconic
