Re: Messages getting lost on Artemis 2.25

2022-10-27 Thread Clebert Suconic
>> This version worked a lot better for us, but as long as we're unable to reproduce and fix the missing messages, we're staining on 2.20 for now.<<< I have been working closely with some customers running huge workloads, and I have never seen any message loss.. I believe you probably ran on a

Re: Messages getting lost on Artemis 2.25

2022-10-26 Thread Walter de Boer
@Clebert, thanks for the update! @Caspar, with 2.20 we do see memory issues that I believe are solved in later versions. But we didn't lose messages the same way we did with 2.24. With 2.20 we're monitoring for AMQ222162 after a crash. We then recover the old replica's just in case

Re: Messages getting lost on Artemis 2.25

2022-10-26 Thread Clebert Suconic
d you are reverting back to that version? If so, maybe we should > check the changes between 2.20 and 2.22. > > > > > From: Clebert Suconic > Sent: 25 October 2022 18:56 > To: users@activemq.apache.org > Subject: Re: Messages getting lost on Artemi

Re: Messages getting lost on Artemis 2.25

2022-10-26 Thread Gašper Čefarin
, since you said you are reverting back to that version? If so, maybe we should check the changes between 2.20 and 2.22. From: Clebert Suconic Sent: 25 October 2022 18:56 To: users@activemq.apache.org Subject: Re: Messages getting lost on Artemis 2.25

Re: Messages getting lost on Artemis 2.25

2022-10-25 Thread Clebert Suconic
There's a bug fix I'm pushing now on the page counters: https://github.com/apache/activemq-artemis/pull/4271 notice there's no page loss.. just the counter could go off sync if paging. I'm also going to add an option to remove the page-counters record soon. On Thu, Oct 20, 2022 at 4:37 PM

Re: Messages getting lost on Artemis 2.25

2022-10-20 Thread Walter de Boer
Clebert, We see the AMQ222162 message after restarting on a JVM crash. In that case we can recover the messages. Maybe this is any help to Gašper The last time we really lost messages we didn’t notice any particular errors at server not client side. I think we’ll find some commercial

Re: Messages getting lost on Artemis 2.25

2022-10-20 Thread Clebert Suconic
I believe you're saying you got that message? are you getting into a failover scenario and perhaps a split brain? you could use ZooKeeper to avoid such thing. In case you're using retention, look at the replay method in the Address.. where you can extra messages from your queue in the period

Re: Messages getting lost on Artemis 2.25

2022-10-20 Thread Walter de Boer
Gašper, Did you see "AMQ222162 Moving data directory ..." in your logs? In that case you should use ./artemis data exp (and ./artemis data imp) to export the old replicas and find your messages there (and import them agian). See Artemis Data Tools in the manual. Regards, Walter

Re: Messages getting lost on Artemis 2.25

2022-10-20 Thread Walter de Boer
P.s. we'll also look at retention and enable message logging. We're not even sure now wether the missing messages have been accepted to the queue or not...

Re: Messages getting lost on Artemis 2.25

2022-10-20 Thread Walter de Boer
Clebert, Thanks for your reply. I'm aware you need a way to reproduce it. Unfortunately this only happens once in a while and we're not able to reproduce yet. We wil investigate further, thanks for the instructions! We'll keep you posted. We'll also have a look at OFF-WITH-REDISTRIBUTION, I

Re: Messages getting lost on Artemis 2.25

2022-10-20 Thread Clebert Suconic
re.client] > >>> AMQ212037: Connection failure to 10.108.28.52/10.108.28.52:9000 has been > >>> detected: AMQ219015: The connection was disconnected because of server > >>> shutdown [code=DISCONNECTED] > >>> > >>> The only thing that come

Re: Messages getting lost on Artemis 2.25

2022-10-20 Thread Clebert Suconic
port for cluster communication from default 61616 to 9000 (i've experienced >>> some problems unrelated to message loss when changing the port). >>> >>> Any advice on reproducing the issue or where to look for more data >>> appreciated. >>> >

Re: Messages getting lost on Artemis 2.25

2022-10-20 Thread Clebert Suconic
uld be the problem is changing >> the port for cluster communication from default 61616 to 9000 (i've >> experienced some problems unrelated to message loss when changing the port). >> >> Any advice on reproducing the issue or where to look for more data >> appreciated. &g

Re: Messages getting lost on Artemis 2.25

2022-10-20 Thread Clebert Suconic
ted. > > -Original Message- > From: Clebert Suconic > Sent: Wednesday, October 19, 2022 4:58 PM > To: users@activemq.apache.org > Subject: Re: Messages getting lost on Artemis 2.25 > > > To sporočilo izvira izven naše organizacije. Bodite pozorni pri vsebini in &g

RE: Messages getting lost on Artemis 2.25

2022-10-20 Thread Gašper Čefarin
). Any advice on reproducing the issue or where to look for more data appreciated. -Original Message- From: Clebert Suconic Sent: Wednesday, October 19, 2022 4:58 PM To: users@activemq.apache.org Subject: Re: Messages getting lost on Artemis 2.25 To sporočilo izvira izven naše organizacije

Re: Messages getting lost on Artemis 2.25

2022-10-19 Thread Clebert Suconic
Basically I'm telling you how to investigate it.. and if you find an issue on the broker, we will need a way to reproduce it. I have no other report about a message loss situation... (we do have situations with page-counters going wrong while paging..which I'm working now to fix it... but no

Re: Messages getting lost on Artemis 2.25

2022-10-19 Thread Clebert Suconic
I am not aware of any issues that would lead to message loss... Garbage Collection itself has no effect on anything regarding paging or journal. Are you able to chase which message is lost on a test? you could use the retention feature, replay the message.. and you could also look on the

Re: Messages getting lost on Artemis 2.25

2022-10-19 Thread Walter de Boer
All, This week we lost 23.000 messages in a few days time on our production Cluster running Artemis 2.26.0, see our settings below. We've reverted back to Artemis 2.20.0 just in case A few observatoins: * In version 2.24.0, 2.25.0 and 2.26.0 running on ZGC we noticed messages being

Re: Messages getting lost on Artemis 2.25

2022-10-11 Thread Jelmer Marinus
We are trying to create a scenario which reproduces the issue in a few specific steps. But we haven't yet. From: Clebert Suconic Sent: Thursday, October 6, 2022 1:51 PM To: users@activemq.apache.org Subject: Re: Messages getting lost on Artemis 2.25 Do you have

Re: Messages getting lost on Artemis 2.25

2022-10-06 Thread Clebert Suconic
Do you have a test to reproduce the issue ? On Thu, Oct 6, 2022 at 3:16 AM Jelmer Marinus wrote: > Hi, > > During the testing of the message-counters (see other mail) we also seem > to have lost messages. It looks like these messages get lost during > redistribution or when they get produced