[
https://issues.apache.org/jira/browse/ARTEMIS-4276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17723982#comment-17723982
]
Liviu Citu edited comment on ARTEMIS-4276 at 5/18/23 5:32 PM:
--------------------------------------------------------------
> This is a weakness in the application design which will lead to the same
> problems with duplicate messages as you have when a broker failure causes the
> consumer-group relationship to change.
I am not seeing it as a weakness rather than an incomplete solution as we might
still have duplicated messages (as you said) which in the end will fail during
database import (see below). However, I think it is still better to have a
local cache than nothing. There might be some other complex solutions when load
balancing is used but this local cache will (still) help to reduce unnecessary
database insert failures.
> This leads to the same transaction being imported in the database twice..."
What I meant actually is that *_it will try to import_* the record in the
database. Of course that our database IO meta layer has a mechanism in place to
avoid same transaction being imported twice (the database records have audit
trail which include transaction version). This is because same database tables
can also be affected by other applications part of our software (UI, batch
utilities, etc) so it is not only the gateway interface who import data in the
system. I just wanted to pin point a potential issue that could arise in
applications during failover switch.
Regarding:
> Idempotency is something you, as the application developer, must implement.
There are some third parties that have this out-of-the box. For instance, I
have seen Kafka having idempotent consumers. Never used it though.
was (Author: JIRAUSER300236):
> This is a weakness in the application design which will lead to the same
> problems with duplicate messages as you have when a broker failure causes the
> consumer-group relationship to change.
I am not seeing it as a weakness rather than an incomplete solution as we might
still have duplicated messages (as you said) which in the end will fail during
database import (see below). However, in the context of load balancing I think
it is still better to have a local cache than nothing. There might be some
other complex solutions indeed but this local cache will (still) help
especially for those applications which do not have load balancing.
> This leads to the same transaction being imported in the database twice..."
What I meant actually is that *_it will try to import_* the record in the
database. Of course that our database IO meta layer has a mechanism in place to
avoid same transaction being imported twice (the database records have audit
trail which include transaction version). This is because same database tables
can also be affected by other applications part of our software (UI, batch
utilities, etc) so it is not only the gateway interface who import data in the
system. I just wanted to pin point a potential issue that could arise in
applications during failover switch.
Regarding:
> Idempotency is something you, as the application developer, must implement.
There are some third parties that have this out-of-the box. For instance, I
have seen Kafka having idempotent consumers. Never used it though.
> Message Group does not replicate properly during failover
> ---------------------------------------------------------
>
> Key: ARTEMIS-4276
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4276
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Affects Versions: 2.28.0
> Reporter: Liviu Citu
> Priority: Major
>
> Hi,
> We are currently migrating our software from Classic to Artemis and we plan
> to use failover functionality.
> We were using message group functionality by setting *JMSXGroupID* and this
> was working as expected. However after failover switch I noticed that
> messages are sent to wrong consumers.
> Our gateway/interface application is actually a collection of servers:
> * gateway adapter server: receives messages from an external systems and
> puts them on a specific/virtual topic
> * gateway loader server (can be balanced): picks up the messages from the
> topic and do processing
> * gateway fail queue: monitors all messages that failed processing and has a
> functionality of resubmitting the message (users will correct the processing
> errors and then resubmit transaction)
> *JMSXGroupID* is used to ensure that during message resubmit the same
> consumer/loader is processing the message as it was originally processed.
> However, if the message resubmit is happening during failover switch we have
> noticed that the message is not sent to the right consumer as it should.
> Basically the first available consumer is used which is not what we want.
> I have searched for configuration changes but couldn't find any relevant
> information.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
