Re: H/A colocated replication environment not working as expected

[andytaylor]

Can you make sue your messages are persistent. If they are try turning
scaledown off and see if there are any messages in the journal when the
back up comes live. If you are still having problems send me you're
configuration and I will have a look tomorrow

Andy

On 1 December 2016 at 17:07, grishezz [via ActiveMQ] <
ml-node+s2283324n4719679...@n4.nabble.com> wrote:

> You can see the entire logs, as well as the configuration I used in the
> first post in this thread, I am also copying the relevant parts here:
>
> server0:
> =
> 19:02:06,104 INFO  [org.apache.activemq.artemis.core.server] AMQ221109:
> Apache ActiveMQ Artemis Backup Server version 1.5.0 [null] started, waiting
> live to fail before it gets active
> 19:02:06,920 INFO  [org.apache.activemq.artemis.core.server] AMQ221024:
> Backup server 
> ActiveMQServerImpl::serverUUID=63157c62-b58c-11e6-ace0-b8a720524153
> is synchronized with live-server.
> 19:02:06,929 INFO  [org.apache.activemq.artemis.core.server] AMQ221031:
> backup announced
>
> server1:
> =
> 19:02:07,076 INFO  [org.apache.activemq.artemis.core.server] AMQ221109:
> Apache ActiveMQ Artemis Backup Server version 1.5.0 [null] started, waiting
> live to fail before it gets active
> 19:02:08,015 INFO  [org.apache.activemq.artemis.core.server] AMQ221024:
> Backup server 
> ActiveMQServerImpl::serverUUID=60b03b78-b58c-11e6-937d-b8a720524153
> is synchronized with live-server.
> 19:02:08,022 INFO  [org.apache.activemq.artemis.core.server] AMQ221031:
> backup announced
> ...
> 19:04:51,883 WARN  [org.apache.activemq.artemis.core.client] AMQ212037:
> Connection failure has been detected: AMQ119015: The connection was
> disconnected because of server shutdown [code=DISCONNECTED]
> 19:04:51,885 WARN  [org.apache.activemq.artemis.core.server] AMQ222095:
> Connection failed with failedOver=false
> 19:04:51,954 WARN  [org.apache.activemq.artemis.core.server] AMQ222095:
> Connection failed with failedOver=false
> 19:04:51,961 INFO  [org.apache.activemq.artemis.core.server] AMQ221029:
> stopped bridge sf.my-cluster.60b03b78-b58c-11e6-937d-b8a720524153
> 19:04:51,982 INFO  [org.apache.activemq.artemis.core.server] AMQ221037:
> ActiveMQServerImpl::serverUUID=60b03b78-b58c-11e6-937d-b8a720524153 to
> become 'live'
> 19:04:51,983 WARN  [org.apache.activemq.artemis.core.client] AMQ212037:
> Connection failure has been detected: AMQ119015: The connection was
> disconnected because of server shutdown [code=DISCONNECTED]
> ...
> 19:04:52,127 INFO  [org.apache.activemq.artemis.core.server] Scaled down
> 0 messages total.
>
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
> http://activemq.2283324.n4.nabble.com/H-A-colocated-
> replication-environment-not-working-as-expected-tp4719539p4719679.html
> To start a new topic under ActiveMQ - User, email
> ml-node+s2283324n2341805...@n4.nabble.com
> To unsubscribe from ActiveMQ - User, click here
> 
> .
> NAML
> 
>




--
View this message in context: 
http://activemq.2283324.n4.nabble.com/H-A-colocated-replication-environment-not-working-as-expected-tp4719539p4719683.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: H/A colocated replication environment not working as expected

[grishezz]

You can see the entire logs, as well as the configuration I used in the first
post in this thread, I am also copying the relevant parts here:

server0:
=
19:02:06,104 INFO  [org.apache.activemq.artemis.core.server] AMQ221109:
Apache ActiveMQ Artemis Backup Server version 1.5.0 [null] started, waiting
live to fail before it gets active 
19:02:06,920 INFO  [org.apache.activemq.artemis.core.server] AMQ221024:
Backup server
ActiveMQServerImpl::serverUUID=63157c62-b58c-11e6-ace0-b8a720524153 is
synchronized with live-server. 
19:02:06,929 INFO  [org.apache.activemq.artemis.core.server] AMQ221031:
backup announced 

server1:
=
19:02:07,076 INFO  [org.apache.activemq.artemis.core.server] AMQ221109:
Apache ActiveMQ Artemis Backup Server version 1.5.0 [null] started, waiting
live to fail before it gets active 
19:02:08,015 INFO  [org.apache.activemq.artemis.core.server] AMQ221024:
Backup server
ActiveMQServerImpl::serverUUID=60b03b78-b58c-11e6-937d-b8a720524153 is
synchronized with live-server. 
19:02:08,022 INFO  [org.apache.activemq.artemis.core.server] AMQ221031:
backup announced 
...
19:04:51,883 WARN  [org.apache.activemq.artemis.core.client] AMQ212037:
Connection failure has been detected: AMQ119015: The connection was
disconnected because of server shutdown [code=DISCONNECTED] 
19:04:51,885 WARN  [org.apache.activemq.artemis.core.server] AMQ222095:
Connection failed with failedOver=false 
19:04:51,954 WARN  [org.apache.activemq.artemis.core.server] AMQ222095:
Connection failed with failedOver=false 
19:04:51,961 INFO  [org.apache.activemq.artemis.core.server] AMQ221029:
stopped bridge sf.my-cluster.60b03b78-b58c-11e6-937d-b8a720524153 
19:04:51,982 INFO  [org.apache.activemq.artemis.core.server] AMQ221037:
ActiveMQServerImpl::serverUUID=60b03b78-b58c-11e6-937d-b8a720524153 to
become 'live' 
19:04:51,983 WARN  [org.apache.activemq.artemis.core.client] AMQ212037:
Connection failure has been detected: AMQ119015: The connection was
disconnected because of server shutdown [code=DISCONNECTED] 
...
19:04:52,127 INFO  [org.apache.activemq.artemis.core.server] Scaled down 0
messages total.




--
View this message in context: 
http://activemq.2283324.n4.nabble.com/H-A-colocated-replication-environment-not-working-as-expected-tp4719539p4719679.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: H/A colocated replication environment not working as expected

[andytaylor]

couple of questions

1. do you see logs in each broker saying that a backup server has been
started and announced?
2. when you kill the server1 do you see logs in server2 saying that failure
has been detected?



On 1 December 2016 at 14:51, grishezz [via ActiveMQ] <
ml-node+s2283324n4719668...@n4.nabble.com> wrote:

> I did. This example is for shared-storage. My scenario is not fit for
> shared-storage.
>
> Honestly, I am beginning to doubt if it even works at all (perhaps a bug
> was introduced into the network-based replication mechanism at some
> point?). Are there any live systems actually using it with either 1.4 or
> 1.5?
>
> I would love to see a working example of the scenario I described above in
> an integration test for example...
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
> http://activemq.2283324.n4.nabble.com/H-A-colocated-
> replication-environment-not-working-as-expected-tp4719539p4719668.html
> To start a new topic under ActiveMQ - User, email
> ml-node+s2283324n2341805...@n4.nabble.com
> To unsubscribe from ActiveMQ - User, click here
> 
> .
> NAML
> 
>




--
View this message in context: 
http://activemq.2283324.n4.nabble.com/H-A-colocated-replication-environment-not-working-as-expected-tp4719539p4719674.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: H/A colocated replication environment not working as expected

[grishezz]

I did. This example is for shared-storage. My scenario is not fit for
shared-storage.

Honestly, I am beginning to doubt if it even works at all (perhaps a bug was
introduced into the network-based replication mechanism at some point?). Are
there any live systems actually using it with either 1.4 or 1.5?

I would love to see a working example of the scenario I described above in
an integration test for example...



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/H-A-colocated-replication-environment-not-working-as-expected-tp4719539p4719668.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: H/A colocated replication environment not working as expected

[andytaylor]

take a look at
https://github.com/apache/activemq-artemis/tree/master/examples/features/ha/colocated-failover-scale-down

On 30 November 2016 at 08:53, grishezz [via ActiveMQ] <
ml-node+s2283324n4719600...@n4.nabble.com> wrote:

> Well, I see data files are indeed created under the server1 colocated
> backup. And the configuration above includes a  clause. Other
> than that, I am not sure what does it mean to "make sure" - can I look at
> some information/properties via the jmx interface?
>
> As you can see in the attached logs, it says:
> 19:04:52,127 INFO  [org.apache.activemq.artemis.core.server] Scaled down
> 0 messages total.
>
> But why these 100 messages are not scaled down, I do not know.
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
> http://activemq.2283324.n4.nabble.com/H-A-colocated-
> replication-environment-not-working-as-expected-tp4719539p4719600.html
> To start a new topic under ActiveMQ - User, email
> ml-node+s2283324n2341805...@n4.nabble.com
> To unsubscribe from ActiveMQ - User, click here
> 
> .
> NAML
> 
>




--
View this message in context: 
http://activemq.2283324.n4.nabble.com/H-A-colocated-replication-environment-not-working-as-expected-tp4719539p4719601.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: H/A colocated replication environment not working as expected

[grishezz]

Well, I see data files are indeed created under the server1 colocated backup.
And the configuration above includes a  clause. Other than that,
I am not sure what does it mean to "make sure" - can I look at some
information/properties via the jmx interface?

As you can see in the attached logs, it says:
19:04:52,127 INFO  [org.apache.activemq.artemis.core.server] Scaled down 0
messages total. 

But why these 100 messages are not scaled down, I do not know.



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/H-A-colocated-replication-environment-not-working-as-expected-tp4719539p4719600.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: H/A colocated replication environment not working as expected

[andytaylor]

you need to make sure that a backup is created colocated with server1 and
that it scales down.

On 29 November 2016 at 22:09, Justin Bertram-2 [via ActiveMQ] <
ml-node+s2283324n471959...@n4.nabble.com> wrote:

> Sorry for the confusion.  The  option is only
> available for shared-storage.  The fail-over semantics for replication
> don't use that parameter.  Carry on.
>
>
> Justin
>
> - Original Message -
> From: "grishezz" <[hidden email]
> >
> To: [hidden email] 
> Sent: Tuesday, November 29, 2016 4:01:55 PM
> Subject: Re: H/A colocated replication environment not working as expected
>
> I only saw the  option available for shared-storage.
> Can I somehow turn it on for replicated-colocated topology?
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/H-A-colocated-replication-environment-not-working-as-expected-
> tp4719539p4719590.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
> http://activemq.2283324.n4.nabble.com/H-A-colocated-
> replication-environment-not-working-as-expected-tp4719539p4719591.html
> To start a new topic under ActiveMQ - User, email
> ml-node+s2283324n2341805...@n4.nabble.com
> To unsubscribe from ActiveMQ - User, click here
> 
> .
> NAML
> 
>




--
View this message in context: 
http://activemq.2283324.n4.nabble.com/H-A-colocated-replication-environment-not-working-as-expected-tp4719539p4719594.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: H/A colocated replication environment not working as expected

[Justin Bertram]

Sorry for the confusion.  The  option is only available 
for shared-storage.  The fail-over semantics for replication don't use that 
parameter.  Carry on.


Justin

- Original Message -
From: "grishezz" 
To: users@activemq.apache.org
Sent: Tuesday, November 29, 2016 4:01:55 PM
Subject: Re: H/A colocated replication environment not working as expected

I only saw the  option available for shared-storage.
Can I somehow turn it on for replicated-colocated topology?



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/H-A-colocated-replication-environment-not-working-as-expected-tp4719539p4719590.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: H/A colocated replication environment not working as expected

[grishezz]

I only saw the  option available for shared-storage.
Can I somehow turn it on for replicated-colocated topology?



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/H-A-colocated-replication-environment-not-working-as-expected-tp4719539p4719590.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: H/A colocated replication environment not working as expected

[grishezz]

What would be server2 in the topology I described? What configuration / which
actions do I need to take to make the messages stored on server1 (i.e. the
backup server colocated with server1) be available for consumption?



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/H-A-colocated-replication-environment-not-working-as-expected-tp4719539p4719589.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: H/A colocated replication environment not working as expected

[Justin Bertram]

One more quick point...

Stopping a server gracefully (e.g. using CTRL-C) will not trigger fail-over 
unless  is true.  In general I recommend using a method 
like 'kill -9 ' to test fail-over.


Justin

- Original Message -
From: "andytaylor" 
To: users@activemq.apache.org
Sent: Tuesday, November 29, 2016 8:47:36 AM
Subject: Re: H/A colocated replication environment not working as expected

server0 is backed up by the backup server that is colocated with server1,
not server1 itself. Server2 can either be configured to start as an actual
live or scaledown to the live server, if the latter is configured you
should see the message in server1, if not they will be in the backup
started on server1 when it starts

On 29 November 2016 at 14:18, grishezz [via ActiveMQ] <
ml-node+s2283324n471954...@n4.nabble.com> wrote:

> Right. But now I kill server0 (with all 100 messages) before any messages
> were consumed. My expectation is for these messages to "migrate" to the
> other live server (server1), so that they could be consumed from there.
>
> I am saying "migrate", but really at this moment server0 is already dead,
> so what I thought is supposed to happen is for server1 to detect the
> failure of server0 and thus "restore" those 100 messages it was holding as
> a backup and make them available for consumption. Am I mistaken in this
> expectation?
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
> http://activemq.2283324.n4.nabble.com/H-A-colocated-
> replication-environment-not-working-as-expected-tp4719539p4719546.html
> To start a new topic under ActiveMQ - User, email
> ml-node+s2283324n2341805...@n4.nabble.com
> To unsubscribe from ActiveMQ - User, click here
> 
> .
> NAML
> 
>




--
View this message in context: 
http://activemq.2283324.n4.nabble.com/H-A-colocated-replication-environment-not-working-as-expected-tp4719539p4719550.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: H/A colocated replication environment not working as expected

[andytaylor]

server0 is backed up by the backup server that is colocated with server1,
not server1 itself. Server2 can either be configured to start as an actual
live or scaledown to the live server, if the latter is configured you
should see the message in server1, if not they will be in the backup
started on server1 when it starts

On 29 November 2016 at 14:18, grishezz [via ActiveMQ] <
ml-node+s2283324n471954...@n4.nabble.com> wrote:

> Right. But now I kill server0 (with all 100 messages) before any messages
> were consumed. My expectation is for these messages to "migrate" to the
> other live server (server1), so that they could be consumed from there.
>
> I am saying "migrate", but really at this moment server0 is already dead,
> so what I thought is supposed to happen is for server1 to detect the
> failure of server0 and thus "restore" those 100 messages it was holding as
> a backup and make them available for consumption. Am I mistaken in this
> expectation?
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
> http://activemq.2283324.n4.nabble.com/H-A-colocated-
> replication-environment-not-working-as-expected-tp4719539p4719546.html
> To start a new topic under ActiveMQ - User, email
> ml-node+s2283324n2341805...@n4.nabble.com
> To unsubscribe from ActiveMQ - User, click here
> 
> .
> NAML
> 
>




--
View this message in context: 
http://activemq.2283324.n4.nabble.com/H-A-colocated-replication-environment-not-working-as-expected-tp4719539p4719550.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: H/A colocated replication environment not working as expected

[grishezz]

Right. But now I kill server0 (with all 100 messages) before any messages
were consumed. My expectation is for these messages to "migrate" to the
other live server (server1), so that they could be consumed from there. 

I am saying "migrate", but really at this moment server0 is already dead, so
what I thought is supposed to happen is for server1 to detect the failure of
server0 and thus "restore" those 100 messages it was holding as a backup and
make them available for consumption. Am I mistaken in this expectation?



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/H-A-colocated-replication-environment-not-working-as-expected-tp4719539p4719546.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: H/A colocated replication environment not working as expected

[andytaylor]

Then I would expect to see 100 messages at the broker you sent to and none
on the other live broker

On 29 November 2016 at 12:34, grishezz [via ActiveMQ] <
ml-node+s2283324n4719543...@n4.nabble.com> wrote:

> In this scenario message load balancing is disabled.
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
> http://activemq.2283324.n4.nabble.com/H-A-colocated-
> replication-environment-not-working-as-expected-tp4719539p4719543.html
> To start a new topic under ActiveMQ - User, email
> ml-node+s2283324n2341805...@n4.nabble.com
> To unsubscribe from ActiveMQ - User, click here
> 
> .
> NAML
> 
>




--
View this message in context: 
http://activemq.2283324.n4.nabble.com/H-A-colocated-replication-environment-not-working-as-expected-tp4719539p4719544.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: H/A colocated replication environment not working as expected

[grishezz]

In this scenario message load balancing is disabled.



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/H-A-colocated-replication-environment-not-working-as-expected-tp4719539p4719543.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: H/A colocated replication environment not working as expected

[andytaylor]

Im not 100% sure of your topology, however leaving backups out of it I
would expect to see 50 messages on server0 and 50 on server1 if you have
load balancing enabled.

On 29 November 2016 at 09:26, grishezz [via ActiveMQ] <
ml-node+s2283324n4719539...@n4.nabble.com> wrote:

> Hello,
>
> I am trying to set-up the following environment and have ran into
> difficulties:
>
> H/A cluster of 2 servers: server0 & server1 replication=colocated
> master/slave, and message-load-balancing=OFF.
>
> Running the following scenario:
>
> 1) start server0
> 2) start server1
> 3) start consumer0 (connected to server0)
> 4) start consumer1 (connected to server1)
> 5) start producer0 (connected to server0)
> 6) producer0 sends 100 messages to exampleQueue
> 7) consumer0 is blocked on the first message read using debugger
> 8) kill server0
>
> Now, I expected these 100 messages that were not yet acknowledged to
> become available on server1 (as server0 backup is supposed to be there).
> However client1 is receiving no messages (and I can't see them via the jmx
> console either - though here I wasn't sure if they should actually move to
> the live exampleQueue on server1).
>
> Below are the 2 logs and a configuration of one of the servers (the other
> is an exact mirror image). I wonder if the "Connection failed with
> failedOver=false" message might have something to do with the scenario not
> working as expected. I have tried many different config permutations and
> have gone through the documentation several times... I hope I am just
> missing some minor detail.
>
> Help would be greatly appreciated!
>
> 
> server0 log:
> 
> 19:01:58,344 WARN  [org.apache.activemq.artemis.core.server] AMQ222018:
> AIO was not located on this platform, it will fall back to using pure Java
> NIO. If your platform is Linux, install LibAIO to enable the AIO journal
> 19:01:58,399 WARN  [org.apache.activemq.artemis.core.server] AMQ222018:
> AIO was not located on this platform, it will fall back to using pure Java
> NIO. If your platform is Linux, install LibAIO to enable the AIO journal
> 19:01:58,441 INFO  [org.apache.activemq.artemis.integration.bootstrap]
> AMQ101000: Starting ActiveMQ Artemis Server
> 19:02:00,364 INFO  [org.apache.activemq.artemis.core.server] AMQ221000:
> live Message Broker is starting with configuration Broker Configuration
> (clustered=true,journalDirectory=C:\dev_temp\
> apache-artemis-1.5.0\examples\features\ha\mark-poc\target\
> server0\data/server0/data/messaging/journal,bindingsDirectory=C:\dev_temp\
> apache-artemis-1.5.0\examples\features\ha\mark-poc\target\
> server0\data/server0/data/messaging/bindings,
> largeMessagesDirectory=C:\dev_temp\apache-artemis-1.5.0\
> examples\features\ha\mark-poc\target\server0\data/server0/
> data/messaging/largemessages,pagingDirectory=C:\dev_temp\
> apache-artemis-1.5.0\examples\features\ha\mark-poc\target\
> server0\data/server0/data/messaging/paging)
> 19:02:00,391 INFO  [org.apache.activemq.artemis.core.server] AMQ221013:
> Using NIO Journal
> 19:02:00,420 WARN  [org.apache.activemq.artemis.core.server] AMQ222007:
> Security risk! Apache ActiveMQ Artemis is running with the default cluster
> admin user and default password. Please see the cluster chapter in the
> ActiveMQ Artemis User Guide for instructions on how to change this.
> 19:02:00,457 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
> Protocol module found: [artemis-server]. Adding protocol support for: CORE
> 19:02:00,459 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
> Protocol module found: [artemis-amqp-protocol]. Adding protocol support
> for: AMQP
> 19:02:00,461 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
> Protocol module found: [artemis-hornetq-protocol]. Adding protocol support
> for: HORNETQ
> 19:02:00,463 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
> Protocol module found: [artemis-mqtt-protocol]. Adding protocol support
> for: MQTT
> 19:02:00,464 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
> Protocol module found: [artemis-openwire-protocol]. Adding protocol support
> for: OPENWIRE
> 19:02:00,465 INFO  [org.apache.activemq.artemis.core.server] AMQ221043:
> Protocol module found: [artemis-stomp-protocol]. Adding protocol support
> for: STOMP
> 19:02:00,737 INFO  [org.apache.activemq.artemis.core.server] AMQ221003:
> Deploying queue jms.queue.exampleQueue
> 19:02:00,766 WARN  [org.apache.activemq.artemis.core.server] AMQ222165:
> No Dead Letter Address configured for queue jms.queue.exampleQueue in
> AddressSettings
> 19:02:00,767 WARN  [org.apache.activemq.artemis.core.server] AMQ222166:
> No Expiry Address configured for queue jms.queue.exampleQueue in
> AddressSettings
> 19:02:00,998 INFO  [org.apache.activemq.artemis.core.server] AMQ221020:
> Started Acceptor at localhost:61616 for protocols
> [CORE,MQTT,AMQP,HORNETQ,STOMP,OPENWIRE]
> 19:02:00,999 INFO  [org.apache.activemq.artemis.core.server] AMQ221007:
>