Hi Hari, Thanks for the very detailed report! It looks like you are doing the right things so I'm not sure why it is not working.
Try adding: log-enable=debug+:HA in your qpidd.conf. That may give a clue, or send me the log files & I'll take a look. I see you're using qpid 0.18, can you try it with the latest release 0.26? There has been a lot of work on the HA module. Note that in the latest release your ha-brokers-url needs to be a list of the real broker addresses rather than a VIP, in your case: ha-brokers-url=192.168.10.125,192.168.10.198,192.168.10.218 Thanks, Alan. On Tue, 2014-04-08 at 18:08 -0400, Hari Pyla wrote: > Hi, > I am trying to replicate a queue in an active-passive qpid C++ broker > HA setup. > I was wondering if I am missing something or if my configuration is > incorrect. > > ===queue creation=== > The ha-public-url is 192.168.10.195:5672. > [root@guest-1 ~]# qpid-config -b 192.168.10.195:5672 add queue bees > --replicate all > > [root@guest-1 ~]# qpid-stat -b 192.168.10.195:5672 -q > Queues > queue dur autoDel excl msg > msgIn msgOut bytes bytesIn bytesOut cons bind > ==================================================================== pro > 16e18112-8864-4e82-996a-d7cdf6f6fbf0# Y Y 0 > 1 1 0 541 541 1 1 > 305a62dc-0777-4744-b6bc-cb22fa00f0bd# Y Y 0 > 1 1 0 541 541 1 1 > 3779d617-4261-4c22-84c0-4ceb85b52cdf# Y Y 0 > 1 1 0 541 541 1 1 > 67784f9f-1757-41f1-96e1-f4fe852538b1:0.0 Y Y 0 > 0 0 0 0 0 1 2 > bees 0 0 0 0 0 0 0 1 > > I was hoping that after this point I should be able to see the queue > 'bees' to be present on all 3 nodes > in my cluster, however, it is present on only the active node (guest-3), > So, I was wondering as to \ > how to replicate a queue accross the passive nodes as well. > > As you can see: > 192.168.10.125 guest-1 //passive node > 192.168.10.198 guest-2 //passive node > 192.168.10.218 guest-3 //active node > > [root@guest-1 ~]# qpid-stat -b 192.168.10.125:5672 -q --ha-admin > Queues > queue dur autoDel excl msg > msgIn msgOut bytes bytesIn bytesOut cons bind > ==================================================================== > > [root@guest-1 ~]# qpid-stat -b 192.168.10.198:5672 -q --ha-admin > Queues > queue dur autoDel excl msg > msgIn msgOut bytes bytesIn bytesOut cons bind > ==================================================================== > 72d6ffe4-0f06-43b8-b975-faa6bc10a31a:0.0 Y Y 0 > 0 0 0 0 0 1 2 > > [root@guest-1 ~]# qpid-stat -b 192.168.10.218:5672 -q --ha-admin > Queues > queue dur autoDel excl msg > msgIn msgOut bytes bytesIn bytesOut cons bind > ==================================================================== > 16e18112-8864-4e82-996a-d7cdf6f6fbf0# Y Y 0 > 1 1 0 541 541 1 1 > 305a62dc-0777-4744-b6bc-cb22fa00f0bd# Y Y 0 > 1 1 0 541 541 1 1 > 3779d617-4261-4c22-84c0-4ceb85b52cdf# Y Y 0 > 1 1 0 541 541 1 1 > 4f56854b-8355-4228-a6dc-85f8fbbdd531:0.0 Y Y 0 > 0 0 0 0 0 1 2 > bees 0 0 0 0 0 0 0 1 > > Second, in my setup I also have a client (qpid spout) that constantly > writes messages to the queue (bees). > While this is happening in the backdrop, I terminated the active broker. > I can see rgmanager in action. > It restores the cluster by restarting the qpidd service and promotes a > qpidd-primary. > However, upon terminating the active broker, the original queue (bees), > now no longer exists > and spout emits the following error message. > > qpid.messaging.exceptions.NotFound: no such queue: bees > Traceback (most recent call last): > File "/usr/share/doc/python-qpid-0.18/examples/api/spout", line 101, > in <module> > snd = ssn.sender(addr) > File "<string>", line 6, in sender > File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", > line 600, in sender > raise e > > I am guessing this is because the queue replication did not happen in > the first place. > I am trying to setup replication of queues such that in the event the > primary broker were to go down > then one of the backup brokers still has the state (queues, bindings, > messages etc.,) and can resume > execution whenever it becomes the primary. > > I also tried replicating with the HA-module as described in > (http://qpid.apache.org/releases/qpid-0.18/books/AMQP-Messaging-Broker-CPP-Book/html/ch01s14.html) > I still have the same issue. Any ideas?. > > ===configuration=== > i) Qpid version: 0.18-20 > > ii) /etc/qpidd.conf file is follows > cluster-mechanism=DIGEST-MD5 ANONYMOUS > acl-file=/etc/qpid/qpidd.acl > auth=no > load-module=/usr/lib64/qpid/daemon/ssl.so > load-module=/usr/lib64/qpid/daemon/ha.so > no-data-dir=yes > log-to-stderr=no > log-enable=debug+ > log-to-syslog=yes > > ha-cluster=yes > ha-brokers-url=192.168.10.194 > ha-public-url=192.168.10.195 > ha-replicate=all > ha-queue-replication=yes > > iii) cluster status is as follows: > [root@guest-1 ~]# clustat > [root@guest-1 ~]# clustat > Cluster Status for test_cluster @ Tue Apr 8 17:03:41 2014 > Member Status: Quorate > > Member Name ID Status > ------ ---- ---- ------ > guest-1 1 Online, Local, rgmanager > guest-2 2 Online, rgmanager > guest-3 3 Online, rgmanager > > Service Name Owner (Last) State > ------- ---- ----- ------ ----- > service:guest-1-qpidd-service guest-1 started > service:guest-2-qpidd-service guest-2 started > service:guest-3-qpidd-service guest-3 started > service:qpidd-primary-service guest-3 started > > //active node > [root@guest-3 ~]# qpid-ha query > Status: active > Brokers URL: amqp:tcp:192.168.10.194:5672 > Public URL: amqp:tcp:192.168.10.195:5672 > Expected Backups: None > Replicate: all > > //backup node > [root@guest-1 ~]# qpid-ha query > Status: joining > Brokers URL: amqp:tcp:192.168.10.194:5672 > Public URL: amqp:tcp:192.168.10.195:5672 > Expected Backups: None > Replicate: all > > //backup node > [root@guest-2 ~]# qpid-ha query > Status: joining > Brokers URL: amqp:tcp:192.168.10.194:5672 > Public URL: amqp:tcp:192.168.10.195:5672 > Expected Backups: None > Replicate: all > > iv) /etc/cluster/cluster.conf file is as follows > <?xml version="1.0"?> > <cluster config_version="25" name="test_cluster"> > <logging debug="on"/> > <!-- cluster configuration --> > <clusternodes> > <clusternode name="guest-1" nodeid="1"/> > <clusternode name="guest-2" nodeid="2"/> > <clusternode name="guest-3" nodeid="3"/> > </clusternodes> > <!-- resource manager configuration --> > <rm> > <!-- failoverdomain configuration --> > <failoverdomains> > <failoverdomain name="guest-1-domain" restricted="1"> > <failoverdomainnode name="guest-1"/> > </failoverdomain> > <failoverdomain name="guest-2-domain" restricted="1"> > <failoverdomainnode name="guest-2"/> > </failoverdomain> > <failoverdomain name="guest-3-domain" restricted="1"> > <failoverdomainnode name="guest-3"/> > </failoverdomain> > </failoverdomains> > <!-- resources configuration --> > <resources> > <!-- start a qpidd broker acting as a backup --> > <script file="/etc/init.d/qpidd" name="qpidd"/> > <!-- promote the qpidd broker on this node to primary --> > <script file="/etc/init.d/qpidd-primary" name="qpidd-primary"/> > <!-- assign a virtual IP address for broker replication > traffic --> > <ip address="192.168.10.194" monitor_link="1"/> > <!-- assign a virtual IP address for qpid client traffic --> > <ip address="192.168.10.195" monitor_link="1"/> > </resources> > <!-- service configuration --> > <!-- There is a qpidd service on each node, it should be > restarted if it fails --> > <service domain="guest-1-domain" name="guest-1-qpidd-service" > recovery="restart"> > <script ref="qpidd"/> > </service> > <service domain="guest-2-domain" name="guest-2-qpidd-service" > recovery="restart"> > <script ref="qpidd"/> > </service> > <service domain="guest-3-domain" name="guest-3-qpidd-service" > recovery="restart"> > <script ref="qpidd"/> > </service> > <!-- There should always be a single qpidd-primary service, it > can run on any node --> > <service autostart="1" exclusive="0" > name="qpidd-primary-service" recovery="relocate"> > <script ref="qpidd-primary"/> > <!-- The primary has the IP addresses for brokers and > clients to connect --> > <ip ref="192.168.10.194"/> > <ip ref="192.168.10.195"/> > </service> > </rm> > </cluster> > > Thanks, > --Hari > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
