I feared the worst, and the worst was true. Something dumb was keeping this from working. Though I had scoured the iptables configs on both boxen and testing multiple protocols, it seems that an absolute allow on server C fixed my issue. Sorry for the time wasted.
Aaron On Wed, Aug 4, 2010 at 10:19 AM, Aaron Cline <[email protected]> wrote: > So this isn't a 3 node problem, its just a Server C won't join with Server > A or Server B problem. I've change my testing to the action that I'm simply > trying to make a cluster with Server A (tlb1) and Server C (tlb111). > > I have debug logs from servers A and C and I'm including them as > attachments. They are quite large. > > Process, I removed everything using the uninstall and cleanup steps that I > included in a previous email. I then installed heartbeat with the config > outlined in the first email. I started server A, waited until the logs > pengine seemed to be all the way started, turned off stonith via: > > crm configure property stonith-enabled=false > > I then started heartbeat on tlb111. At about 09:48:36 you can see the > WARNings that tlb111 is not in the membership list in the tlb1 log. tlb111 > then establishes itself as its own DC in its own cluster. > > Please help. This is so frustrating. I feel like the answer is going to > be some idiotic on my part, but I sure can't figure out what that idiocy > might be. > > If I should post the logs into the email instead of as attachments, I will > gladly do that. I wasn't sure of the correct protocol here. > > Thanks. > > Aaron > > > On Wed, Aug 4, 2010 at 3:12 AM, Andrew Beekhof <[email protected]> wrote: > >> Nowhere near enough logs included to make any informed comment on what >> the problem might be. >> >> On Tue, Aug 3, 2010 at 11:08 PM, Aaron Cline <[email protected]> >> wrote: >> > Hi all: >> > >> > I'm trying to setup a 3 member cluster to do HTTP load balancing. The >> > cluster members are in a public cloud where I can't use multi-cast, so >> I'm >> > using heartbeat instead of corosync. I can get server A and server B >> into >> > the cluster fine. They seem to see each other fine and crm_mon shows >> them >> > both online. When I try to add server C, I eventually get the log >> messages >> > below indicating that server C is not going to be allowed into the >> cluster. >> > Then crm_mon will either not show server C at all, or will show it as >> > offline. >> > >> > So, here are my software versions: >> > >> > corosynclib-1.2.7-1.1.el5 >> > corosync-1.2.7-1.1.el5 >> > resource-agents-1.0.3-2.6.el5 >> > pacemaker-1.0.9.1-1.15.el5 >> > heartbeat-libs-3.0.3-2.3.el5 >> > heartbeat-3.0.3-2.3.el5 >> > pacemaker-libs-1.0.9.1-1.15.el5 >> > >> > I am running on CentOS 5.5. >> > >> > Here is my ha.cf: >> > >> > logfacility daemon >> > keepalive 2 >> > deadtime 20 >> > warntime 6 >> > initdead 40 >> > udpport 694 >> > ucast eth1 10.0.0.1 # The Private IP address of your MASTER server. >> > ucast eth1 10.0.0.2 # The Private IP address of your MASTER server. >> > autojoin any >> > node tlb1.sojern.com >> > node tlb11.sojern.com >> > node tlb111.sojern.com >> > crm yes >> > use_logd false >> > debug 1 >> > traditional_compression off >> > compression bz2 >> > coredumps true >> > >> > Here's my current 'crm configuration show' >> > >> > node $id="6e53b1ed-b0ad-4adf-9bdf-3dd3cd3c99c6" tlb11.sojern.com >> > node $id="758f95e1-521f-419e-aadc-55dd752e06cd" tlb1.sojern.com >> > node $id="f3946321-af3a-4433-ba87-c79744598e88" tlb111.sojern.com >> > property $id="cib-bootstrap-options" \ >> > dc-version="1.0.9-89bd754939df5150de7cd76835f98fe90851b677" \ >> > cluster-infrastructure="Heartbeat" \ >> > stonith-enabled="false" \ >> > size="3" >> > >> > >> > And here are what I believe are the pertinent log messages ending in the >> > "Ignoring HA message" from server C (tlb111) >> > >> > Aug 3 15:53:19 tlb11 crmd: [2820]: notice: crmd_ha_status_callback: >> Status >> > update: Node tlb111.sojern.com now has status [active] (DC=true) >> > Aug 3 15:53:20 tlb11 cib: [2816]: info: cib_client_status_callback: >> Status >> > update: Client tlb111.sojern.com/cib now has status [join] >> > Aug 3 15:53:20 tlb11 cib: [2816]: info: crm_new_peer: Node 0 is now >> known >> > as tlb111.sojern.com >> > Aug 3 15:53:20 tlb11 cib: [2816]: info: crm_update_peer_proc: >> > tlb111.sojern.com.cib is now online >> > Aug 3 15:53:27 tlb11 heartbeat: [2805]: WARN: 1 lost packet(s) for [ >> > tlb111.sojern.com] [45:47] >> > Aug 3 15:53:27 tlb11 heartbeat: [2805]: info: No pkts missing from >> > tlb111.sojern.com! >> > Aug 3 15:53:29 tlb11 crmd: [2820]: notice: crmd_client_status_callback: >> > Status update: Client tlb111.sojern.com/crmd now has status [online] >> > (DC=true) >> > Aug 3 15:53:29 tlb11 crmd: [2820]: info: crm_update_peer_proc: >> > tlb111.sojern.com.crmd is now online >> > Aug 3 15:53:29 tlb11 heartbeat: [2805]: WARN: 1 lost packet(s) for [ >> > tlb111.sojern.com] [49:51] >> > Aug 3 15:53:29 tlb11 heartbeat: [2805]: info: No pkts missing from >> > tlb111.sojern.com! >> > Aug 3 15:53:31 tlb11 crmd: [2820]: WARN: crmd_ha_msg_callback: Ignoring >> HA >> > message (op=join_announce) from tlb111.sojern.com: not in our >> membership >> > list (size=2) >> > >> > I asked about this problem in the linux-ha and someone suggested that >> the >> > problem might be with ccm or how heartbeat is configured. >> > >> > Thanks for any help. I'm pretty frustrated at this point. :) >> > >> > Aaron >> > _______________________________________________ >> > Linux-HA mailing list >> > [email protected] >> > http://lists.linux-ha.org/mailman/listinfo/linux-ha >> > See also: http://linux-ha.org/ReportingProblems >> > >> _______________________________________________ >> Linux-HA mailing list >> [email protected] >> http://lists.linux-ha.org/mailman/listinfo/linux-ha >> See also: http://linux-ha.org/ReportingProblems >> > > _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
