Hi, On Mon, May 18, 2009 at 04:21:39PM -0400, Jiann-Ming Su wrote: > On Mon, Apr 27, 2009 at 4:14 PM, Jiann-Ming Su <[email protected]> wrote: > > On Mon, Apr 27, 2009 at 3:51 PM, Jiann-Ming Su <[email protected]> > > wrote: > >> On Sat, Apr 25, 2009 at 8:15 AM, Brandon Allhands <[email protected]> > >> wrote: > >>> Is named set to startup outside of heartbeat? chkconfig --list named? > >>> > >> > >> No, the named initscript is set to off. > >> > >> > > > > In the cib.xml, here's the difference I see: > > > > ? ? ? ? <primitive id="named" class="lsb" type="named" provider="heartbeat"> > > ? ? ? ? ? <operations> > > ? ? ? ? ? ? <op id="a5004e78-2241-4546-8906-02a054b48edf" > > name="monitor" interval="120s" timeout="60s" start_delay="0" > > disabled="false" role="Started"/> > > ? ? ? ? ? </operations> > > ? ? ? ? </primitive> > > ? ? ? ? <primitive class="lsb" id="dhcpd_4" provider="heartbeat" > > type="dhcpd"> > > ? ? ? ? ? <operations> > > ? ? ? ? ? ? <op id="dhcpd_4_mon" interval="120s" name="monitor" > > timeout="60s" start_delay="0" disabled="false" role="Started"/> > > ? ? ? ? ? </operations> > > ? ? ? ? ? <meta_attributes id="dhcpd_4_meta_attrs"> > > ? ? ? ? ? ? <attributes/> > > ? ? ? ? ? </meta_attributes> > > ? ? ? ? ? <instance_attributes id="dhcpd_4"> > > ? ? ? ? ? ? <attributes/> > > ? ? ? ? ? </instance_attributes> > > ? ? ? ? </primitive> > > > > The named instance doesn't have the meta_attributes or > > instance_attributes. ?Also, I added named using the gui, while dhcp > > and the others were added using the haresources2cib.py script. ?Not > > sure if any of that makes a difference on the behavior I've observed. > > Ran into a situation today where the active node in two different > clusters had a problem monitoring named which caused each cluster to > go into a strange state where dhcpd was down and named was reporting > down but was actually up. > > Here's what I was able to find in the messages: > > May 18 14:20:39 dhcp2-admin lrmd: [2893]: WARN: named:monitor process > (PID 27438) timed out (try 1). Killing with signal SIGTERM (15). > May 18 14:20:39 dhcp2-admin lrmd: [2893]: WARN: operation monitor[142] > on lsb::named::named for client 2896, its parameters: > CRM_meta_interval=[120000] CRM_meta_start_delay=[0] CRM_meta_role=[S > tarted] CRM_meta_id=[473ee35e-ead0-49e6-b021-61a0e36d2704] > CRM_meta_timeout=[60000] crm_feature_set=[2.0] > CRM_meta_disabled=[false] CRM_meta_name=[monitor] : pid [27438] timed > out > May 18 14:20:39 dhcp2-admin crmd: [2896]: ERROR: process_lrm_event: > LRM operation named_monitor_120000 (142) Timed Out (timeout=60000ms) > May 18 14:20:40 dhcp2-admin crmd: [2896]: info: do_lrm_rsc_op: > Performing op=lease-parser_5_stop_0 > key=28:1:43858383-d401-49a0-8852-e5e37418c2b9) > May 18 14:20:40 dhcp2-admin lrmd: [2893]: info: rsc:lease-parser_5: stop > May 18 14:20:40 dhcp2-admin lrmd: [27900]: WARN: For LSB init script, > no additional parameters are needed. > May 18 14:20:40 dhcp2-admin crmd: [2896]: info: process_lrm_event: LRM > operation lease-parser_5_monitor_120000 (call=146, rc=-2) Cancelled > May 18 14:20:40 dhcp2-admin lrmd: [2893]: info: RA output: > (lease-parser_5:stop:stdout) Stopping lease-parser: > May 18 14:20:41 dhcp2-admin lrmd: [2893]: info: RA output: > (lease-parser_5:stop:stdout) [ OK > May 18 14:20:41 dhcp2-admin lrmd: [2893]: info: RA output: > (lease-parser_5:stop:stdout) ] > May 18 14:20:41 dhcp2-admin lrmd: [2893]: info: RA output: > (lease-parser_5:stop:stdout) > May 18 14:20:41 dhcp2-admin crmd: [2896]: info: process_lrm_event: LRM > operation lease-parser_5_stop_0 (call=147, rc=0) complete > May 18 14:20:42 dhcp2-admin crmd: [2896]: info: do_lrm_rsc_op: > Performing op=dhcpd_4_stop_0 > key=26:1:43858383-d401-49a0-8852-e5e37418c2b9) > May 18 14:20:42 dhcp2-admin lrmd: [2893]: info: rsc:dhcpd_4: stop > May 18 14:20:42 dhcp2-admin lrmd: [27919]: WARN: For LSB init script, > no additional parameters are needed. > May 18 14:20:42 dhcp2-admin crmd: [2896]: info: process_lrm_event: LRM > operation dhcpd_4_monitor_120000 (call=144, rc=-2) Cancelled > May 18 14:20:42 dhcp2-admin lrmd: [2893]: info: RA output: > (dhcpd_4:stop:stdout) Shutting down dhcpd: > May 18 14:20:47 dhcp2-admin lrmd: [2893]: info: RA output: > (dhcpd_4:stop:stdout) [FAILED] > May 18 14:20:47 dhcp2-admin lrmd: [2893]: info: RA output: > (dhcpd_4:stop:stdout) > May 18 14:20:47 dhcp2-admin crmd: [2896]: ERROR: process_lrm_event: > LRM operation dhcpd_4_stop_0 (call=148, rc=1) Error unknown error > May 18 14:23:39 dhcp2-admin lrmd: [2893]: WARN: named:monitor process > (PID 28698) timed out (try 1). Killing with signal SIGTERM (15). > May 18 14:23:39 dhcp2-admin lrmd: [2893]: WARN: operation monitor[142] > on lsb::named::named for client 2896, its parameters: > CRM_meta_interval=[120000] CRM_meta_start_delay=[0] CRM_meta_role=[S > tarted] CRM_meta_id=[473ee35e-ead0-49e6-b021-61a0e36d2704] > CRM_meta_timeout=[60000] crm_feature_set=[2.0] > CRM_meta_disabled=[false] CRM_meta_name=[monitor] : pid [28698] timed > out > May 18 14:23:39 dhcp2-admin crmd: [2896]: ERROR: process_lrm_event: > LRM operation named_monitor_120000 (142) Timed Out (timeout=60000ms) > May 18 14:26:39 dhcp2-admin lrmd: [2893]: WARN: named:monitor process > (PID 29834) timed out (try 1). Killing with signal SIGTERM (15). > May 18 14:26:39 dhcp2-admin lrmd: [2893]: WARN: operation monitor[142] > on lsb::named::named for client 2896, its parameters: > CRM_meta_interval=[120000] CRM_meta_start_delay=[0] CRM_meta_role=[S > tarted] CRM_meta_id=[473ee35e-ead0-49e6-b021-61a0e36d2704] > CRM_meta_timeout=[60000] crm_feature_set=[2.0] > CRM_meta_disabled=[false] CRM_meta_name=[monitor] : pid [29834] timed > out > May 18 14:26:39 dhcp2-admin crmd: [2896]: ERROR: process_lrm_event: > LRM operation named_monitor_120000 (142) Timed Out (timeout=60000ms) > May 18 14:28:37 dhcp2-admin cib: [2892]: info: cib_stats: Processed 16 > operations (1875.00us average, 0% utilization) in the last 10min > May 18 14:29:39 dhcp2-admin lrmd: [2893]: WARN: named:monitor process > (PID 31059) timed out (try 1). Killing with signal SIGTERM (15). > May 18 14:29:39 dhcp2-admin lrmd: [2893]: WARN: operation monitor[142] > on lsb::named::named for client 2896, its parameters: > CRM_meta_interval=[120000] CRM_meta_start_delay=[0] CRM_meta_role=[S > tarted] CRM_meta_id=[473ee35e-ead0-49e6-b021-61a0e36d2704] > CRM_meta_timeout=[60000] crm_feature_set=[2.0] > CRM_meta_disabled=[false] CRM_meta_name=[monitor] : pid [31059] timed > out > May 18 14:29:39 dhcp2-admin crmd: [2896]: ERROR: process_lrm_event: > LRM operation named_monitor_120000 (142) Timed Out (timeout=60000ms) > May 18 14:30:43 dhcp2-admin lrmd: [2893]: WARN: G_SIG_dispatch: > Dispatch function for SIGCHLD was delayed 1000 ms (> 100 ms) before > being called (GSource: 0x9015368) > May 18 14:30:43 dhcp2-admin lrmd: [2893]: info: G_SIG_dispatch: > started at 901261472 should have started at 901261372 > May 18 14:32:39 dhcp2-admin lrmd: [2893]: WARN: named:monitor process > (PID 32197) timed out (try 1). Killing with signal SIGTERM (15). > May 18 14:32:39 dhcp2-admin lrmd: [2893]: WARN: operation monitor[142] > on lsb::named::named for client 2896, its parameters: > CRM_meta_interval=[120000] CRM_meta_start_delay=[0] CRM_meta_role=[S > tarted] CRM_meta_id=[473ee35e-ead0-49e6-b021-61a0e36d2704] > CRM_meta_timeout=[60000] crm_feature_set=[2.0] > CRM_meta_disabled=[false] CRM_meta_name=[monitor] : pid [32197] timed > out > May 18 14:32:39 dhcp2-admin crmd: [2896]: ERROR: process_lrm_event: > LRM operation named_monitor_120000 (142) Timed Out (timeout=60000ms) > May 18 15:04:42 dhcp2-admin lrmd: [2893]: WARN: G_SIG_dispatch: > Dispatch function for SIGCHLD was delayed 1000 ms (> 100 ms) before > being called (GSource: 0x9015368) > May 18 15:04:42 dhcp2-admin lrmd: [2893]: info: G_SIG_dispatch: > started at 901465342 should have started at 901465242 > May 18 15:10:46 dhcp2-admin lrmd: [2893]: WARN: G_SIG_dispatch: > Dispatch function for SIGCHLD was delayed 1000 ms (> 100 ms) before > being called (GSource: 0x9015368) > May 18 15:10:46 dhcp2-admin lrmd: [2893]: info: G_SIG_dispatch: > started at 901501652 should have started at 901501552 > > And from the other cluster: > > May 18 14:18:12 dhcp2-a lrmd: [2929]: WARN: G_SIG_dispatch: Dispatch > function for SIGCHLD was delayed 1000 ms (> 100 ms) before being > called (GSource: 0x8ad7368) > May 18 14:18:12 dhcp2-a lrmd: [2929]: info: G_SIG_dispatch: started at > 936276860 should have started at 936276760 > May 18 14:18:47 dhcp2-a lrmd: [2929]: WARN: named:monitor process (PID > 2755) timed out (try 1). Killing with signal SIGTERM (15).May 18 > 14:18:47 dhcp2-a lrmd: [2929]: WARN: operation monitor[75] on > lsb::named::named for client 2932, its parameters: > CRM_meta_interval=[120000] CRM_meta_start_delay=[0] > CRM_meta_role=[Started] C > RM_meta_id=[a5004e78-2241-4546-8906-02a054b48edf] > CRM_meta_timeout=[60000] crm_feature_set=[2.0] > CRM_meta_disabled=[false] CRM_meta_name=[monitor] : pid [2755] timed > out > May 18 14:18:47 dhcp2-a crmd: [2932]: ERROR: process_lrm_event: LRM > operation named_monitor_120000 (75) Timed Out (timeout=60000ms) > May 18 14:18:48 dhcp2-a crmd: [2932]: info: do_lrm_rsc_op: Performing > op=lease-parser_5_stop_0 > key=28:179:68fc3118-d505-4de6-b318-d3a9832f1869) > May 18 14:18:48 dhcp2-a lrmd: [2929]: info: rsc:lease-parser_5: stop > May 18 14:18:48 dhcp2-a lrmd: [3255]: WARN: For LSB init script, no > additional parameters are needed. > May 18 14:18:48 dhcp2-a crmd: [2932]: info: process_lrm_event: LRM > operation lease-parser_5_monitor_120000 (call=79, rc=-2) Cancelled > May 18 14:18:48 dhcp2-a lrmd: [2929]: info: RA output: > (lease-parser_5:stop:stdout) Stopping lease-parser: > May 18 14:18:49 dhcp2-a lrmd: [2929]: info: RA output: > (lease-parser_5:stop:stdout) [ > May 18 14:18:49 dhcp2-a lrmd: [2929]: info: RA output: > (lease-parser_5:stop:stdout) OK ] > May 18 14:18:49 dhcp2-a lrmd: [2929]: info: RA output: > (lease-parser_5:stop:stdout) > May 18 14:18:49 dhcp2-a crmd: [2932]: info: process_lrm_event: LRM > operation lease-parser_5_stop_0 (call=80, rc=0) complete > May 18 14:18:50 dhcp2-a crmd: [2932]: info: do_lrm_rsc_op: Performing > op=dhcpd_4_stop_0 key=26:179:68fc3118-d505-4de6-b318-d3a9832f1869) > May 18 14:18:50 dhcp2-a lrmd: [2929]: info: rsc:dhcpd_4: stop > May 18 14:18:50 dhcp2-a lrmd: [3271]: WARN: For LSB init script, no > additional parameters are needed. > May 18 14:18:50 dhcp2-a crmd: [2932]: info: process_lrm_event: LRM > operation dhcpd_4_monitor_120000 (call=77, rc=-2) Cancelled > May 18 14:18:50 dhcp2-a lrmd: [2929]: info: RA output: > (dhcpd_4:stop:stdout) Shutting down dhcpd: > May 18 14:18:54 dhcp2-a lrmd: [2929]: info: RA output: > (dhcpd_4:stop:stdout) [FAILED] > May 18 14:18:54 dhcp2-a lrmd: [2929]: info: RA output: (dhcpd_4:stop:stdout) > May 18 14:18:54 dhcp2-a crmd: [2932]: ERROR: process_lrm_event: LRM > operation dhcpd_4_stop_0 (call=81, rc=1) Error unknown error > May 18 14:20:33 dhcp2-a cib: [2928]: info: cib_stats: Processed 12 > operations (1666.00us average, 0% utilization) in the last 10min > May 18 14:21:47 dhcp2-a lrmd: [2929]: WARN: named:monitor process (PID > 4062) timed out (try 1). Killing with signal SIGTERM (15). > May 18 14:21:47 dhcp2-a lrmd: [2929]: WARN: operation monitor[75] on > lsb::named::named for client 2932, its parameters: > CRM_meta_interval=[120000] CRM_meta_start_delay=[0] > CRM_meta_role=[Started] > CRM_meta_id=[a5004e78-2241-4546-8906-02a054b48edf] > CRM_meta_timeout=[60000] crm_feature_set=[2.0] > CRM_meta_disabled=[false] CRM_meta_name=[monitor] : pid [4062] timed > out > May 18 14:21:47 dhcp2-a crmd: [2932]: ERROR: process_lrm_event: LRM > operation named_monitor_120000 (75) Timed Out (timeout=60000ms) > May 18 14:24:47 dhcp2-a lrmd: [2929]: WARN: named:monitor process (PID > 5201) timed out (try 1). Killing with signal SIGTERM (15). > May 18 14:24:47 dhcp2-a lrmd: [2929]: WARN: operation monitor[75] on > lsb::named::named for client 2932, its parameters: > CRM_meta_interval=[120000] CRM_meta_start_delay=[0] > CRM_meta_role=[Started] C > RM_meta_id=[a5004e78-2241-4546-8906-02a054b48edf] > CRM_meta_timeout=[60000] crm_feature_set=[2.0] > CRM_meta_disabled=[false] CRM_meta_name=[monitor] : pid [5201] timed > outMay 18 14:24:47 dhcp2-a crmd: [2932]: ERROR: process_lrm_event: LRM > operation named_monitor_120000 (75) Timed Out (timeout=60000ms) > May 18 14:27:47 dhcp2-a lrmd: [2929]: WARN: named:monitor process (PID > 6400) timed out (try 1). Killing with signal SIGTERM (15). > May 18 14:27:47 dhcp2-a lrmd: [2929]: WARN: operation monitor[75] on > lsb::named::named for client 2932, its parameters: > CRM_meta_interval=[120000] CRM_meta_start_delay=[0] > CRM_meta_role=[Started] C > RM_meta_id=[a5004e78-2241-4546-8906-02a054b48edf] > CRM_meta_timeout=[60000] crm_feature_set=[2.0] > CRM_meta_disabled=[false] CRM_meta_name=[monitor] : pid [6400] timed > out > May 18 14:27:47 dhcp2-a crmd: [2932]: ERROR: process_lrm_event: LRM > operation named_monitor_120000 (75) Timed Out (timeout=60000ms)May 18 > 14:30:33 dhcp2-a cib: [2928]: info: cib_stats: Processed 6 operations > (5000.00us average, 0% utilization) in the last 10min > > For the time being, I've taken the monitoring function out of the > named process. The only thing I can think of that may have caused > this is I changed iptables and restarted it. Any insights? Should I > consider moving to the 2.99.x branch? Thanks for any help.
The status operation of the LSB named RA timed out, for whatever reason (don't know in which way it is implemented). If you think it's not, then you ran into some really serious bug there. Also, the dhcp stop action failed, that's dangerous, leaves the resource in an unknown state. Make sure that all your RAs behave! If you move to 2.99.x, then you'll also have to move to pacemaker. Not a bad thing, if you can afford it. Otherwise, use at least 2.1.4. Thanks, Dejan > -- > Jiann-Ming Su > "I have to decide between two equally frightening options. > If I wanted to do that, I'd vote." --Duckman > "The system's broke, Hank. The election baby has peed in > the bath water. You got to throw 'em both out." --Dale Gribble > "Those who vote decide nothing. > Those who count the votes decide everything.? --Joseph Stalin > _______________________________________________ > Linux-HA mailing list > [email protected] > http://lists.linux-ha.org/mailman/listinfo/linux-ha > See also: http://linux-ha.org/ReportingProblems _______________________________________________ Linux-HA mailing list [email protected] http://lists.linux-ha.org/mailman/listinfo/linux-ha See also: http://linux-ha.org/ReportingProblems
