Hi,

On Fri, Feb 22, 2008 at 05:10:50AM -0600, Michael Brennen wrote:
> Hello all,
>
> heartbeat 2.1.3, compiled from source, crm mode
>
> I am testing a four node lvs/real server cluster.
>
> lvc3 / lvc4: lvs
> lvc1 / lvc2: real servers
>
> The real servers must mount an iscsi volume, so I've included an iscsi ocf 
> configuration in the two node clone that runs on the real servers.  When I 
> first build the cluster from a totally clean base it comes up okay.  When I 
> fail over the lvs primary for the first time, the iscsi device tries to 
> attach to lvc4, which it is forbidden to do by the location rules, and it 
> errors out. After that it will not start or clean up at all.  I noted the 
> following error in the logs, and I've snipped some of the log errors around 
> it (below).
>
> Feb 22 04:06:38 lvc3 crmd: [24322]: WARN: log_data_element:
>      do_lrm_invoke: Bad command <primitive id="rip_iscsi:0"/>
>
> The cib.xml file is attached, run with -Ql.  I assume this is a bug, or 
> perhaps a configuration error, but I'm not sure where to look. I will 
> supply more information as needed.  Thanks,

Try to clean the configuration a bit with ciblint. It reports
numerous errors. Otherwise, it looks like the cluster got
seriously confused by this CIB. Please generate a report using
hb_report and create a bugzilla if the error persists.

Thanks,

Dejan


>
>    -- Michael
>
>
> Feb 22 04:06:38 lvc3 crmd: [24322]: WARN: log_data_element: do_lrm_invoke: 
> Bad command <rsc_op transition_key="mgmtd-24323">
> Feb 22 04:06:38 lvc3 crmd: [24322]: WARN: log_data_element: do_lrm_invoke: 
> Bad command   <primitive id="rip_iscsi:0"/>
> Feb 22 04:06:38 lvc3 crmd: [24322]: WARN: log_data_element: do_lrm_invoke: 
> Bad command   <attributes crm_feature_set="2.0"/>
> Feb 22 04:06:38 lvc3 crmd: [24322]: WARN: log_data_element: do_lrm_invoke: 
> Bad command </rsc_op>
> Feb 22 04:06:38 lvc3 crmd: [24322]: info: send_direct_ack: ACK'ing resource 
> op rip_iscsi:0_delete_0 from mgmtd-24323: lrm_invoke-lrmd-1203674798-25
> Feb 22 04:06:38 lvc3 crmd: [24322]: WARN: msg_to_op(1173): failed to get 
> the value of field lrm_opstatus from a ha_msg
> Feb 22 04:06:38 lvc3 crmd: [24322]: info: msg_to_op: Message follows:
> Feb 22 04:06:38 lvc3 crmd: [24322]: info: MSG: Dumping message with 22 
> fields
> Feb 22 04:06:38 lvc3 crmd: [24322]: info: MSG[0] : [lrm_t=op]
> Feb 22 04:06:38 lvc3 crmd: [24322]: info: MSG[1] : 
> [lrm_rid=lvc_stonith_child:0]
> Feb 22 04:06:38 lvc3 crmd: [24322]: info: MSG[2] : [lrm_op=start]
> Feb 22 04:06:38 lvc3 crmd: [24322]: info: MSG[3] : [lrm_timeout=420000]
> Feb 22 04:06:38 lvc3 crmd: [24322]: info: MSG[4] : [lrm_interval=0]
> Feb 22 04:06:38 lvc3 crmd: [24322]: info: MSG[5] : [lrm_delay=120000]
> Feb 22 04:06:38 lvc3 crmd: [24322]: info: MSG[6] : [lrm_copyparams=1]
> Feb 22 04:06:38 lvc3 crmd: [24322]: info: MSG[7] : [lrm_t_run=0]
> Feb 22 04:06:38 lvc3 crmd: [24322]: info: MSG[8] : [lrm_t_rcchange=0]
> Feb 22 04:06:38 lvc3 crmd: [24322]: info: MSG[9] : [lrm_exec_time=0]
> Feb 22 04:06:38 lvc3 crmd: [24322]: info: MSG[10] : [lrm_queue_time=0]
> Feb 22 04:06:38 lvc3 crmd: [24322]: info: MSG[11] : [lrm_targetrc=-1]
> Feb 22 04:06:38 lvc3 crmd: [24322]: info: MSG[12] : [lrm_app=crmd]
> Feb 22 04:06:38 lvc3 crmd: [24322]: info: MSG[13] : 
> [lrm_userdata=11:17:90758679-c04d-4606-8c99-ad5360a6c81a]
> Feb 22 04:06:38 lvc3 crmd: [24322]: info: MSG[14] : 
> [(2)lrm_param=0xa06cda0(285 355)]
>
> Feb 22 04:06:38 lvc3 crmd: [24322]: info: MSG: Dumping message with 12 
> fields
> Feb 22 04:06:38 lvc3 crmd: [24322]: info: MSG[0] : 
> [CRM_meta_prereq=nothing]
> Feb 22 04:06:38 lvc3 crmd: [24322]: info: MSG[1] : 
> [CRM_meta_start_delay=120000]
> Feb 22 04:06:39 lvc3 crmd: [24322]: info: MSG[2] : [ipaddr=172.16.100.110]
> Feb 22 04:06:39 lvc3 crmd: [24322]: info: MSG[3] : 
> [CRM_meta_id=lvc_stonith_op_start]
> Feb 22 04:06:39 lvc3 crmd: [24322]: info: MSG[4] : [password=*****]
> Feb 22 04:06:39 lvc3 crmd: [24322]: info: MSG[5] : 
> [CRM_meta_timeout=420000]
> Feb 22 04:06:39 lvc3 crmd: [24322]: info: MSG[6] : [crm_feature_set=2.0]
> Feb 22 04:06:39 lvc3 crmd: [24322]: info: MSG[7] : [login=*******]
> Feb 22 04:06:39 lvc3 crmd: [24322]: info: MSG[8] : [CRM_meta_clone_max=2]
> Feb 22 04:06:39 lvc3 crmd: [24322]: info: MSG[9] : [CRM_meta_name=start]
> Feb 22 04:06:39 lvc3 crmd: [24322]: info: MSG[10] : [CRM_meta_clone=0]
> Feb 22 04:06:39 lvc3 crmd: [24322]: info: MSG[11] : 
> [CRM_meta_clone_node_max=1]
> Feb 22 04:06:39 lvc3 crmd: [24322]: info: MSG[15] : [lrm_callid=25]
> Feb 22 04:06:39 lvc3 crmd: [24322]: info: MSG[16] : [lrm_app=crmd]
> Feb 22 04:06:39 lvc3 crmd: [24322]: info: MSG[17] : [lrm_callid=25]
> Feb 22 04:06:39 lvc3 crmd: [24322]: info: MSG[18] : [lrm_t_run=0]
> Feb 22 04:06:39 lvc3 crmd: [24322]: info: MSG[19] : [lrm_t_rcchange=0]
> Feb 22 04:06:39 lvc3 crmd: [24322]: info: MSG[20] : [lrm_exec_time=0]
> Feb 22 04:06:39 lvc3 crmd: [24322]: info: MSG[21] : [lrm_queue_time=0]
> Feb 22 04:06:39 lvc3 crmd: [24322]: info: do_lrm_invoke: Forcing a local 
> LRM refresh
>
>
> Feb 22 04:06:39 lvc3 lrmd: [24319]: info: RA output: 
> (vip_ldir:monitor:stderr) ldirectord for /etc/ha.d/ldirectord.cf is running 
> with pid: 24537
> Feb 22 04:06:43 lvc3 lrmd: [24319]: info: RA output: 
> (vip_ldir:monitor:stderr) ldirectord for /etc/ha.d/ldirectord.cf is running 
> with pid: 24537
> Feb 22 04:06:44 lvc3 cib: [28749]: info: retrieveCib: Reading cluster 
> configuration from: /var/lib/heartbeat/crm/cib.xml (digest: 
> /var/lib/heartbeat/crm/cib.xml.sig)
> Feb 22 04:06:44 lvc3 cib: [28749]: info: retrieveCib: Reading cluster 
> configuration from: /var/lib/heartbeat/crm/cib.xml (digest: 
> /var/lib/heartbeat/crm/cib.xml.sig)
> Feb 22 04:06:44 lvc3 cib: [28749]: info: retrieveCib: Reading cluster 
> configuration from: /var/lib/heartbeat/crm/cib.xml.last (digest: 
> /var/lib/heartbeat/crm/cib.xml.sig.last)
> Feb 22 04:06:44 lvc3 cib: [28749]: info: write_cib_contents: Wrote version 
> 0.74.1 of the CIB to disk (digest:5ee3a3e8fd9f861923c0e2ca24f85220)
> Feb 22 04:06:44 lvc3 crmd: [24322]: ERROR: do_lrm_invoke: Not creating 
> resource for a delete event: (null)
> Feb 22 04:06:44 lvc3 crmd: [24322]: WARN: log_data_element: do_lrm_invoke: 
> Bad command <rsc_op transition_key="mgmtd-24323">
> Feb 22 04:06:44 lvc3 crmd: [24322]: WARN: log_data_element: do_lrm_invoke: 
> Bad command   <primitive id="rip_iscsi:1"/>
> Feb 22 04:06:44 lvc3 crmd: [24322]: WARN: log_data_element: do_lrm_invoke: 
> Bad command   <attributes crm_feature_set="2.0"/>
> Feb 22 04:06:44 lvc3 crmd: [24322]: WARN: log_data_element: do_lrm_invoke: 
> Bad command </rsc_op>
> Feb 22 04:06:44 lvc3 crmd: [24322]: info: send_direct_ack: ACK'ing resource 
> op rip_iscsi:1_delete_0 from mgmtd-24323: lrm_invoke-lrmd-1203674804-27

>  <cib generated="true" admin_epoch="0" have_quorum="true" ignore_dtd="false" 
> num_peers="4" cib_feature_revision="2.0" ccm_transition="3" 
> dc_uuid="73847c14-18c1-49ec-a26f-9ef33514d43c" epoch="75" num_updates="11">
>    <configuration>
>      <crm_config>
>        <cluster_property_set id="cib-bootstrap-options">
>          <attributes>
>            <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" 
> value="2.1.3-node: 552305612591183b1628baa5bc6e903e0f1e26a3"/>
>            <nvpair id="bootstrap01" name="transition-idle-timeout" 
> value="60"/>
>            <nvpair id="bootstrap02" name="default-resource-stickiness" 
> value="0"/>
>            <nvpair id="bootstrap04" name="stonith-enabled" value="true"/>
>            <nvpair id="bootstrap05" name="stonith-action" value="reboot"/>
>            <nvpair id="bootstrap06" name="symmetric-cluster" value="true"/>
>            <nvpair id="bootstrap07" name="no-quorum-policy" value="ignore"/>
>            <nvpair id="bootstrap08" name="stop-orphan-resources" 
> value="true"/>
>            <nvpair id="bootstrap09" name="stop-orphan-actions" value="true"/>
>            <nvpair id="bootstrap10" name="is-managed-default" value="true"/>
>            <nvpair name="last-lrm-refresh" 
> id="cib-bootstrap-options-last-lrm-refresh" value="1203674804"/>
>          </attributes>
>        </cluster_property_set>
>      </crm_config>
>      <nodes>
>        <node id="73847c14-18c1-49ec-a26f-9ef33514d43c" 
> uname="lvc4.fishnet.us" type="normal"/>
>        <node id="f2a8d1b1-0514-444b-a6f0-244036d27d80" 
> uname="lvc3.fishnet.us" type="normal"/>
>        <node id="a616e3d6-5b5f-45f7-b09a-97766e7ffd0d" 
> uname="lvc2.fishnet.us" type="normal"/>
>        <node id="2ea010dd-5ac7-4f87-a408-6c3b067181fa" 
> uname="lvc1.fishnet.us" type="normal"/>
>      </nodes>
>      <resources>
>        <clone id="lvc_stonith">
>          <instance_attributes id="lvc_stonith_ia">
>            <attributes>
>              <nvpair name="target_role" value="stopped" 
> id="nv_lvc_stonith_tgt"/>
>              <nvpair name="clone_max" value="2" id="nv_lvc_stonith_max"/>
>              <nvpair name="clone_node_max" value="1" 
> id="nv_lvc_stonith_nmax"/>
>            </attributes>
>          </instance_attributes>
>          <primitive id="lvc_stonith_child" class="stonith" type="apcmaster">
>            <operations>
>              <op name="monitor" interval="24h" timeout="60s" prereq="nothing" 
> id="lvc_stonith_op_mon"/>
>              <op name="start" timeout="5m" prereq="nothing" 
> id="lvc_stonith_op_start" start_delay="2m"/>
>            </operations>
>            <instance_attributes id="lvc_stonith_prim_ia">
>              <attributes>
>                <nvpair name="ipaddr" value="172.16.100.110" 
> id="lvc_stonith_prim_ipaddr"/>
>                <nvpair name="login" value="topd3sk" 
> id="lvc_stonith_prim_login"/>
>                <nvpair name="password" value="654" 
> id="lvc_stonith_prim_passwd"/>
>              </attributes>
>            </instance_attributes>
>          </primitive>
>          <meta_attributes id="lvc_stonith-meta-options">
>            <attributes>
>              <nvpair id="lvc_stonith-meta-options-target_role" 
> name="target_role" value="started"/>
>            </attributes>
>          </meta_attributes>
>        </clone>
>        <group id="group_lvs">
>          <meta_attributes id="group_lvs_meta_attrs">
>            <attributes>
>              <nvpair name="target_role" id="group_lvs_metaattr_target_role" 
> value="started"/>
>              <nvpair id="group_lvs_metaattr_ordered" name="ordered" 
> value="true"/>
>              <nvpair id="group_lvs_metaattr_collocated" name="collocated" 
> value="true"/>
>            </attributes>
>          </meta_attributes>
>          <primitive id="vip_ldir" class="lsb" type="ldirectord" 
> provider="heartbeat">
>            <meta_attributes id="vip_ldir_meta_attrs">
>              <attributes>
>                <nvpair id="vip_ldir_metaattr_resource_stickiness" 
> name="resource_stickiness" value="1"/>
>              </attributes>
>            </meta_attributes>
>            <operations>
>              <op id="vip_ldir_op_mon" name="monitor" description="check 
> ldirectord status" interval="3s" timeout="6s" start_delay="15s" 
> disabled="false" role="Started"/>
>            </operations>
>          </primitive>
>          <primitive id="vip_106" class="ocf" type="IPaddr" 
> provider="heartbeat">
>            <instance_attributes id="vip_106_instance_attrs">
>              <attributes>
>                <nvpair id="vip_106_nv_ip" name="ip" value="172.16.100.106"/>
>              </attributes>
>            </instance_attributes>
>            <meta_attributes id="vip_106_meta_attrs">
>              <attributes>
>                <nvpair id="vip_106_metaattr_resource_stickiness" 
> name="resource_stickiness" value="1"/>
>              </attributes>
>            </meta_attributes>
>            <operations>
>              <op id="vip_106_prim_op_mon" name="monitor" description="check 
> virtual ip status" interval="3s" timeout="6s" start_delay="15s"/>
>            </operations>
>          </primitive>
>        </group>
>        <clone id="iscsi_fs">
>          <meta_attributes id="iscsi_fs_meta_attrs">
>            <attributes>
>              <nvpair name="target_role" id="iscsi_fs_metaattr_target_role" 
> value="stopped"/>
>              <nvpair id="iscsi_fs_metaattr_clone_max" name="clone_max" 
> value="2"/>
>              <nvpair id="iscsi_fs_metaattr_clone_node_max" 
> name="clone_node_max" value="1"/>
>            </attributes>
>          </meta_attributes>
>          <primitive id="rip_iscsi" class="ocf" type="iscsi" 
> provider="heartbeat">
>            <instance_attributes id="rip_iscsi_instance_attrs">
>              <attributes>
>                <nvpair id="rip_iscsi_attr_portal" name="portal" 
> value="172.16.100.109"/>
>                <nvpair id="rip_iscsi_attr_target" name="target" 
> value="iqn.2007-12.us.fishnet:storage.lvciscsi"/>
>              </attributes>
>            </instance_attributes>
>            <operations>
>              <op id="rip_iscsi_op_mon" name="monitor" interval="15s" 
> timeout="30s" start_delay="2m" prereq="nothing" disabled="false" 
> role="Started"/>
>              <op id="rip_iscsi_op_status" name="status" interval="1m" 
> timeout="2m" start_delay="1m" disabled="false" role="Started"/>
>              <op id="rip_iscsi_op_start" name="start" timeout="45s" 
> start_delay="5s"/>
>            </operations>
>          </primitive>
>        </clone>
>        <clone id="mount_fs">
>          <meta_attributes id="mount_fs_meta_attrs">
>            <attributes>
>              <nvpair name="target_role" id="mount_fs_metaattr_target_role" 
> value="started"/>
>              <nvpair id="mount_fs_metaattr_clone_max" name="clone_max" 
> value="2"/>
>              <nvpair id="mount_fs_metaattr_clone_node_max" 
> name="clone_node_max" value="1"/>
>            </attributes>
>          </meta_attributes>
>          <primitive id="rip_fs_106" class="ocf" type="Filesystem" 
> provider="heartbeat">
>            <instance_attributes id="rip_fs_106_instance_attrs">
>              <attributes>
>                <nvpair id="rip_fs_106_dev" name="device" value="/dev/sdb1"/>
>                <nvpair id="rip_fs_106_dir" name="directory" 
> value="/var/www/html"/>
>                <nvpair id="rip_fs_106_fs" name="fstype" value="ocfs2"/>
>              </attributes>
>            </instance_attributes>
>            <operations>
>              <op id="rip_fs_106_op_mon" name="monitor" description="monitor 
> fs mount" interval="2s" timeout="4s" start_delay="10s"/>
>            </operations>
>          </primitive>
>        </clone>
>        <clone id="ip_lo_106">
>          <meta_attributes id="ip_lo_106_meta_attrs">
>            <attributes>
>              <nvpair name="target_role" id="ip_lo_106_metaattr_target_role" 
> value="started"/>
>              <nvpair id="ip_lo_106_metaattr_clone_max" name="clone_max" 
> value="2"/>
>              <nvpair id="ip_lo_106_metaattr_clone_node_max" 
> name="clone_node_max" value="1"/>
>            </attributes>
>          </meta_attributes>
>          <primitive id="rip_106" class="ocf" type="IPaddr" 
> provider="heartbeat">
>            <instance_attributes id="rip_106_instance_attrs">
>              <attributes>
>                <nvpair id="rip_106_ia_ip" name="ip" value="172.16.100.106"/>
>                <nvpair id="rip_106_ia_nic" name="nic" value="lo"/>
>                <nvpair id="rip_106_ia_netmask" name="cidr_netmask" 
> value="255.255.255.255"/>
>              </attributes>
>            </instance_attributes>
>            <meta_attributes id="rip_106:0_meta_attrs">
>              <attributes>
>                <nvpair id="rip_106:0_metaattr_is_managed" name="is_managed" 
> value="true"/>
>              </attributes>
>            </meta_attributes>
>            <operations>
>              <op id="rip_106_op_mon" name="monitor" description="check 
> localhost vip" interval="5s" timeout="15s" start_delay="15s"/>
>            </operations>
>          </primitive>
>        </clone>
>        <clone id="real_apache">
>          <meta_attributes id="real_apache_meta_attrs">
>            <attributes>
>              <nvpair name="target_role" id="real_apache_metaattr_target_role" 
> value="started"/>
>              <nvpair id="real_apache_metaattr_clone_max" name="clone_max" 
> value="2"/>
>              <nvpair id="real_apache_metaattr_clone_node_max" 
> name="clone_node_max" value="1"/>
>            </attributes>
>          </meta_attributes>
>          <primitive id="rip_apache" class="lsb" type="httpd" 
> provider="heartbeat">
>            <operations>
>              <op id="rip_apache_op_mon" name="monitor" description="check 
> https status" interval="5s" timeout="15s" start_delay="15s"/>
>              <op id="rip_apache_op_status" name="status" interval="1m" 
> timeout="2m" start_delay="1m" disabled="false" role="Started"/>
>              <op id="rip_apache_op_start" name="start" timeout="45s" 
> start_delay="5s"/>
>            </operations>
>          </primitive>
>        </clone>
>        <clone id="real_smtp">
>          <meta_attributes id="real_smtp_meta_attrs">
>            <attributes>
>              <nvpair name="target_role" id="real_smtp_metaattr_target_role" 
> value="started"/>
>              <nvpair id="real_smtp_metaattr_clone_max" name="clone_max" 
> value="2"/>
>              <nvpair id="real_smtp_metaattr_clone_node_max" 
> name="clone_node_max" value="1"/>
>            </attributes>
>          </meta_attributes>
>          <primitive id="rip_smtp" class="lsb" type="postfix" 
> provider="heartbeat">
>            <operations>
>              <op id="rip_smtp_op_mon" name="monitor" description="check 
> postfix status" interval="5s" timeout="15s" start_delay="15s"/>
>              <op id="rip_smtp_op_start" name="start" timeout="45s" 
> start_delay="5s"/>
>            </operations>
>          </primitive>
>        </clone>
>      </resources>
>      <constraints>
>        <rsc_location id="location_stonith" rsc="lvc_stonith">
>          <rule id="location_stonith_rule" score="-INFINITY" boolean_op="and">
>            <expression attribute="#uname" id="location_stonith_03" 
> operation="ne" value="lvc3.fishnet.us"/>
>            <expression attribute="#uname" id="location_stonith_04" 
> operation="ne" value="lvc4.fishnet.us"/>
>          </rule>
>        </rsc_location>
>        <rsc_location id="location_lvs" rsc="group_lvs">
>          <rule id="location_lvs_rule" score="-INFINITY" boolean_op="and">
>            <expression attribute="#uname" id="location_lvs_03" operation="ne" 
> value="lvc3.fishnet.us"/>
>            <expression attribute="#uname" id="location_lvs_04" operation="ne" 
> value="lvc4.fishnet.us"/>
>          </rule>
>        </rsc_location>
>        <rsc_location id="location_iscsi106" rsc="iscsi_fs">
>          <rule id="prefered_location_iscsi106" score="-INFINITY">
>            <expression attribute="#uname" id="location_iscsi_01" 
> operation="ne" value="lvc1.fishnet.us"/>
>            <expression attribute="#uname" id="location_iscsi_02" 
> operation="ne" value="lvc2.fishnet.us"/>
>          </rule>
>        </rsc_location>
>        <rsc_location id="location_mount106" rsc="mount_fs">
>          <rule id="prefered_location_mount106" score="-INFINITY">
>            <expression attribute="#uname" id="location_mount106_01" 
> operation="ne" value="lvc1.fishnet.us"/>
>            <expression attribute="#uname" id="location_mount106_02" 
> operation="ne" value="lvc2.fishnet.us"/>
>          </rule>
>        </rsc_location>
>        <rsc_location id="location_iplo_106" rsc="ip_lo_106">
>          <rule id="prefered_location_iplo_106" score="-INFINITY">
>            <expression attribute="#uname" id="location_iplo_106_01" 
> operation="ne" value="lvc1.fishnet.us"/>
>            <expression attribute="#uname" id="location_iplo_106_02" 
> operation="ne" value="lvc2.fishnet.us"/>
>          </rule>
>        </rsc_location>
>        <rsc_location id="location_apache_106" rsc="real_apache">
>          <rule id="prefered_location_apache_106" score="-INFINITY">
>            <expression attribute="#uname" id="location_apache_106_01" 
> operation="ne" value="lvc1.fishnet.us"/>
>            <expression attribute="#uname" id="location_apache_106_02" 
> operation="ne" value="lvc2.fishnet.us"/>
>          </rule>
>        </rsc_location>
>        <rsc_location id="location_smtp_106" rsc="real_smtp">
>          <rule id="prefered_location_smtp_106" score="-INFINITY">
>            <expression attribute="#uname" id="location_smtp_106_01" 
> operation="ne" value="lvc1.fishnet.us"/>
>            <expression attribute="#uname" id="location_smtp_106_02" 
> operation="ne" value="lvc2.fishnet.us"/>
>          </rule>
>        </rsc_location>
>        <rsc_order id="order_iscsi_106" from="mount_fs" action="start" 
> type="after" to="iscsi_fs" to_action="start"/>
>        <rsc_order id="order_mount_106" from="ip_lo_106" action="start" 
> type="after" to="mount_fs" to_action="start"/>
>        <rsc_order id="order_ip_106" from="real_apache" action="start" 
> type="after" to="ip_lo_106" to_action="start"/>
>        <rsc_order id="order_ip_apache" from="real_smtp" action="start" 
> type="after" to="ip_lo_106" to_action="start"/>
>        <rsc_order id="order_stonith" from="lvc_stonith" action="start" 
> type="after" to="real_smtp" to_action="start"/>
>        <rsc_colocation id="colocation_iscsi_mount" from="mount_fs" 
> to="iscsi_fs" score="INFINITY"/>
>        <rsc_colocation id="colocation_mount_ip" from="ip_lo_106" 
> to="mount_fs" score="INFINITY"/>
>        <rsc_colocation id="colocation_ip_smtp" from="real_smtp" 
> to="ip_lo_106" score="INFINITY"/>
>        <rsc_colocation id="colocation_ip_apache" from="real_apache" 
> to="ip_lo_106" score="INFINITY"/>
>      </constraints>
>    </configuration>
>    <status>
>      <node_state id="73847c14-18c1-49ec-a26f-9ef33514d43c" 
> uname="lvc4.fishnet.us" crmd="online" crm-debug-origin="do_cib_replaced" 
> shutdown="0" in_ccm="true" ha="active" join="member" expected="member">
>        <transient_attributes id="73847c14-18c1-49ec-a26f-9ef33514d43c">
>          <instance_attributes 
> id="status-73847c14-18c1-49ec-a26f-9ef33514d43c">
>            <attributes>
>              <nvpair 
> id="status-73847c14-18c1-49ec-a26f-9ef33514d43c-fail-count-rip_iscsi:0" 
> name="fail-count-rip_iscsi:0" value="1"/>
>              <nvpair 
> id="status-73847c14-18c1-49ec-a26f-9ef33514d43c-fail-count-rip_iscsi:1" 
> name="fail-count-rip_iscsi:1" value="1"/>
>              <nvpair 
> id="status-73847c14-18c1-49ec-a26f-9ef33514d43c-fail-count-rip_smtp:1" 
> name="fail-count-rip_smtp:1" value="1"/>
>            </attributes>
>          </instance_attributes>
>        </transient_attributes>
>        <lrm id="73847c14-18c1-49ec-a26f-9ef33514d43c">
>          <lrm_resources>
>            <lrm_resource id="lvc_stonith_child:1" type="apcmaster" 
> class="stonith">
>              <lrm_rsc_op id="lvc_stonith_child:1_monitor_0" 
> operation="monitor" crm-debug-origin="build_active_RAs" 
> transition_key="4:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;4:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="3" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="2e7b3e5d4acd2090ca2d7a4fd81771cf"/>
>            </lrm_resource>
>            <lrm_resource id="rip_106:0" type="IPaddr" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_106:0_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="11:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;11:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="10" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="f7e5d836a39c71c9237d0d0409c6a569"/>
>            </lrm_resource>
>            <lrm_resource id="vip_ldir" type="ldirectord" class="lsb" 
> provider="heartbeat">
>              <lrm_rsc_op id="vip_ldir_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="5:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;5:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="4" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>            </lrm_resource>
>            <lrm_resource id="rip_fs_106:0" type="Filesystem" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_fs_106:0_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="9:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;9:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="8" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="b2ee3f038b575748f49cd92c767b78f6"/>
>            </lrm_resource>
>            <lrm_resource id="rip_106:1" type="IPaddr" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_106:1_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="12:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;12:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="11" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="f7e5d836a39c71c9237d0d0409c6a569"/>
>            </lrm_resource>
>            <lrm_resource id="rip_apache:0" type="httpd" class="lsb" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_apache:0_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="13:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;13:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="12" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>            </lrm_resource>
>            <lrm_resource id="rip_fs_106:1" type="Filesystem" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_fs_106:1_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="10:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;10:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="9" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="b2ee3f038b575748f49cd92c767b78f6"/>
>            </lrm_resource>
>            <lrm_resource id="rip_apache:1" type="httpd" class="lsb" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_apache:1_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="14:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;14:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="13" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>            </lrm_resource>
>            <lrm_resource id="rip_smtp:0" type="postfix" class="lsb" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_smtp:0_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="15:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="0:0;15:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="14" 
> crm_feature_set="2.0" rc_code="0" op_status="0" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>              <lrm_rsc_op id="rip_smtp:0_stop_0" operation="stop" 
> crm-debug-origin="build_active_RAs" 
> transition_key="74:1:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="0:0;74:1:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="16" 
> crm_feature_set="2.0" rc_code="0" op_status="0" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>            </lrm_resource>
>            <lrm_resource id="rip_smtp:1" type="postfix" class="lsb" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_smtp:1_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="7:1:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="0:0;7:1:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="15" 
> crm_feature_set="2.0" rc_code="0" op_status="0" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>              <lrm_rsc_op id="rip_smtp:1_stop_0" operation="stop" 
> crm-debug-origin="build_active_RAs" 
> transition_key="68:6:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:1;68:6:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="21" 
> crm_feature_set="2.0" rc_code="1" op_status="4" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>            </lrm_resource>
>            <lrm_resource id="vip_106" type="IPaddr" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="vip_106_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="6:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;6:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="5" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="7d3d9502767a26cef0ccd1ee4fe2f9b8"/>
>            </lrm_resource>
>            <lrm_resource id="lvc_stonith_child:0" type="apcmaster" 
> class="stonith">
>              <lrm_rsc_op id="lvc_stonith_child:0_monitor_0" 
> operation="monitor" crm-debug-origin="build_active_RAs" 
> transition_key="3:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;3:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="2" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="2e7b3e5d4acd2090ca2d7a4fd81771cf"/>
>            </lrm_resource>
>          </lrm_resources>
>        </lrm>
>      </node_state>
>      <node_state uname="lvc2.fishnet.us" crmd="online" in_ccm="true" 
> ha="active" join="member" expected="member" shutdown="0" 
> id="a616e3d6-5b5f-45f7-b09a-97766e7ffd0d">
>        <transient_attributes id="a616e3d6-5b5f-45f7-b09a-97766e7ffd0d">
>          <instance_attributes 
> id="status-a616e3d6-5b5f-45f7-b09a-97766e7ffd0d">
>            <attributes>
>              <nvpair 
> id="status-a616e3d6-5b5f-45f7-b09a-97766e7ffd0d-probe_complete" 
> name="probe_complete" value="true"/>
>            </attributes>
>          </instance_attributes>
>        </transient_attributes>
>        <lrm id="a616e3d6-5b5f-45f7-b09a-97766e7ffd0d">
>          <lrm_resources>
>            <lrm_resource id="lvc_stonith_child:1" type="apcmaster" 
> class="stonith">
>              <lrm_rsc_op id="lvc_stonith_child:1_monitor_0" 
> operation="monitor" crm-debug-origin="build_active_RAs" 
> transition_key="34:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;34:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="3" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="2e7b3e5d4acd2090ca2d7a4fd81771cf"/>
>            </lrm_resource>
>            <lrm_resource id="rip_106:0" type="IPaddr" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_106:0_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="41:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;41:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="10" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="f7e5d836a39c71c9237d0d0409c6a569"/>
>            </lrm_resource>
>            <lrm_resource id="vip_ldir" type="ldirectord" class="lsb" 
> provider="heartbeat">
>              <lrm_rsc_op id="vip_ldir_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="35:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;35:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="4" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>            </lrm_resource>
>            <lrm_resource id="rip_fs_106:0" type="Filesystem" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_fs_106:0_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="39:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;39:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="8" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="b2ee3f038b575748f49cd92c767b78f6"/>
>            </lrm_resource>
>            <lrm_resource id="rip_106:1" type="IPaddr" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_106:1_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="42:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;42:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="11" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="f7e5d836a39c71c9237d0d0409c6a569"/>
>            </lrm_resource>
>            <lrm_resource id="rip_apache:0" type="httpd" class="lsb" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_apache:0_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="43:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="0:0;43:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="12" 
> crm_feature_set="2.0" rc_code="0" op_status="0" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>              <lrm_rsc_op id="rip_apache:0_stop_0" operation="stop" 
> crm-debug-origin="build_active_RAs" 
> transition_key="52:3:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="0:0;52:3:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="17" 
> crm_feature_set="2.0" rc_code="0" op_status="0" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>            </lrm_resource>
>            <lrm_resource id="rip_fs_106:1" type="Filesystem" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_fs_106:1_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="40:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;40:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="9" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="b2ee3f038b575748f49cd92c767b78f6"/>
>            </lrm_resource>
>            <lrm_resource id="rip_apache:1" type="httpd" class="lsb" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_apache:1_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="44:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="0:0;44:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="13" 
> crm_feature_set="2.0" rc_code="0" op_status="0" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>              <lrm_rsc_op id="rip_apache:1_stop_0" operation="stop" 
> crm-debug-origin="build_active_RAs" 
> transition_key="61:1:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="0:0;61:1:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="16" 
> crm_feature_set="2.0" rc_code="0" op_status="0" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>            </lrm_resource>
>            <lrm_resource id="rip_smtp:0" type="postfix" class="lsb" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_smtp:0_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="12:1:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="0:0;12:1:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="14" 
> crm_feature_set="2.0" rc_code="0" op_status="0" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>              <lrm_rsc_op id="rip_smtp:0_stop_0" operation="stop" 
> crm-debug-origin="build_active_RAs" 
> transition_key="63:3:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="0:0;63:3:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="18" 
> crm_feature_set="2.0" rc_code="0" op_status="0" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>            </lrm_resource>
>            <lrm_resource id="rip_smtp:1" type="postfix" class="lsb" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_smtp:1_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="13:1:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="0:0;13:1:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="15" 
> crm_feature_set="2.0" rc_code="0" op_status="0" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>              <lrm_rsc_op id="rip_smtp:1_stop_0" operation="stop" 
> crm-debug-origin="build_active_RAs" 
> transition_key="67:3:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="0:0;67:3:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="19" 
> crm_feature_set="2.0" rc_code="0" op_status="0" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>            </lrm_resource>
>            <lrm_resource id="vip_106" type="IPaddr" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="vip_106_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="36:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;36:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="5" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="7d3d9502767a26cef0ccd1ee4fe2f9b8"/>
>            </lrm_resource>
>            <lrm_resource id="lvc_stonith_child:0" type="apcmaster" 
> class="stonith">
>              <lrm_rsc_op id="lvc_stonith_child:0_monitor_0" 
> operation="monitor" crm-debug-origin="build_active_RAs" 
> transition_key="33:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;33:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="2" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="2e7b3e5d4acd2090ca2d7a4fd81771cf"/>
>            </lrm_resource>
>            <lrm_resource id="rip_iscsi:0" type="iscsi" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_iscsi:0_monitor_0" operation="monitor" 
> crm-debug-origin="do_update_resource" 
> transition_key="11:18:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="0:7;11:18:90758679-c04d-4606-8c99-ad5360a6c81a" 
> call_id="26" crm_feature_set="2.0" rc_code="7" op_status="0" interval="0" 
> op_digest="66a4c310933e82dab6e72f2a3a25f855"/>
>            </lrm_resource>
>            <lrm_resource id="rip_iscsi:1" type="iscsi" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_iscsi:1_monitor_0" operation="monitor" 
> crm-debug-origin="do_update_resource" 
> transition_key="12:18:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="0:7;12:18:90758679-c04d-4606-8c99-ad5360a6c81a" 
> call_id="27" crm_feature_set="2.0" rc_code="7" op_status="0" interval="0" 
> op_digest="66a4c310933e82dab6e72f2a3a25f855"/>
>            </lrm_resource>
>          </lrm_resources>
>        </lrm>
>      </node_state>
>      <node_state uname="lvc1.fishnet.us" crmd="online" in_ccm="true" 
> ha="active" join="member" expected="member" shutdown="0" 
> id="2ea010dd-5ac7-4f87-a408-6c3b067181fa">
>        <transient_attributes id="2ea010dd-5ac7-4f87-a408-6c3b067181fa">
>          <instance_attributes 
> id="status-2ea010dd-5ac7-4f87-a408-6c3b067181fa">
>            <attributes>
>              <nvpair 
> id="status-2ea010dd-5ac7-4f87-a408-6c3b067181fa-probe_complete" 
> name="probe_complete" value="true"/>
>            </attributes>
>          </instance_attributes>
>        </transient_attributes>
>        <lrm id="2ea010dd-5ac7-4f87-a408-6c3b067181fa">
>          <lrm_resources>
>            <lrm_resource id="lvc_stonith_child:1" type="apcmaster" 
> class="stonith">
>              <lrm_rsc_op id="lvc_stonith_child:1_monitor_0" 
> operation="monitor" crm-debug-origin="build_active_RAs" 
> transition_key="49:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;49:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="3" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="2e7b3e5d4acd2090ca2d7a4fd81771cf"/>
>            </lrm_resource>
>            <lrm_resource id="rip_106:0" type="IPaddr" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_106:0_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="56:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;56:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="10" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="f7e5d836a39c71c9237d0d0409c6a569"/>
>            </lrm_resource>
>            <lrm_resource id="vip_ldir" type="ldirectord" class="lsb" 
> provider="heartbeat">
>              <lrm_rsc_op id="vip_ldir_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="50:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;50:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="4" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>            </lrm_resource>
>            <lrm_resource id="rip_fs_106:0" type="Filesystem" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_fs_106:0_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="54:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;54:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="8" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="b2ee3f038b575748f49cd92c767b78f6"/>
>            </lrm_resource>
>            <lrm_resource id="rip_106:1" type="IPaddr" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_106:1_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="57:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;57:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="11" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="f7e5d836a39c71c9237d0d0409c6a569"/>
>            </lrm_resource>
>            <lrm_resource id="rip_apache:0" type="httpd" class="lsb" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_apache:0_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="58:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="0:0;58:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="12" 
> crm_feature_set="2.0" rc_code="0" op_status="0" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>              <lrm_rsc_op id="rip_apache:0_stop_0" operation="stop" 
> crm-debug-origin="build_active_RAs" 
> transition_key="66:1:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="0:0;66:1:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="16" 
> crm_feature_set="2.0" rc_code="0" op_status="0" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>            </lrm_resource>
>            <lrm_resource id="rip_fs_106:1" type="Filesystem" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_fs_106:1_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="55:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;55:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="9" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="b2ee3f038b575748f49cd92c767b78f6"/>
>            </lrm_resource>
>            <lrm_resource id="rip_apache:1" type="httpd" class="lsb" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_apache:1_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="59:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;59:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="13" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>            </lrm_resource>
>            <lrm_resource id="rip_smtp:0" type="postfix" class="lsb" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_smtp:0_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="15:1:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;15:1:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="14" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>            </lrm_resource>
>            <lrm_resource id="rip_smtp:1" type="postfix" class="lsb" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_smtp:1_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="16:1:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;16:1:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="15" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>            </lrm_resource>
>            <lrm_resource id="vip_106" type="IPaddr" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="vip_106_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="51:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;51:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="5" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="7d3d9502767a26cef0ccd1ee4fe2f9b8"/>
>            </lrm_resource>
>            <lrm_resource id="lvc_stonith_child:0" type="apcmaster" 
> class="stonith">
>              <lrm_rsc_op id="lvc_stonith_child:0_monitor_0" 
> operation="monitor" crm-debug-origin="build_active_RAs" 
> transition_key="48:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;48:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="2" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="2e7b3e5d4acd2090ca2d7a4fd81771cf"/>
>            </lrm_resource>
>            <lrm_resource id="rip_iscsi:0" type="iscsi" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_iscsi:0_monitor_0" operation="monitor" 
> crm-debug-origin="do_update_resource" 
> transition_key="14:18:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="0:7;14:18:90758679-c04d-4606-8c99-ad5360a6c81a" 
> call_id="23" crm_feature_set="2.0" rc_code="7" op_status="0" interval="0" 
> op_digest="66a4c310933e82dab6e72f2a3a25f855"/>
>            </lrm_resource>
>            <lrm_resource id="rip_iscsi:1" type="iscsi" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_iscsi:1_monitor_0" operation="monitor" 
> crm-debug-origin="do_update_resource" 
> transition_key="15:18:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="0:7;15:18:90758679-c04d-4606-8c99-ad5360a6c81a" 
> call_id="24" crm_feature_set="2.0" rc_code="7" op_status="0" interval="0" 
> op_digest="66a4c310933e82dab6e72f2a3a25f855"/>
>            </lrm_resource>
>          </lrm_resources>
>        </lrm>
>      </node_state>
>      <node_state uname="lvc3.fishnet.us" crmd="online" in_ccm="true" 
> ha="active" join="member" expected="member" shutdown="0" 
> id="f2a8d1b1-0514-444b-a6f0-244036d27d80">
>        <transient_attributes id="f2a8d1b1-0514-444b-a6f0-244036d27d80">
>          <instance_attributes 
> id="status-f2a8d1b1-0514-444b-a6f0-244036d27d80">
>            <attributes>
>              <nvpair 
> id="status-f2a8d1b1-0514-444b-a6f0-244036d27d80-probe_complete" 
> name="probe_complete" value="true"/>
>            </attributes>
>          </instance_attributes>
>        </transient_attributes>
>        <lrm id="f2a8d1b1-0514-444b-a6f0-244036d27d80">
>          <lrm_resources>
>            <lrm_resource id="lvc_stonith_child:1" type="apcmaster" 
> class="stonith">
>              <lrm_rsc_op id="lvc_stonith_child:1_monitor_0" 
> operation="monitor" crm-debug-origin="build_active_RAs" 
> transition_key="19:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;19:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="3" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="2e7b3e5d4acd2090ca2d7a4fd81771cf"/>
>            </lrm_resource>
>            <lrm_resource id="rip_106:0" type="IPaddr" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_106:0_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="26:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:1;26:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="10" 
> crm_feature_set="2.0" rc_code="1" op_status="4" interval="0" 
> op_digest="f7e5d836a39c71c9237d0d0409c6a569"/>
>            </lrm_resource>
>            <lrm_resource id="vip_ldir" type="ldirectord" class="lsb" 
> provider="heartbeat">
>              <lrm_rsc_op id="vip_ldir_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="20:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;20:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="4" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>              <lrm_rsc_op id="vip_ldir_start_0" operation="start" 
> crm-debug-origin="build_active_RAs" 
> transition_key="26:1:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="0:0;26:1:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="14" 
> crm_feature_set="2.0" rc_code="0" op_status="0" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>              <lrm_rsc_op id="vip_ldir_monitor_3000" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="18:3:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="0:0;18:3:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="17" 
> crm_feature_set="2.0" rc_code="0" op_status="0" interval="3000" 
> op_digest="c79032448cb59ad45a47a15dd4e869d9"/>
>            </lrm_resource>
>            <lrm_resource id="rip_fs_106:0" type="Filesystem" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_fs_106:0_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="24:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;24:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="8" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="b2ee3f038b575748f49cd92c767b78f6"/>
>            </lrm_resource>
>            <lrm_resource id="rip_106:1" type="IPaddr" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_106:1_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="27:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:1;27:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="11" 
> crm_feature_set="2.0" rc_code="1" op_status="4" interval="0" 
> op_digest="f7e5d836a39c71c9237d0d0409c6a569"/>
>            </lrm_resource>
>            <lrm_resource id="rip_apache:0" type="httpd" class="lsb" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_apache:0_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="28:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;28:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="12" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>            </lrm_resource>
>            <lrm_resource id="rip_fs_106:1" type="Filesystem" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_fs_106:1_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="25:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;25:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="9" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="b2ee3f038b575748f49cd92c767b78f6"/>
>            </lrm_resource>
>            <lrm_resource id="rip_apache:1" type="httpd" class="lsb" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_apache:1_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="29:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;29:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="13" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>            </lrm_resource>
>            <lrm_resource id="rip_smtp:0" type="postfix" class="lsb" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_smtp:0_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="9:1:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;9:1:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="15" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>            </lrm_resource>
>            <lrm_resource id="rip_smtp:1" type="postfix" class="lsb" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_smtp:1_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="10:1:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;10:1:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="16" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
>            </lrm_resource>
>            <lrm_resource id="vip_106" type="IPaddr" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="vip_106_monitor_0" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="21:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="0:0;21:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="5" 
> crm_feature_set="2.0" rc_code="0" op_status="0" interval="0" 
> op_digest="7d3d9502767a26cef0ccd1ee4fe2f9b8"/>
>              <lrm_rsc_op id="vip_106_monitor_3000" operation="monitor" 
> crm-debug-origin="build_active_RAs" 
> transition_key="21:3:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="0:0;21:3:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="18" 
> crm_feature_set="2.0" rc_code="0" op_status="0" interval="3000" 
> op_digest="372d8036cca579f3c2f3314373659c7e"/>
>            </lrm_resource>
>            <lrm_resource type="apcmaster" class="stonith" 
> id="lvc_stonith_child:0">
>              <lrm_rsc_op id="lvc_stonith_child:0_monitor_0" 
> operation="monitor" crm-debug-origin="build_active_RAs" 
> transition_key="18:0:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="4:7;18:0:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="2" 
> crm_feature_set="2.0" rc_code="7" op_status="4" interval="0" 
> op_digest="2e7b3e5d4acd2090ca2d7a4fd81771cf"/>
>              <lrm_rsc_op operation="start" 
> transition_key="11:17:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="25" 
> crm_feature_set="2.0" interval="0" 
> op_digest="2e7b3e5d4acd2090ca2d7a4fd81771cf" id="lvc_stonith_child:0_start_0" 
> transition_magic="0:0;11:17:90758679-c04d-4606-8c99-ad5360a6c81a" rc_code="0" 
> op_status="0"/>
>            </lrm_resource>
>            <lrm_resource id="rip_iscsi:0" type="iscsi" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_iscsi:0_monitor_0" operation="monitor" 
> crm-debug-origin="do_update_resource" 
> transition_key="8:18:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="0:7;8:18:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="26" 
> crm_feature_set="2.0" rc_code="7" op_status="0" interval="0" 
> op_digest="66a4c310933e82dab6e72f2a3a25f855"/>
>            </lrm_resource>
>            <lrm_resource id="rip_iscsi:1" type="iscsi" class="ocf" 
> provider="heartbeat">
>              <lrm_rsc_op id="rip_iscsi:1_monitor_0" operation="monitor" 
> crm-debug-origin="do_update_resource" 
> transition_key="9:18:90758679-c04d-4606-8c99-ad5360a6c81a" 
> transition_magic="0:7;9:18:90758679-c04d-4606-8c99-ad5360a6c81a" call_id="27" 
> crm_feature_set="2.0" rc_code="7" op_status="0" interval="0" 
> op_digest="66a4c310933e82dab6e72f2a3a25f855"/>
>            </lrm_resource>
>          </lrm_resources>
>        </lrm>
>      </node_state>
>    </status>
>  </cib>
> 

> _______________________________________________
> Linux-HA mailing list
> [email protected]
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
_______________________________________________
Linux-HA mailing list
[email protected]
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems

Reply via email to