Hi, I installed Solaris Cluster Express 12/08 on 2 pvm virtual nodes (on 2 physical SXCE104 machines).
Each virtual node has 3 NICs (3 VNICs from xVM dom0), but each physical node has only one physical NIC connected to the one physical switch. I know about security issues and such... There is a problem with interconnects so I can't create the cluster. Any idea? :) [root at node1 ~]# /usr/cluster/bin/clnode status -v === Cluster Nodes === --- Node Status --- Node Name Status --------- ------ node1 Online node2 Offline --- Node IPMP Group Status --- Node Name Group Name Status Adapter Status --------- ---------- ------ ------- ------ node1 sc_ipmp0 Online xnf0 Online [root at node1 ~]# /usr/cluster/bin/clintr show === Transport Cables === Transport Cable: node1:xnf1,switch1 at 1 Endpoint1: node1:xnf1 Endpoint2: switch1 at 1 State: Enabled Transport Cable: node1:xnf2,switch2 at 1 Endpoint1: node1:xnf2 Endpoint2: switch2 at 1 State: Enabled Transport Cable: node2:xnf1,switch1 at 2 Endpoint1: node2:xnf1 Endpoint2: switch1 at 2 State: Enabled Transport Cable: node2:xnf2,switch2 at 2 Endpoint1: node2:xnf2 Endpoint2: switch2 at 2 State: Enabled === Transport Switches === Transport Switch: switch1 State: Enabled Type: switch Port Names: 1 2 Port State(1): Enabled Port State(2): Enabled Transport Switch: switch2 State: Enabled Type: switch Port Names: 1 2 Port State(1): Enabled Port State(2): Enabled --- Transport Adapters for node1 --- Transport Adapter: xnf1 State: Enabled Transport Type: dlpi device_name: xnf device_instance: 1 lazy_free: 1 dlpi_heartbeat_timeout: 10000 dlpi_heartbeat_quantum: 1000 nw_bandwidth: 80 bandwidth: 70 ip_address: 172.16.0.129 netmask: 255.255.255.128 Port Names: 0 Port State(0): Enabled Transport Adapter: xnf2 State: Enabled Transport Type: dlpi device_name: xnf device_instance: 2 lazy_free: 1 dlpi_heartbeat_timeout: 10000 dlpi_heartbeat_quantum: 1000 nw_bandwidth: 80 bandwidth: 70 ip_address: 172.16.1.1 netmask: 255.255.255.128 Port Names: 0 Port State(0): Enabled --- Transport Adapters for node2 --- Transport Adapter: xnf1 State: Enabled Transport Type: dlpi device_name: xnf device_instance: 1 lazy_free: 1 dlpi_heartbeat_timeout: 10000 dlpi_heartbeat_quantum: 1000 nw_bandwidth: 80 bandwidth: 70 ip_address: 172.16.0.130 netmask: 255.255.255.128 Port Names: 0 Port State(0): Enabled Transport Adapter: xnf2 State: Enabled Transport Type: dlpi device_name: xnf device_instance: 2 lazy_free: 1 dlpi_heartbeat_timeout: 10000 dlpi_heartbeat_quantum: 1000 nw_bandwidth: 80 bandwidth: 70 ip_address: 172.16.1.2 netmask: 255.255.255.128 Port Names: 0 Port State(0): Enabled [root at node1 ~]# /usr/cluster/bin/clintr status -v === Cluster Transport Paths === Endpoint1 Endpoint2 Status --------- --------- ------ node1:xnf2 node2:xnf2 faulted node1:xnf1 node2:xnf1 faulted Jan 10 11:22:14 node1 genunix: [ID 965873 kern.notice] NOTICE: CMM: Node node1 (nodeid = 1) with votecount = 1 added. Jan 10 11:22:14 node1 genunix: [ID 843983 kern.notice] NOTICE: CMM: Node node1: attempting to join cluster. Jan 10 11:22:14 node1 genunix: [ID 525628 kern.notice] NOTICE: CMM: Cluster has reached quorum. Jan 10 11:22:14 node1 genunix: [ID 377347 kern.notice] NOTICE: CMM: Node node1 (nodeid = 1) is up; new incarnation number = 1231582933. Jan 10 11:22:14 node1 genunix: [ID 108990 kern.notice] NOTICE: CMM: Cluster members: node1. Jan 10 11:22:14 node1 genunix: [ID 279084 kern.notice] NOTICE: CMM: node reconfiguration #1 completed. Jan 10 11:22:17 node1 genunix: [ID 499756 kern.notice] NOTICE: CMM: Node node1: joined cluster. Jan 10 11:22:17 node1 ip: [ID 856290 kern.notice] ip: joining multicasts failed (18) on clprivnet0 - will use link layer broadcasts for multicast Jan 10 11:22:28 node1 Cluster.CCR: [ID 914260 daemon.warning] Failed to retrieve global fencing status from the global name server Jan 10 11:22:28 node1 last message repeated 1 time Jan 10 11:22:48 node1 Cluster.CCR: [ID 409585 daemon.error] /usr/cluster/bin/scgdevs: Cannot register devices as HA. Jan 10 11:22:53 node1 xntpd[909]: [ID 702911 daemon.notice] xntpd 3-5.93e+sun 03/08/29 16:23:05 (1.4) Jan 10 11:22:53 node1 xntpd[909]: [ID 301315 daemon.notice] tickadj = 5, tick = 10000, tvu_maxslew = 495, est. hz = 100 Jan 10 11:22:53 node1 xntpd[909]: [ID 266339 daemon.notice] using kernel phase-lock loop 0041, drift correction 0.00000 Jan 10 11:22:53 node1 last message repeated 1 time Jan 10 11:23:00 node1 : [ID 386282 daemon.error] ccr_initialize failure Jan 10 11:23:04 node1 last message repeated 8 times Jan 10 11:23:04 node1 svc.startd[8]: [ID 748625 daemon.error] system/cluster/scdpm:default failed repeatedly: transitioned to maintenance (see 'svcs -xv' for details) Jan 10 11:24:01 node1 xpvd: [ID 395608 kern.info] xenbus at 0, xenbus0 Jan 10 11:24:01 node1 genunix: [ID 936769 kern.info] xenbus0 is /xpvd/xenbus at 0 Jan 10 11:28:15 node1 genunix: [ID 965873 kern.notice] NOTICE: CMM: Node node2 (nodeid = 2) with votecount = 0 added. Jan 10 11:28:15 node1 genunix: [ID 108990 kern.notice] NOTICE: CMM: Cluster members: node1. Jan 10 11:28:15 node1 genunix: [ID 279084 kern.notice] NOTICE: CMM: node reconfiguration #2 completed. Jan 10 11:28:16 node1 genunix: [ID 884114 kern.notice] NOTICE: clcomm: Adapter xnf1 constructed Jan 10 11:28:16 node1 ip: [ID 856290 kern.notice] ip: joining multicasts failed (18) on clprivnet0 - will use link layer broadcasts for multicast Jan 10 11:28:16 node1 genunix: [ID 884114 kern.notice] NOTICE: clcomm: Adapter xnf2 constructed Jan 10 11:28:25 node1 rpc_scadmd[1196]: [ID 801593 daemon.notice] stdout: Jan 10 11:28:25 node1 rpc_scadmd[1196]: [ID 801593 daemon.notice] stderr: Jan 10 11:28:26 node1 rpc_scadmd[1196]: [ID 801593 daemon.notice] stdout: Jan 10 11:28:26 node1 rpc_scadmd[1196]: [ID 801593 daemon.notice] stderr: Jan 10 11:29:16 node1 genunix: [ID 604153 kern.notice] NOTICE: clcomm: Path node1:xnf1 - node2:xnf1 errors during initiation Jan 10 11:29:16 node1 genunix: [ID 618107 kern.warning] WARNING: Path node1:xnf1 - node2:xnf1 initiation encountered errors, errno = 62. Remote node may be down or unreachable through this path. Jan 10 11:29:16 node1 genunix: [ID 604153 kern.notice] NOTICE: clcomm: Path node1:xnf2 - node2:xnf2 errors during initiation Jan 10 11:29:16 node1 genunix: [ID 618107 kern.warning] WARNING: Path node1:xnf2 - node2:xnf2 initiation encountered errors, errno = 62. Remote node may be down or unreachable through this path. Jan 10 11:30:24 node1 genunix: [ID 537175 kern.notice] NOTICE: CMM: Node node2 (nodeid: 2, incarnation #: 1231583261) has become reachable. Jan 10 11:30:24 node1 xnf: [ID 601036 kern.warning] WARNING: xnf2: oversized packet (1518 bytes) dropped Jan 10 11:30:24 node1 last message repeated 1 time Jan 10 11:30:24 node1 genunix: [ID 387288 kern.notice] NOTICE: clcomm: Path node1:xnf2 - node2:xnf2 online Jan 10 11:30:24 node1 genunix: [ID 387288 kern.notice] NOTICE: clcomm: Path node1:xnf1 - node2:xnf1 online Jan 10 11:30:28 node1 xnf: [ID 601036 kern.warning] WARNING: xnf2: oversized packet (1518 bytes) dropped Jan 10 11:30:54 node1 last message repeated 2 times Jan 10 11:30:54 node1 xnf: [ID 601036 kern.warning] WARNING: xnf1: oversized packet (1518 bytes) dropped Jan 10 11:31:08 node1 last message repeated 2 times Jan 10 11:31:28 node1 xnf: [ID 601036 kern.warning] WARNING: xnf2: oversized packet (1518 bytes) dropped Jan 10 11:32:28 node1 last message repeated 1 time Jan 10 11:33:29 node1 genunix: [ID 489438 kern.notice] NOTICE: clcomm: Path node1:xnf2 - node2:xnf2 being drained Jan 10 11:33:29 node1 genunix: [ID 387288 kern.notice] NOTICE: clcomm: Path node1:xnf2 - node2:xnf2 online [root at node1 ~]# uname -a SunOS node1 5.11 snv_101a i86pc i386 i86xpv # xm start -c node2 v3.1.4-xvm chgset 'Mon Nov 24 22:48:21 2008 -0800 15909:8ac8abf844b5' SunOS Release 5.11 Version snv_101a 64-bit Copyright 1983-2008 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Hostname: node2 Configuring devices. /usr/cluster/bin/scdidadm: Could not load DID instance list. /usr/cluster/bin/scdidadm: Cannot open /etc/cluster/ccr/global/did_instances. Booting as part of a cluster name is non-existent for this module for a list of valid names, use name '?' NOTICE: CMM: Node node1 (nodeid = 1) with votecount = 1 added. NOTICE: CMM: Node node2 (nodeid = 2) with votecount = 0 added. NOTICE: clcomm: Adapter xnf2 constructed NOTICE: clcomm: Adapter xnf1 constructed NOTICE: CMM: Node node2: attempting to join cluster. NOTICE: CMM: Node node1 (nodeid: 1, incarnation #: 1231582933) has become reachable. WARNING: xnf1: oversized packet (1518 bytes) dropped NOTICE: clcomm: Path node2:xnf1 - node1:xnf1 online NOTICE: clcomm: Path node2:xnf2 - node1:xnf2 online WARNING: xnf1: oversized packet (1518 bytes) dropped WARNING: xnf1: oversized packet (1518 bytes) dropped WARNING: xnf2: oversized packet (1518 bytes) dropped WARNING: xnf2: oversized packet (1518 bytes) dropped WARNING: xnf2: oversized packet (1518 bytes) dropped # uname -srvi SunOS 5.11 snv_104 i86xpv # dladm show-link LINK CLASS MTU STATE OVER bge0 phys 1500 unknown -- e1000g0 phys 1500 up -- vnic1 vnic 1500 unknown e1000g0 vnic2 vnic 1500 unknown e1000g0 vnic18 vnic 1500 unknown e1000g0 vnic19 vnic 1500 unknown e1000g0 vnic20 vnic 1500 unknown e1000g0 -- Regards, Piotr Jasiukajtis | estibi | SCA OS0072 http://estseg.blogspot.com
