Hi Piotr,
Your situation is strange for sure. Both nodes are
able to see each other (at least at heartbeat level), but
then node2 is hanging somewhere, it appears.
However, in your latest e-mail, i am not sure which part
you think is strange. ethertype 0x833 are the Cluster heartbeat
messages, so you would expect to see that. Not
sure about 0x88CC (the multicast packet), but i don't
think that is the problem per-se.
About the ARP entries:
> clprivnet0 172.16.4.1 255.255.255.255 SPLA
00:00:00:00:00:01
That is fine too. The weired looking MAC address is basically
derived from the nodeid.
So.... Back to the original problem, which i am having a bit
of trouble understanding. You mention:
>>> There is a problem with interconnects so I can't create the cluster.
But that doesn't seem to be the case, both nodes are able to see
each other and form the cluster, right? Here is the message
on node1 which confirms that.
>>> Jan 10 11:30:24 node1 genunix: [ID 537175 kern.notice] NOTICE: CMM:
Node
>>> node2 (nodeid: 2, incarnation #: 1231583261) has become reachable.
And on node2,
>>> NOTICE: CMM: Node node2: attempting to join cluster.
>>> NOTICE: CMM: Node node1 (nodeid: 1, incarnation #: 1231582933) has
>>> become reachable.
So that seem to work find. Sure.. the status of the adapters
is faulted:
>>> node1:xnf2 node2:xnf2 faulted
>>> node1:xnf1 node2:xnf1 faulted
Which tells me that while the heartbeats are flowing,
perhaps the TCP traffic is having trouble getting across, we
can look at that, but perhaps your basic problem is:
>> Btw, I can ping node2 via interconnects from node1.
>> I can't login to node2 neither from console nor via sshd.
But then you have this output... Perhaps that is not from node2?
>>> # uname -srvi
>>> SunOS 5.11 snv_104 i86xpv
>>>
>>> # dladm show-link
>>> LINK CLASS MTU STATE OVER
It is a bit confusing because the included console outputs
don't indicate the node they are from.
I am gonna assume that the problem you refer to is that node2 never
gets to the console login prompt and is stuck "somewhere". Let me
know if the "problem" you refer to is something else.
Typically, if i see a cluster related hang, it is typically the
cluster nodes waiting for quorum. Here, there is no quorum (node1
is 1 vote, node2 0 and both are part of the cluster already).
So, it is quite possible that the hang on the node2 is not
directly related to the cluster (indirectly, perhaps it is).
You might have to dive in a bit deep here, unfortunately.
Can you gather kernel buffers for the following components:
heartbeats, buffer name: hbt_dbg
path manager, buffer name: pm_dbg
CMM, buffer name: cmm_dbg_buf
pxfs, buffer name: pxfs_dbg <== THis one just in case the hang is while
trying to mount globaldevices.
from BOTH nodes and see if you can figure out what is going on?
For node2, you might have to take a core and then run mdb on
the core files to gather the buffers. In case the debug buffers
don't provide any clues, look in the core dump for anything else
holding up the boot sequence (a running SMF method and/or
a rc script).
PS: To view the content of a buffer, do the following from mdb:
*buffer_name/s
To summarize: Assuming that the problem is that node2 is not booting
up all the way to the login prompt, it is not clear if this is
a cluster issue (a message such as: Waiting for Quorum should
typically be there in such cases), but in case it is, hope the
above suggestions would give you a clue.
Once we have a clearer understanding of what the state of things
are, we might wanna take a closer look at the faulted state of
the interconnects. I have a feeling that it may lie at the root
of everything. The error messages for oversized packet looks
strange. It wouldn't be the first time a network driver had
some issue correctly calculating the size of a packet which
comes in over the private interconnect.
While looking at the core (on node2), keep your eyes open for
any non-cluster stuff holding up the boot sequence too.
PS: It helps to set a console prompt which is the name of
the machine so that console output can be correlated to
the name of the machine it is taken on.
HTH,
-ashu
Piotr Jasiukajtis wrote:
> I think it's strange:
>
> [root at node1 ~]# snoop -v -r -d xnf1
> Using device xnf1 (promiscuous mode)
> ETHER: ----- Ether Header -----
> ETHER:
> ETHER: Packet 1 arrived at 23:01:0.56727
> ETHER: Packet size = 60 bytes
> ETHER: Destination = 0:16:3e:46:23:9b,
> ETHER: Source = 0:16:3e:46:31:9b,
> ETHER: Ethertype = 0833 (Unknown)
> ETHER:
>
> ETHER: ----- Ether Header -----
> ETHER:
> ETHER: Packet 2 arrived at 23:01:1.05863
> ETHER: Packet size = 42 bytes
> ETHER: Destination = 0:16:3e:46:31:9b,
> ETHER: Source = 0:16:3e:46:23:9b,
> ETHER: VLAN ID = 0
> ETHER: VLAN Priority = 7
> ETHER: Ethertype = 0833 (Unknown)
> ETHER:
>
> ETHER: ----- Ether Header -----
> ETHER:
> ETHER: Packet 3 arrived at 23:01:1.26870
> ETHER: Packet size = 60 bytes
> ETHER: Destination = 1:80:c2:0:0:e, (multicast)
> ETHER: Source = 0:1a:70:20:c3:f9,
> ETHER: Ethertype = 88CC (Unknown)
> ETHER:
>
>
> [root at node1 ~]# arp -an | grep 172.16
> xnf1 172.16.0.129 255.255.255.255 SPLA 00:16:3e:46:23:9b
> xnf1 172.16.0.130 255.255.255.255 00:16:3e:46:31:9b
> clprivnet0 172.16.4.1 255.255.255.255 SPLA 00:00:00:00:00:01
> xnf2 172.16.1.1 255.255.255.255 SPLA 00:16:82:74:29:ba
> xnf2 172.16.1.2 255.255.255.255 00:16:3e:74:2a:ba
>
>
>
>
>
> Piotr Jasiukajtis pisze:
>> Btw, I can ping node2 via interconnects from node1.
>> I can't login to node2 neither from console nor via sshd.
>>
>>
>>
>> Piotr Jasiukajtis pisze:
>>> Hi,
>>>
>>> I installed Solaris Cluster Express 12/08 on 2 pvm virtual nodes (on 2
>>> physical SXCE104 machines).
>>>
>>> Each virtual node has 3 NICs (3 VNICs from xVM dom0),
>>> but each physical node has only one physical NIC connected to the one
>>> physical switch. I know about security issues and such...
>>>
>>> There is a problem with interconnects so I can't create the cluster.
>>>
>>> Any idea? :)
>>>
>>>
>>>
>>> [root at node1 ~]# /usr/cluster/bin/clnode status -v
>>>
>>> === Cluster Nodes ===
>>>
>>> --- Node Status ---
>>>
>>> Node Name Status
>>> --------- ------
>>> node1 Online
>>> node2 Offline
>>>
>>>
>>> --- Node IPMP Group Status ---
>>>
>>> Node Name Group Name Status Adapter Status
>>> --------- ---------- ------ ------- ------
>>> node1 sc_ipmp0 Online xnf0 Online
>>>
>>>
>>> [root at node1 ~]# /usr/cluster/bin/clintr show
>>>
>>> === Transport Cables ===
>>>
>>> Transport Cable: node1:xnf1,switch1 at 1
>>> Endpoint1: node1:xnf1
>>> Endpoint2: switch1 at 1
>>> State: Enabled
>>>
>>> Transport Cable: node1:xnf2,switch2 at 1
>>> Endpoint1: node1:xnf2
>>> Endpoint2: switch2 at 1
>>> State: Enabled
>>>
>>> Transport Cable: node2:xnf1,switch1 at 2
>>> Endpoint1: node2:xnf1
>>> Endpoint2: switch1 at 2
>>> State: Enabled
>>>
>>> Transport Cable: node2:xnf2,switch2 at 2
>>> Endpoint1: node2:xnf2
>>> Endpoint2: switch2 at 2
>>> State: Enabled
>>>
>>>
>>> === Transport Switches ===
>>>
>>> Transport Switch: switch1
>>> State: Enabled
>>> Type: switch
>>> Port Names: 1 2
>>> Port State(1): Enabled
>>> Port State(2): Enabled
>>>
>>> Transport Switch: switch2
>>> State: Enabled
>>> Type: switch
>>> Port Names: 1 2
>>> Port State(1): Enabled
>>> Port State(2): Enabled
>>>
>>>
>>> --- Transport Adapters for node1 ---
>>>
>>> Transport Adapter: xnf1
>>> State: Enabled
>>> Transport Type: dlpi
>>> device_name: xnf
>>> device_instance: 1
>>> lazy_free: 1
>>> dlpi_heartbeat_timeout: 10000
>>> dlpi_heartbeat_quantum: 1000
>>> nw_bandwidth: 80
>>> bandwidth: 70
>>> ip_address: 172.16.0.129
>>> netmask: 255.255.255.128
>>> Port Names: 0
>>> Port State(0): Enabled
>>>
>>> Transport Adapter: xnf2
>>> State: Enabled
>>> Transport Type: dlpi
>>> device_name: xnf
>>> device_instance: 2
>>> lazy_free: 1
>>> dlpi_heartbeat_timeout: 10000
>>> dlpi_heartbeat_quantum: 1000
>>> nw_bandwidth: 80
>>> bandwidth: 70
>>> ip_address: 172.16.1.1
>>> netmask: 255.255.255.128
>>> Port Names: 0
>>> Port State(0): Enabled
>>>
>>>
>>> --- Transport Adapters for node2 ---
>>>
>>> Transport Adapter: xnf1
>>> State: Enabled
>>> Transport Type: dlpi
>>> device_name: xnf
>>> device_instance: 1
>>> lazy_free: 1
>>> dlpi_heartbeat_timeout: 10000
>>> dlpi_heartbeat_quantum: 1000
>>> nw_bandwidth: 80
>>> bandwidth: 70
>>> ip_address: 172.16.0.130
>>> netmask: 255.255.255.128
>>> Port Names: 0
>>> Port State(0): Enabled
>>>
>>> Transport Adapter: xnf2
>>> State: Enabled
>>> Transport Type: dlpi
>>> device_name: xnf
>>> device_instance: 2
>>> lazy_free: 1
>>> dlpi_heartbeat_timeout: 10000
>>> dlpi_heartbeat_quantum: 1000
>>> nw_bandwidth: 80
>>> bandwidth: 70
>>> ip_address: 172.16.1.2
>>> netmask: 255.255.255.128
>>> Port Names: 0
>>> Port State(0): Enabled
>>>
>>>
>>> [root at node1 ~]# /usr/cluster/bin/clintr status -v
>>>
>>> === Cluster Transport Paths ===
>>>
>>> Endpoint1 Endpoint2 Status
>>> --------- --------- ------
>>> node1:xnf2 node2:xnf2 faulted
>>> node1:xnf1 node2:xnf1 faulted
>>>
>>>
>>>
>>> Jan 10 11:22:14 node1 genunix: [ID 965873 kern.notice] NOTICE: CMM: Node
>>> node1 (nodeid = 1) with votecount = 1 added.
>>> Jan 10 11:22:14 node1 genunix: [ID 843983 kern.notice] NOTICE: CMM: Node
>>> node1: attempting to join cluster.
>>> Jan 10 11:22:14 node1 genunix: [ID 525628 kern.notice] NOTICE: CMM:
>>> Cluster has reached quorum.
>>> Jan 10 11:22:14 node1 genunix: [ID 377347 kern.notice] NOTICE: CMM: Node
>>> node1 (nodeid = 1) is up; new incarnation number = 1231582933.
>>> Jan 10 11:22:14 node1 genunix: [ID 108990 kern.notice] NOTICE: CMM:
>>> Cluster members: node1.
>>> Jan 10 11:22:14 node1 genunix: [ID 279084 kern.notice] NOTICE: CMM: node
>>> reconfiguration #1 completed.
>>> Jan 10 11:22:17 node1 genunix: [ID 499756 kern.notice] NOTICE: CMM: Node
>>> node1: joined cluster.
>>> Jan 10 11:22:17 node1 ip: [ID 856290 kern.notice] ip: joining multicasts
>>> failed (18) on clprivnet0 - will use link layer broadcasts for multicast
>>> Jan 10 11:22:28 node1 Cluster.CCR: [ID 914260 daemon.warning] Failed to
>>> retrieve global fencing status from the global name server
>>> Jan 10 11:22:28 node1 last message repeated 1 time
>>> Jan 10 11:22:48 node1 Cluster.CCR: [ID 409585 daemon.error]
>>> /usr/cluster/bin/scgdevs: Cannot register devices as HA.
>>> Jan 10 11:22:53 node1 xntpd[909]: [ID 702911 daemon.notice] xntpd
>>> 3-5.93e+sun 03/08/29 16:23:05 (1.4)
>>> Jan 10 11:22:53 node1 xntpd[909]: [ID 301315 daemon.notice] tickadj = 5,
>>> tick = 10000, tvu_maxslew = 495, est. hz = 100
>>> Jan 10 11:22:53 node1 xntpd[909]: [ID 266339 daemon.notice] using kernel
>>> phase-lock loop 0041, drift correction 0.00000
>>> Jan 10 11:22:53 node1 last message repeated 1 time
>>> Jan 10 11:23:00 node1 : [ID 386282 daemon.error] ccr_initialize failure
>>> Jan 10 11:23:04 node1 last message repeated 8 times
>>> Jan 10 11:23:04 node1 svc.startd[8]: [ID 748625 daemon.error]
>>> system/cluster/scdpm:default failed repeatedly: transitioned to
>>> maintenance (see 'svcs -xv' for details)
>>> Jan 10 11:24:01 node1 xpvd: [ID 395608 kern.info] xenbus at 0, xenbus0
>>> Jan 10 11:24:01 node1 genunix: [ID 936769 kern.info] xenbus0 is
>>> /xpvd/xenbus at 0
>>> Jan 10 11:28:15 node1 genunix: [ID 965873 kern.notice] NOTICE: CMM: Node
>>> node2 (nodeid = 2) with votecount = 0 added.
>>> Jan 10 11:28:15 node1 genunix: [ID 108990 kern.notice] NOTICE: CMM:
>>> Cluster members: node1.
>>> Jan 10 11:28:15 node1 genunix: [ID 279084 kern.notice] NOTICE: CMM: node
>>> reconfiguration #2 completed.
>>> Jan 10 11:28:16 node1 genunix: [ID 884114 kern.notice] NOTICE: clcomm:
>>> Adapter xnf1 constructed
>>> Jan 10 11:28:16 node1 ip: [ID 856290 kern.notice] ip: joining multicasts
>>> failed (18) on clprivnet0 - will use link layer broadcasts for multicast
>>> Jan 10 11:28:16 node1 genunix: [ID 884114 kern.notice] NOTICE: clcomm:
>>> Adapter xnf2 constructed
>>> Jan 10 11:28:25 node1 rpc_scadmd[1196]: [ID 801593 daemon.notice] stdout:
>>> Jan 10 11:28:25 node1 rpc_scadmd[1196]: [ID 801593 daemon.notice] stderr:
>>> Jan 10 11:28:26 node1 rpc_scadmd[1196]: [ID 801593 daemon.notice] stdout:
>>> Jan 10 11:28:26 node1 rpc_scadmd[1196]: [ID 801593 daemon.notice] stderr:
>>> Jan 10 11:29:16 node1 genunix: [ID 604153 kern.notice] NOTICE: clcomm:
>>> Path node1:xnf1 - node2:xnf1 errors during initiation
>>> Jan 10 11:29:16 node1 genunix: [ID 618107 kern.warning] WARNING: Path
>>> node1:xnf1 - node2:xnf1 initiation encountered errors, errno = 62.
>>> Remote node may be down or unreachable through this path.
>>> Jan 10 11:29:16 node1 genunix: [ID 604153 kern.notice] NOTICE: clcomm:
>>> Path node1:xnf2 - node2:xnf2 errors during initiation
>>> Jan 10 11:29:16 node1 genunix: [ID 618107 kern.warning] WARNING: Path
>>> node1:xnf2 - node2:xnf2 initiation encountered errors, errno = 62.
>>> Remote node may be down or unreachable through this path.
>>> Jan 10 11:30:24 node1 genunix: [ID 537175 kern.notice] NOTICE: CMM: Node
>>> node2 (nodeid: 2, incarnation #: 1231583261) has become reachable.
>>> Jan 10 11:30:24 node1 xnf: [ID 601036 kern.warning] WARNING: xnf2:
>>> oversized packet (1518 bytes) dropped
>>> Jan 10 11:30:24 node1 last message repeated 1 time
>>> Jan 10 11:30:24 node1 genunix: [ID 387288 kern.notice] NOTICE: clcomm:
>>> Path node1:xnf2 - node2:xnf2 online
>>> Jan 10 11:30:24 node1 genunix: [ID 387288 kern.notice] NOTICE: clcomm:
>>> Path node1:xnf1 - node2:xnf1 online
>>> Jan 10 11:30:28 node1 xnf: [ID 601036 kern.warning] WARNING: xnf2:
>>> oversized packet (1518 bytes) dropped
>>> Jan 10 11:30:54 node1 last message repeated 2 times
>>> Jan 10 11:30:54 node1 xnf: [ID 601036 kern.warning] WARNING: xnf1:
>>> oversized packet (1518 bytes) dropped
>>> Jan 10 11:31:08 node1 last message repeated 2 times
>>> Jan 10 11:31:28 node1 xnf: [ID 601036 kern.warning] WARNING: xnf2:
>>> oversized packet (1518 bytes) dropped
>>> Jan 10 11:32:28 node1 last message repeated 1 time
>>> Jan 10 11:33:29 node1 genunix: [ID 489438 kern.notice] NOTICE: clcomm:
>>> Path node1:xnf2 - node2:xnf2 being drained
>>> Jan 10 11:33:29 node1 genunix: [ID 387288 kern.notice] NOTICE: clcomm:
>>> Path node1:xnf2 - node2:xnf2 online
>>>
>>>
>>>
>>>
>>> [root at node1 ~]# uname -a
>>> SunOS node1 5.11 snv_101a i86pc i386 i86xpv
>>>
>>>
>>>
>>> # xm start -c node2
>>> v3.1.4-xvm chgset 'Mon Nov 24 22:48:21 2008 -0800 15909:8ac8abf844b5'
>>> SunOS Release 5.11 Version snv_101a 64-bit
>>> Copyright 1983-2008 Sun Microsystems, Inc. All rights reserved.
>>> Use is subject to license terms.
>>> Hostname: node2
>>> Configuring devices.
>>> /usr/cluster/bin/scdidadm: Could not load DID instance list.
>>> /usr/cluster/bin/scdidadm: Cannot open
>>> /etc/cluster/ccr/global/did_instances.
>>> Booting as part of a cluster
>>> name is non-existent for this module
>>> for a list of valid names, use name '?'
>>> NOTICE: CMM: Node node1 (nodeid = 1) with votecount = 1 added.
>>> NOTICE: CMM: Node node2 (nodeid = 2) with votecount = 0 added.
>>> NOTICE: clcomm: Adapter xnf2 constructed
>>> NOTICE: clcomm: Adapter xnf1 constructed
>>> NOTICE: CMM: Node node2: attempting to join cluster.
>>> NOTICE: CMM: Node node1 (nodeid: 1, incarnation #: 1231582933) has
>>> become reachable.
>>> WARNING: xnf1: oversized packet (1518 bytes) dropped
>>> NOTICE: clcomm: Path node2:xnf1 - node1:xnf1 online
>>> NOTICE: clcomm: Path node2:xnf2 - node1:xnf2 online
>>> WARNING: xnf1: oversized packet (1518 bytes) dropped
>>> WARNING: xnf1: oversized packet (1518 bytes) dropped
>>> WARNING: xnf2: oversized packet (1518 bytes) dropped
>>> WARNING: xnf2: oversized packet (1518 bytes) dropped
>>> WARNING: xnf2: oversized packet (1518 bytes) dropped
>>>
>>>
>>> # uname -srvi
>>> SunOS 5.11 snv_104 i86xpv
>>>
>>> # dladm show-link
>>> LINK CLASS MTU STATE OVER
>>> bge0 phys 1500 unknown --
>>> e1000g0 phys 1500 up --
>>> vnic1 vnic 1500 unknown e1000g0
>>> vnic2 vnic 1500 unknown e1000g0
>>> vnic18 vnic 1500 unknown e1000g0
>>> vnic19 vnic 1500 unknown e1000g0
>>> vnic20 vnic 1500 unknown e1000g0
>>>
>>>
>>>
>>
>
>
