[ClusterLabs] FW: ocf_heartbeat_IPaddr2 - Real MAC of interface is revealed

2018-03-13 Thread Andreas M. Iwanowski 
Dear folks,

We are currently trying to set up a multimaster cluster and use a cloned 
ocf_heartbeat_IPaddr2 resource to share the IP address.

We have, however, run into a problem that, when a cluster member is taken 
offline, the MAC for the IP address changes from the multicast-MAC to the 
interface mac of the remaining host.
When the other host is put pack online, pings to the cluster IP time out when 
it changes back to multicast (until the ARP cache on the router expires).

Is there any way to prevent network devices from learning the interface MACs? 
I.e. even if one host is servicing both resources, use the multicast MAC?
Any help would be appreciated!

Here is the pcs status:
===
Cluster name: test_svc
WARNING: corosync and pacemaker node names do not match (IPs used in setup?)
Stack: corosync
Current DC: host1 (version 1.1.16-12.el7_4.8-94ff4df) - partition with quorum 
Last updated: Tue Mar 13 07:12:07 2018 Last change: Sun Mar 11 17:17:04 2018 by 
hacluster via crmd on host1

2 nodes configured
2 resources configured

Online: [ host1 host2 ]

Full list of resources:

 Clone Set: RedmineIP-clone [RedmineIP] (unique)
 RedmineIP:0(ocf::heartbeat:IPaddr2):   Started host1
 RedmineIP:1(ocf::heartbeat:IPaddr2):   Started host2

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled
===





___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Antw: Re: Antw: Re: single node fails to start the ocfs2 resource

2018-03-13 Thread Ulrich Windl 


>>> Klaus Wenninger  schrieb am 13.03.2018 um 16:18 in
Nachricht :
> On 03/13/2018 03:43 PM, Muhammad Sharfuddin wrote:
>> Thanks a lot for the explanation. But other then the ocfs2 resource
>> group, this cluster starts all other resources
>>
>> on a single node, without any issue just because the use of
>> "no-quorum-policy=ignore" option.
> 
> Yes I know. And what I tried to point out is that "no-quorum-policy=ignore"
> is dangerous for services that do require a resource-manager. If you don't
> have any of those go with a systemd startup.

But will a two-node cluster ever have a quorum if one node has failed?
(I still think Muhammad did something wrong, besides of that)

> 
> Regards,
> Klaus
> 
>>
>> -- 
>> Regards,
>> Muhammad Sharfuddin
>>
>> On 3/13/2018 7:32 PM, Klaus Wenninger wrote:
>>> On 03/13/2018 02:30 PM, Muhammad Sharfuddin wrote:
 Yes, by saying pacemaker,  I meant to say corosync as well.

 Is there any fix ? or a two node cluster can't run ocfs2 resources
 when one node is offline ?
>>> Actually there can't be a "fix" as 2 nodes are just not enough
>>> for a partial-cluster to be quorate in the classical sense
>>> (more votes than half of the cluster nodes).
>>>
>>> So to still be able to use it we have this 2-node config that
>>> permanently sets quorum. But not to run into issues on
>>> startup we need it to require both nodes seeing each
>>> other once.
>>>
>>> So this is definitely nothing that is specific to ocfs2.
>>> It just looks specific to ocfs2 because you've disabled
>>> quorum for pacemaker.
>>> To be honnest doing this you wouldn't need a resource-manager
>>> at all and could just start up your services using systemd.
>>>
>>> If you don't want a full 3rd node, and still want to handle cases
>>> where one node doesn't come up after a full shutdown of
>>> all nodes, you probably could go for a setup with qdevice.
>>>
>>> Regards,
>>> Klaus
>>>
 -- 
 Regards,
 Muhammad Sharfuddin

 On 3/13/2018 6:16 PM, Klaus Wenninger wrote:
> On 03/13/2018 02:03 PM, Muhammad Sharfuddin wrote:
>> Hi,
>>
>> 1 - if I put a node(node2) offline; ocfs2 resources keep running on
>> online node(node1)
>>
>> 2 - while node2 was offline, via cluster I stop/start the ocfs2
>> resource group successfully so many times in a row.
>>
>> 3 - while node2 was offline; I restart the pacemaker service on the
>> node1 and then tries to start the ocfs2 resource group, dlm started
>> but ocfs2 file system resource does not start.
>>
>> Nutshell:
>>
>> a - both nodes must be online to start the ocfs2 resource.
>>
>> b - if one crashes or offline(gracefully) ocfs2 resource keeps
>> running
>> on the other/surviving node.
>>
>> c - while one node was offline, we can stop/start the ocfs2 resource
>> group on the surviving node but if we stops the pacemaker service,
>> then ocfs2 file system resource does not start with the following
>> info
>> in the logs:
> >From the logs I would say startup of dlm_controld times out
> because it
> is waiting
> for quorum - which doesn't happen because of wait-for-all.
> Question is if you really just stopped pacemaker or if you stopped
> corosync as well.
> In the latter case I would say it is the expected behavior.
>
> Regards,
> Klaus
>  
>> lrmd[4317]:   notice: executing - rsc:p-fssapmnt action:start
>> call_id:53
>> Filesystem(p-fssapmnt)[5139]: INFO: Running start for
>> /dev/mapper/sapmnt on /sapmnt
>> kernel: [  706.162676] dlm: Using TCP for communications
>> kernel: [  706.162916] dlm: BFA9FF042AA045F4822C2A6A06020EE9: joining
>> the lockspace group...
>> dlm_controld[5105]: 759 fence work wait for quorum
>> dlm_controld[5105]: 764 BFA9FF042AA045F4822C2A6A06020EE9 wait for
>> quorum
>> lrmd[4317]:  warning: p-fssapmnt_start_0 process (PID 5139) timed out
>> lrmd[4317]:  warning: p-fssapmnt_start_0:5139 - timed out after
>> 6ms
>> lrmd[4317]:   notice: finished - rsc:p-fssapmnt action:start
>> call_id:53 pid:5139 exit-code:1 exec-time:60002ms queue-time:0ms
>> kernel: [  766.056514] dlm: BFA9FF042AA045F4822C2A6A06020EE9: group
>> event done -512 0
>> kernel: [  766.056528] dlm: BFA9FF042AA045F4822C2A6A06020EE9: group
>> join failed -512 0
>> crmd[4320]:   notice: Result of stop operation for p-fssapmnt on
>> pipci001: 0 (ok)
>> crmd[4320]:   notice: Initiating stop operation dlm_stop_0 locally on
>> pipci001
>> lrmd[4317]:   notice: executing - rsc:dlm action:stop call_id:56
>> dlm_controld[5105]: 766 shutdown ignored, active lockspaces
>> lrmd[4317]:  warning: dlm_stop_0 process (PID 5326) timed out
>> lrmd[4317]:  warning: dlm_stop_0:5326 - timed out after 10ms
>> lrmd[4317]:   notice: finished -

Re: [ClusterLabs] Antw: Re: single node fails to start the ocfs2 resource

2018-03-13 Thread Klaus Wenninger 
On 03/13/2018 03:43 PM, Muhammad Sharfuddin wrote:
> Thanks a lot for the explanation. But other then the ocfs2 resource
> group, this cluster starts all other resources
>
> on a single node, without any issue just because the use of
> "no-quorum-policy=ignore" option.

Yes I know. And what I tried to point out is that "no-quorum-policy=ignore"
is dangerous for services that do require a resource-manager. If you don't
have any of those go with a systemd startup.

Regards,
Klaus

>
> -- 
> Regards,
> Muhammad Sharfuddin
>
> On 3/13/2018 7:32 PM, Klaus Wenninger wrote:
>> On 03/13/2018 02:30 PM, Muhammad Sharfuddin wrote:
>>> Yes, by saying pacemaker,  I meant to say corosync as well.
>>>
>>> Is there any fix ? or a two node cluster can't run ocfs2 resources
>>> when one node is offline ?
>> Actually there can't be a "fix" as 2 nodes are just not enough
>> for a partial-cluster to be quorate in the classical sense
>> (more votes than half of the cluster nodes).
>>
>> So to still be able to use it we have this 2-node config that
>> permanently sets quorum. But not to run into issues on
>> startup we need it to require both nodes seeing each
>> other once.
>>
>> So this is definitely nothing that is specific to ocfs2.
>> It just looks specific to ocfs2 because you've disabled
>> quorum for pacemaker.
>> To be honnest doing this you wouldn't need a resource-manager
>> at all and could just start up your services using systemd.
>>
>> If you don't want a full 3rd node, and still want to handle cases
>> where one node doesn't come up after a full shutdown of
>> all nodes, you probably could go for a setup with qdevice.
>>
>> Regards,
>> Klaus
>>
>>> -- 
>>> Regards,
>>> Muhammad Sharfuddin
>>>
>>> On 3/13/2018 6:16 PM, Klaus Wenninger wrote:
 On 03/13/2018 02:03 PM, Muhammad Sharfuddin wrote:
> Hi,
>
> 1 - if I put a node(node2) offline; ocfs2 resources keep running on
> online node(node1)
>
> 2 - while node2 was offline, via cluster I stop/start the ocfs2
> resource group successfully so many times in a row.
>
> 3 - while node2 was offline; I restart the pacemaker service on the
> node1 and then tries to start the ocfs2 resource group, dlm started
> but ocfs2 file system resource does not start.
>
> Nutshell:
>
> a - both nodes must be online to start the ocfs2 resource.
>
> b - if one crashes or offline(gracefully) ocfs2 resource keeps
> running
> on the other/surviving node.
>
> c - while one node was offline, we can stop/start the ocfs2 resource
> group on the surviving node but if we stops the pacemaker service,
> then ocfs2 file system resource does not start with the following
> info
> in the logs:
 >From the logs I would say startup of dlm_controld times out
 because it
 is waiting
 for quorum - which doesn't happen because of wait-for-all.
 Question is if you really just stopped pacemaker or if you stopped
 corosync as well.
 In the latter case I would say it is the expected behavior.

 Regards,
 Klaus
  
> lrmd[4317]:   notice: executing - rsc:p-fssapmnt action:start
> call_id:53
> Filesystem(p-fssapmnt)[5139]: INFO: Running start for
> /dev/mapper/sapmnt on /sapmnt
> kernel: [  706.162676] dlm: Using TCP for communications
> kernel: [  706.162916] dlm: BFA9FF042AA045F4822C2A6A06020EE9: joining
> the lockspace group...
> dlm_controld[5105]: 759 fence work wait for quorum
> dlm_controld[5105]: 764 BFA9FF042AA045F4822C2A6A06020EE9 wait for
> quorum
> lrmd[4317]:  warning: p-fssapmnt_start_0 process (PID 5139) timed out
> lrmd[4317]:  warning: p-fssapmnt_start_0:5139 - timed out after
> 6ms
> lrmd[4317]:   notice: finished - rsc:p-fssapmnt action:start
> call_id:53 pid:5139 exit-code:1 exec-time:60002ms queue-time:0ms
> kernel: [  766.056514] dlm: BFA9FF042AA045F4822C2A6A06020EE9: group
> event done -512 0
> kernel: [  766.056528] dlm: BFA9FF042AA045F4822C2A6A06020EE9: group
> join failed -512 0
> crmd[4320]:   notice: Result of stop operation for p-fssapmnt on
> pipci001: 0 (ok)
> crmd[4320]:   notice: Initiating stop operation dlm_stop_0 locally on
> pipci001
> lrmd[4317]:   notice: executing - rsc:dlm action:stop call_id:56
> dlm_controld[5105]: 766 shutdown ignored, active lockspaces
> lrmd[4317]:  warning: dlm_stop_0 process (PID 5326) timed out
> lrmd[4317]:  warning: dlm_stop_0:5326 - timed out after 10ms
> lrmd[4317]:   notice: finished - rsc:dlm action:stop call_id:56
> pid:5326 exit-code:1 exec-time:13ms queue-time:0ms
> crmd[4320]:    error: Result of stop operation for dlm on pipci001:
> Timed Out
> crmd[4320]:  warning: Action 15 (dlm_stop_0) on pipci001 failed
> (target: 0 vs. rc: 1): Error
> crmd[4320]:   notice: Transition aborted by operation dlm_stop_0
> 'modify' on pipci001: Event failed
>

Re: [ClusterLabs] Antw: Re: single node fails to start the ocfs2 resource

2018-03-13 Thread Muhammad Sharfuddin 
Thanks a lot for the explanation. But other then the ocfs2 resource 
group, this cluster starts all other resources


on a single node, without any issue just because the use of 
"no-quorum-policy=ignore" option.


--
Regards,
Muhammad Sharfuddin

On 3/13/2018 7:32 PM, Klaus Wenninger wrote:

On 03/13/2018 02:30 PM, Muhammad Sharfuddin wrote:

Yes, by saying pacemaker,  I meant to say corosync as well.

Is there any fix ? or a two node cluster can't run ocfs2 resources
when one node is offline ?

Actually there can't be a "fix" as 2 nodes are just not enough
for a partial-cluster to be quorate in the classical sense
(more votes than half of the cluster nodes).

So to still be able to use it we have this 2-node config that
permanently sets quorum. But not to run into issues on
startup we need it to require both nodes seeing each
other once.

So this is definitely nothing that is specific to ocfs2.
It just looks specific to ocfs2 because you've disabled
quorum for pacemaker.
To be honnest doing this you wouldn't need a resource-manager
at all and could just start up your services using systemd.

If you don't want a full 3rd node, and still want to handle cases
where one node doesn't come up after a full shutdown of
all nodes, you probably could go for a setup with qdevice.

Regards,
Klaus


--
Regards,
Muhammad Sharfuddin

On 3/13/2018 6:16 PM, Klaus Wenninger wrote:

On 03/13/2018 02:03 PM, Muhammad Sharfuddin wrote:

Hi,

1 - if I put a node(node2) offline; ocfs2 resources keep running on
online node(node1)

2 - while node2 was offline, via cluster I stop/start the ocfs2
resource group successfully so many times in a row.

3 - while node2 was offline; I restart the pacemaker service on the
node1 and then tries to start the ocfs2 resource group, dlm started
but ocfs2 file system resource does not start.

Nutshell:

a - both nodes must be online to start the ocfs2 resource.

b - if one crashes or offline(gracefully) ocfs2 resource keeps running
on the other/surviving node.

c - while one node was offline, we can stop/start the ocfs2 resource
group on the surviving node but if we stops the pacemaker service,
then ocfs2 file system resource does not start with the following info
in the logs:

>From the logs I would say startup of dlm_controld times out because it
is waiting
for quorum - which doesn't happen because of wait-for-all.
Question is if you really just stopped pacemaker or if you stopped
corosync as well.
In the latter case I would say it is the expected behavior.

Regards,
Klaus
  

lrmd[4317]:   notice: executing - rsc:p-fssapmnt action:start
call_id:53
Filesystem(p-fssapmnt)[5139]: INFO: Running start for
/dev/mapper/sapmnt on /sapmnt
kernel: [  706.162676] dlm: Using TCP for communications
kernel: [  706.162916] dlm: BFA9FF042AA045F4822C2A6A06020EE9: joining
the lockspace group...
dlm_controld[5105]: 759 fence work wait for quorum
dlm_controld[5105]: 764 BFA9FF042AA045F4822C2A6A06020EE9 wait for
quorum
lrmd[4317]:  warning: p-fssapmnt_start_0 process (PID 5139) timed out
lrmd[4317]:  warning: p-fssapmnt_start_0:5139 - timed out after 6ms
lrmd[4317]:   notice: finished - rsc:p-fssapmnt action:start
call_id:53 pid:5139 exit-code:1 exec-time:60002ms queue-time:0ms
kernel: [  766.056514] dlm: BFA9FF042AA045F4822C2A6A06020EE9: group
event done -512 0
kernel: [  766.056528] dlm: BFA9FF042AA045F4822C2A6A06020EE9: group
join failed -512 0
crmd[4320]:   notice: Result of stop operation for p-fssapmnt on
pipci001: 0 (ok)
crmd[4320]:   notice: Initiating stop operation dlm_stop_0 locally on
pipci001
lrmd[4317]:   notice: executing - rsc:dlm action:stop call_id:56
dlm_controld[5105]: 766 shutdown ignored, active lockspaces
lrmd[4317]:  warning: dlm_stop_0 process (PID 5326) timed out
lrmd[4317]:  warning: dlm_stop_0:5326 - timed out after 10ms
lrmd[4317]:   notice: finished - rsc:dlm action:stop call_id:56
pid:5326 exit-code:1 exec-time:13ms queue-time:0ms
crmd[4320]:    error: Result of stop operation for dlm on pipci001:
Timed Out
crmd[4320]:  warning: Action 15 (dlm_stop_0) on pipci001 failed
(target: 0 vs. rc: 1): Error
crmd[4320]:   notice: Transition aborted by operation dlm_stop_0
'modify' on pipci001: Event failed
crmd[4320]:  warning: Action 15 (dlm_stop_0) on pipci001 failed
(target: 0 vs. rc: 1): Error
pengine[4319]:   notice: Watchdog will be used via SBD if fencing is
required
pengine[4319]:   notice: On loss of CCM Quorum: Ignore
pengine[4319]:  warning: Processing failed op stop for dlm:0 on
pipci001: unknown error (1)
pengine[4319]:  warning: Processing failed op stop for dlm:0 on
pipci001: unknown error (1)
pengine[4319]:  warning: Cluster node pipci001 will be fenced: dlm:0
failed there
pengine[4319]:  warning: Processing failed op start for p-fssapmnt:0
on pipci001: unknown error (1)
pengine[4319]:   notice: Stop of failed resource dlm:0 is implicit
after pipci001 is fenced
pengine[4319]:   notice:  * Fence pipci001
pengine[4319]:   notice: Stop    sbd-stonith#011(pipci001)

Re: [ClusterLabs] Antw: Re: single node fails to start the ocfs2 resource

2018-03-13 Thread Klaus Wenninger 
On 03/13/2018 02:30 PM, Muhammad Sharfuddin wrote:
> Yes, by saying pacemaker,  I meant to say corosync as well.
>
> Is there any fix ? or a two node cluster can't run ocfs2 resources
> when one node is offline ?

Actually there can't be a "fix" as 2 nodes are just not enough
for a partial-cluster to be quorate in the classical sense
(more votes than half of the cluster nodes).

So to still be able to use it we have this 2-node config that
permanently sets quorum. But not to run into issues on
startup we need it to require both nodes seeing each
other once.

So this is definitely nothing that is specific to ocfs2.
It just looks specific to ocfs2 because you've disabled
quorum for pacemaker.
To be honnest doing this you wouldn't need a resource-manager
at all and could just start up your services using systemd.

If you don't want a full 3rd node, and still want to handle cases
where one node doesn't come up after a full shutdown of
all nodes, you probably could go for a setup with qdevice.

Regards,
Klaus

>
> -- 
> Regards,
> Muhammad Sharfuddin
>
> On 3/13/2018 6:16 PM, Klaus Wenninger wrote:
>> On 03/13/2018 02:03 PM, Muhammad Sharfuddin wrote:
>>> Hi,
>>>
>>> 1 - if I put a node(node2) offline; ocfs2 resources keep running on
>>> online node(node1)
>>>
>>> 2 - while node2 was offline, via cluster I stop/start the ocfs2
>>> resource group successfully so many times in a row.
>>>
>>> 3 - while node2 was offline; I restart the pacemaker service on the
>>> node1 and then tries to start the ocfs2 resource group, dlm started
>>> but ocfs2 file system resource does not start.
>>>
>>> Nutshell:
>>>
>>> a - both nodes must be online to start the ocfs2 resource.
>>>
>>> b - if one crashes or offline(gracefully) ocfs2 resource keeps running
>>> on the other/surviving node.
>>>
>>> c - while one node was offline, we can stop/start the ocfs2 resource
>>> group on the surviving node but if we stops the pacemaker service,
>>> then ocfs2 file system resource does not start with the following info
>>> in the logs:
>> >From the logs I would say startup of dlm_controld times out because it
>> is waiting
>> for quorum - which doesn't happen because of wait-for-all.
>> Question is if you really just stopped pacemaker or if you stopped
>> corosync as well.
>> In the latter case I would say it is the expected behavior.
>>
>> Regards,
>> Klaus
>>  
>>> lrmd[4317]:   notice: executing - rsc:p-fssapmnt action:start
>>> call_id:53
>>> Filesystem(p-fssapmnt)[5139]: INFO: Running start for
>>> /dev/mapper/sapmnt on /sapmnt
>>> kernel: [  706.162676] dlm: Using TCP for communications
>>> kernel: [  706.162916] dlm: BFA9FF042AA045F4822C2A6A06020EE9: joining
>>> the lockspace group...
>>> dlm_controld[5105]: 759 fence work wait for quorum
>>> dlm_controld[5105]: 764 BFA9FF042AA045F4822C2A6A06020EE9 wait for
>>> quorum
>>> lrmd[4317]:  warning: p-fssapmnt_start_0 process (PID 5139) timed out
>>> lrmd[4317]:  warning: p-fssapmnt_start_0:5139 - timed out after 6ms
>>> lrmd[4317]:   notice: finished - rsc:p-fssapmnt action:start
>>> call_id:53 pid:5139 exit-code:1 exec-time:60002ms queue-time:0ms
>>> kernel: [  766.056514] dlm: BFA9FF042AA045F4822C2A6A06020EE9: group
>>> event done -512 0
>>> kernel: [  766.056528] dlm: BFA9FF042AA045F4822C2A6A06020EE9: group
>>> join failed -512 0
>>> crmd[4320]:   notice: Result of stop operation for p-fssapmnt on
>>> pipci001: 0 (ok)
>>> crmd[4320]:   notice: Initiating stop operation dlm_stop_0 locally on
>>> pipci001
>>> lrmd[4317]:   notice: executing - rsc:dlm action:stop call_id:56
>>> dlm_controld[5105]: 766 shutdown ignored, active lockspaces
>>> lrmd[4317]:  warning: dlm_stop_0 process (PID 5326) timed out
>>> lrmd[4317]:  warning: dlm_stop_0:5326 - timed out after 10ms
>>> lrmd[4317]:   notice: finished - rsc:dlm action:stop call_id:56
>>> pid:5326 exit-code:1 exec-time:13ms queue-time:0ms
>>> crmd[4320]:    error: Result of stop operation for dlm on pipci001:
>>> Timed Out
>>> crmd[4320]:  warning: Action 15 (dlm_stop_0) on pipci001 failed
>>> (target: 0 vs. rc: 1): Error
>>> crmd[4320]:   notice: Transition aborted by operation dlm_stop_0
>>> 'modify' on pipci001: Event failed
>>> crmd[4320]:  warning: Action 15 (dlm_stop_0) on pipci001 failed
>>> (target: 0 vs. rc: 1): Error
>>> pengine[4319]:   notice: Watchdog will be used via SBD if fencing is
>>> required
>>> pengine[4319]:   notice: On loss of CCM Quorum: Ignore
>>> pengine[4319]:  warning: Processing failed op stop for dlm:0 on
>>> pipci001: unknown error (1)
>>> pengine[4319]:  warning: Processing failed op stop for dlm:0 on
>>> pipci001: unknown error (1)
>>> pengine[4319]:  warning: Cluster node pipci001 will be fenced: dlm:0
>>> failed there
>>> pengine[4319]:  warning: Processing failed op start for p-fssapmnt:0
>>> on pipci001: unknown error (1)
>>> pengine[4319]:   notice: Stop of failed resource dlm:0 is implicit
>>> after pipci001 is fenced
>>> pengine[4319]:   notice:  * Fence pipci001
>>> pengine[4319]:  

Re: [ClusterLabs] Antw: Re: single node fails to start the ocfs2 resource

2018-03-13 Thread Klaus Wenninger 
On 03/13/2018 02:03 PM, Muhammad Sharfuddin wrote:
> Hi,
>
> 1 - if I put a node(node2) offline; ocfs2 resources keep running on
> online node(node1)
>
> 2 - while node2 was offline, via cluster I stop/start the ocfs2
> resource group successfully so many times in a row.
>
> 3 - while node2 was offline; I restart the pacemaker service on the
> node1 and then tries to start the ocfs2 resource group, dlm started
> but ocfs2 file system resource does not start.
>
> Nutshell:
>
> a - both nodes must be online to start the ocfs2 resource.
>
> b - if one crashes or offline(gracefully) ocfs2 resource keeps running
> on the other/surviving node.
>
> c - while one node was offline, we can stop/start the ocfs2 resource
> group on the surviving node but if we stops the pacemaker service,
> then ocfs2 file system resource does not start with the following info
> in the logs:

From the logs I would say startup of dlm_controld times out because it
is waiting
for quorum - which doesn't happen because of wait-for-all.
Question is if you really just stopped pacemaker or if you stopped
corosync as well.
In the latter case I would say it is the expected behavior.

Regards,
Klaus
 
>
> lrmd[4317]:   notice: executing - rsc:p-fssapmnt action:start call_id:53
> Filesystem(p-fssapmnt)[5139]: INFO: Running start for
> /dev/mapper/sapmnt on /sapmnt
> kernel: [  706.162676] dlm: Using TCP for communications
> kernel: [  706.162916] dlm: BFA9FF042AA045F4822C2A6A06020EE9: joining
> the lockspace group...
> dlm_controld[5105]: 759 fence work wait for quorum
> dlm_controld[5105]: 764 BFA9FF042AA045F4822C2A6A06020EE9 wait for quorum
> lrmd[4317]:  warning: p-fssapmnt_start_0 process (PID 5139) timed out
> lrmd[4317]:  warning: p-fssapmnt_start_0:5139 - timed out after 6ms
> lrmd[4317]:   notice: finished - rsc:p-fssapmnt action:start
> call_id:53 pid:5139 exit-code:1 exec-time:60002ms queue-time:0ms
> kernel: [  766.056514] dlm: BFA9FF042AA045F4822C2A6A06020EE9: group
> event done -512 0
> kernel: [  766.056528] dlm: BFA9FF042AA045F4822C2A6A06020EE9: group
> join failed -512 0
> crmd[4320]:   notice: Result of stop operation for p-fssapmnt on
> pipci001: 0 (ok)
> crmd[4320]:   notice: Initiating stop operation dlm_stop_0 locally on
> pipci001
> lrmd[4317]:   notice: executing - rsc:dlm action:stop call_id:56
> dlm_controld[5105]: 766 shutdown ignored, active lockspaces
> lrmd[4317]:  warning: dlm_stop_0 process (PID 5326) timed out
> lrmd[4317]:  warning: dlm_stop_0:5326 - timed out after 10ms
> lrmd[4317]:   notice: finished - rsc:dlm action:stop call_id:56
> pid:5326 exit-code:1 exec-time:13ms queue-time:0ms
> crmd[4320]:    error: Result of stop operation for dlm on pipci001:
> Timed Out
> crmd[4320]:  warning: Action 15 (dlm_stop_0) on pipci001 failed
> (target: 0 vs. rc: 1): Error
> crmd[4320]:   notice: Transition aborted by operation dlm_stop_0
> 'modify' on pipci001: Event failed
> crmd[4320]:  warning: Action 15 (dlm_stop_0) on pipci001 failed
> (target: 0 vs. rc: 1): Error
> pengine[4319]:   notice: Watchdog will be used via SBD if fencing is
> required
> pengine[4319]:   notice: On loss of CCM Quorum: Ignore
> pengine[4319]:  warning: Processing failed op stop for dlm:0 on
> pipci001: unknown error (1)
> pengine[4319]:  warning: Processing failed op stop for dlm:0 on
> pipci001: unknown error (1)
> pengine[4319]:  warning: Cluster node pipci001 will be fenced: dlm:0
> failed there
> pengine[4319]:  warning: Processing failed op start for p-fssapmnt:0
> on pipci001: unknown error (1)
> pengine[4319]:   notice: Stop of failed resource dlm:0 is implicit
> after pipci001 is fenced
> pengine[4319]:   notice:  * Fence pipci001
> pengine[4319]:   notice: Stop    sbd-stonith#011(pipci001)
> pengine[4319]:   notice: Stop    dlm:0#011(pipci001)
> crmd[4320]:   notice: Requesting fencing (reboot) of node pipci001
> stonith-ng[4316]:   notice: Client crmd.4320.4c2f757b wants to fence
> (reboot) 'pipci001' with device '(any)'
> stonith-ng[4316]:   notice: Requesting peer fencing (reboot) of pipci001
> stonith-ng[4316]:   notice: sbd-stonith can fence (reboot) pipci001:
> dynamic-list
>
>
> -- 
> Regards,
> Muhammad Sharfuddin | +923332144823 | nds.com.pk
>
> On 3/13/2018 1:04 PM, Ulrich Windl wrote:
>> Hi!
>>
>> I'd recommend this:
>> Cleanly boot your nodes, avoiding any manual operation with cluster
>> resources. Keep the logs.
>> Then start your tests, keeping the logs for each.
>> Try to fix issues by reading the logs and adjusting the cluster
>> configuration, and not by starting commands that the cluster should
>> start.
>>
>> We had an 2-node OCFS2 cluster running for quite some time with
>> SLES11, but now the cluster is three nodes. To me the output of
>> "crm_mon -1Arfj" combined with having set record-pending=true was
>> very valuable finding problems.
>>
>> Regards,
>> Ulrich
>>
>>
> Muhammad Sharfuddin  schrieb am
> 13.03.2018 um 08:43 in
>> Nachricht

Re: [ClusterLabs] Antw: Re: single node fails to start the ocfs2 resource

2018-03-13 Thread Muhammad Sharfuddin 

Hi,

1 - if I put a node(node2) offline; ocfs2 resources keep running on 
online node(node1)


2 - while node2 was offline, via cluster I stop/start the ocfs2 resource 
group successfully so many times in a row.


3 - while node2 was offline; I restart the pacemaker service on the 
node1 and then tries to start the ocfs2 resource group, dlm started but 
ocfs2 file system resource does not start.


Nutshell:

a - both nodes must be online to start the ocfs2 resource.

b - if one crashes or offline(gracefully) ocfs2 resource keeps running 
on the other/surviving node.


c - while one node was offline, we can stop/start the ocfs2 resource 
group on the surviving node but if we stops the pacemaker service, then 
ocfs2 file system resource does not start with the following info in the 
logs:


lrmd[4317]:   notice: executing - rsc:p-fssapmnt action:start call_id:53
Filesystem(p-fssapmnt)[5139]: INFO: Running start for /dev/mapper/sapmnt 
on /sapmnt

kernel: [  706.162676] dlm: Using TCP for communications
kernel: [  706.162916] dlm: BFA9FF042AA045F4822C2A6A06020EE9: joining 
the lockspace group...

dlm_controld[5105]: 759 fence work wait for quorum
dlm_controld[5105]: 764 BFA9FF042AA045F4822C2A6A06020EE9 wait for quorum
lrmd[4317]:  warning: p-fssapmnt_start_0 process (PID 5139) timed out
lrmd[4317]:  warning: p-fssapmnt_start_0:5139 - timed out after 6ms
lrmd[4317]:   notice: finished - rsc:p-fssapmnt action:start call_id:53 
pid:5139 exit-code:1 exec-time:60002ms queue-time:0ms
kernel: [  766.056514] dlm: BFA9FF042AA045F4822C2A6A06020EE9: group 
event done -512 0
kernel: [  766.056528] dlm: BFA9FF042AA045F4822C2A6A06020EE9: group join 
failed -512 0
crmd[4320]:   notice: Result of stop operation for p-fssapmnt on 
pipci001: 0 (ok)
crmd[4320]:   notice: Initiating stop operation dlm_stop_0 locally on 
pipci001

lrmd[4317]:   notice: executing - rsc:dlm action:stop call_id:56
dlm_controld[5105]: 766 shutdown ignored, active lockspaces
lrmd[4317]:  warning: dlm_stop_0 process (PID 5326) timed out
lrmd[4317]:  warning: dlm_stop_0:5326 - timed out after 10ms
lrmd[4317]:   notice: finished - rsc:dlm action:stop call_id:56 pid:5326 
exit-code:1 exec-time:13ms queue-time:0ms
crmd[4320]:    error: Result of stop operation for dlm on pipci001: 
Timed Out
crmd[4320]:  warning: Action 15 (dlm_stop_0) on pipci001 failed (target: 
0 vs. rc: 1): Error
crmd[4320]:   notice: Transition aborted by operation dlm_stop_0 
'modify' on pipci001: Event failed
crmd[4320]:  warning: Action 15 (dlm_stop_0) on pipci001 failed (target: 
0 vs. rc: 1): Error
pengine[4319]:   notice: Watchdog will be used via SBD if fencing is 
required

pengine[4319]:   notice: On loss of CCM Quorum: Ignore
pengine[4319]:  warning: Processing failed op stop for dlm:0 on 
pipci001: unknown error (1)
pengine[4319]:  warning: Processing failed op stop for dlm:0 on 
pipci001: unknown error (1)
pengine[4319]:  warning: Cluster node pipci001 will be fenced: dlm:0 
failed there
pengine[4319]:  warning: Processing failed op start for p-fssapmnt:0 on 
pipci001: unknown error (1)
pengine[4319]:   notice: Stop of failed resource dlm:0 is implicit after 
pipci001 is fenced

pengine[4319]:   notice:  * Fence pipci001
pengine[4319]:   notice: Stop    sbd-stonith#011(pipci001)
pengine[4319]:   notice: Stop    dlm:0#011(pipci001)
crmd[4320]:   notice: Requesting fencing (reboot) of node pipci001
stonith-ng[4316]:   notice: Client crmd.4320.4c2f757b wants to fence 
(reboot) 'pipci001' with device '(any)'

stonith-ng[4316]:   notice: Requesting peer fencing (reboot) of pipci001
stonith-ng[4316]:   notice: sbd-stonith can fence (reboot) pipci001: 
dynamic-list



--
Regards,
Muhammad Sharfuddin | +923332144823 | nds.com.pk

On 3/13/2018 1:04 PM, Ulrich Windl wrote:

Hi!

I'd recommend this:
Cleanly boot your nodes, avoiding any manual operation with cluster resources. 
Keep the logs.
Then start your tests, keeping the logs for each.
Try to fix issues by reading the logs and adjusting the cluster configuration, 
and not by starting commands that the cluster should start.

We had an 2-node OCFS2 cluster running for quite some time with SLES11, but now the 
cluster is three nodes. To me the output of "crm_mon -1Arfj" combined with 
having set record-pending=true was very valuable finding problems.

Regards,
Ulrich



Muhammad Sharfuddin  schrieb am 13.03.2018 um 08:43 in

Nachricht <7b773ae9-4209-d246-b5c0-2c8b67e62...@nds.com.pk>:

Dear Klaus,

If I understand you properly then, its a fencing issue, and whatever I
am facing is "natural" or "by-design" in a two node cluster where quorum
is incomplete.

I am quite convinced that you have pointed out right because, when I
start the dlm resource via cluster and then tries to start the ocfs2
file system manually from command line, mount command remains hanged and
following events are reported in the logs:

  kernel: [62622.864828] ocfs2: Registered cluster interface user
  kernel:

[ClusterLabs] Antw: Re: single node fails to start the ocfs2 resource

2018-03-13 Thread Ulrich Windl 
Hi!

I'd recommend this:
Cleanly boot your nodes, avoiding any manual operation with cluster resources. 
Keep the logs.
Then start your tests, keeping the logs for each.
Try to fix issues by reading the logs and adjusting the cluster configuration, 
and not by starting commands that the cluster should start.

We had an 2-node OCFS2 cluster running for quite some time with SLES11, but now 
the cluster is three nodes. To me the output of "crm_mon -1Arfj" combined with 
having set record-pending=true was very valuable finding problems.

Regards,
Ulrich


>>> Muhammad Sharfuddin  schrieb am 13.03.2018 um 
>>> 08:43 in
Nachricht <7b773ae9-4209-d246-b5c0-2c8b67e62...@nds.com.pk>:
> Dear Klaus,
> 
> If I understand you properly then, its a fencing issue, and whatever I 
> am facing is "natural" or "by-design" in a two node cluster where quorum 
> is incomplete.
> 
> I am quite convinced that you have pointed out right because, when I 
> start the dlm resource via cluster and then tries to start the ocfs2 
> file system manually from command line, mount command remains hanged and 
> following events are reported in the logs:
> 
>  kernel: [62622.864828] ocfs2: Registered cluster interface user
>  kernel: [62622.884427] dlm: Using TCP for communications
>  kernel: [62622.884750] dlm: BFA9FF042AA045F4822C2A6A06020EE9: 
> joining the lockspace group...
>  dlm_controld[17655]: 62627 fence work wait for quorum
>  dlm_controld[17655]: 62680 BFA9FF042AA045F4822C2A6A06020EE9 wait 
> for quorum
> 
> and then following messages keep reported every 5-10 minutes, till I 
> kill the mount.ocfs2 process:
> 
>  dlm_controld[17655]: 62627 fence work wait for quorum
>  dlm_controld[17655]: 62680 BFA9FF042AA045F4822C2A6A06020EE9 wait 
> for quorum
> 
> I am also very much confused, because yesterday I did the same and was 
> able to mount the ocfs2 file system manually from command line(at least 
> once), and then unmount the file system manually stop the dlm resource 
> from cluster and then complete ocfs2 resource stack(dlm, file systems) 
> start/stop successfully via cluster even when only machine was online.
> 
> In a two-node cluster, which have ocfs2 resources, we can't run the 
> ocfs2 resources when quorum is incomplete(one node is offline) ?
> 
> --
> Regards,
> Muhammad Sharfuddin
> 
> On 3/12/2018 5:58 PM, Klaus Wenninger wrote:
>> On 03/12/2018 01:44 PM, Muhammad Sharfuddin wrote:
>>> Hi Klaus,
>>>
>>> primitive sbd-stonith stonith:external/sbd \
>>>  op monitor interval=3000 timeout=20 \
>>>  op start interval=0 timeout=240 \
>>>  op stop interval=0 timeout=100 \
>>>  params sbd_device="/dev/mapper/sbd" \
>>>  meta target-role=Started
>> Makes more sense now.
>> Using pcmk_delay_max would probably be useful here
>> to prevent a fence-race.
>> That stonith-resource was not in your resource-list below ...
>>
>>> property cib-bootstrap-options: \
>>>  have-watchdog=true \
>>>  stonith-enabled=true \
>>>  no-quorum-policy=ignore \
>>>  stonith-timeout=90 \
>>>  startup-fencing=true
>> You've set no-quorum-policy=ignore for pacemaker.
>> Whether this is a good idea or not in your setup is
>> written on another page.
>> But isn't dlm directly interfering with corosync so
>> that it would get the quorum state from there?
>> As you have 2-node set probably on a 2-node-cluster
>> this would - after both nodes down - wait for all
>> nodes up first.
>>
>> Regards,
>> Klaus
>>
>>> # ps -eaf |grep sbd
>>> root  6129 1  0 17:35 ?00:00:00 sbd: inquisitor
>>> root  6133  6129  0 17:35 ?00:00:00 sbd: watcher:
>>> /dev/mapper/sbd - slot: 1 - uuid: 6e80a337-95db-4608-bd62-d59517f39103
>>> root  6134  6129  0 17:35 ?00:00:00 sbd: watcher: Pacemaker
>>> root  6135  6129  0 17:35 ?00:00:00 sbd: watcher: Cluster
>>>
>>> This cluster does not start ocfs2 resources when I first intentionally
>>> crashed(reboot) both the nodes, then try to start ocfs2 resource while
>>> one node is  offline.
>>>
>>> To fix the issue, I have one permanent solution, bring the other
>>> node(offline) online and things get fixed automatically, i.e ocfs2
>>> resources mounts.
>>>
>>> -- 
>>> Regards,
>>> Muhammad Sharfuddin
>>>
>>> On 3/12/2018 5:25 PM, Klaus Wenninger wrote:
 Hi Muhammad!

 Could you be a little bit more elaborate on your fencing-setup!
 I read about you using SBD but I don't see any sbd-fencing-resource.
 For the case you wanted to use watchdog-fencing with SBD this
 would require stonith-watchdog-timeout property to be set.
 But watchdog-fencing relies on quorum (without 2-node trickery)
 and thus wouldn't work on a 2-node-cluster anyway.

 Didn't read through the whole thread - so I might be missing
 something ...

 Regards,
 Klaus

 On 03/12/2018 12:51 PM, Muhammad Sharfuddin wrote:
> Hello

[ClusterLabs] Antw: Re: Antw: Re: Antw: Re: [Cluster-devel] DLM connection channel switch take too long time (> 5mins)

2018-03-13 Thread Ulrich Windl 


>>> Guoqing Jiang  schrieb am 13.03.2018 um 08:42 in Nachricht
<6f0f512b-c641-72cd-d03b-0a903c84a...@suse.com>:

> 
> On 03/13/2018 03:09 PM, Ulrich Windl wrote:
>>
> Guoqing Jiang  schrieb am 13.03.2018 um 05:00 in 
> Nachricht
>> <87a8e0e7-63f3-2329-6598-1fa80b1b1...@suse.com>:
>>
>>> On 03/08/2018 07:24 PM, Ulrich Windl wrote:
 Hi!

 What surprises me most is that a connect(...O_NONBLOCK) actually blocks:

 EINPROGRESS
 The  socket  is  non-blocking  and the connection cannot be
>>> com-
 pleted immediately.

>>> Maybe it is because that the socket is created by sock_create_kern, and
>>> O_NONBLOCK flag is not worked  since __sctp_connect has the following
>>> description.
>>>
>>>  /* in-kernel sockets don't generally have a file allocated to them
>>>* if all they do is call sock_create_kern().
>>>*/
>>>   if (sk->sk_socket->file)
>>>   f_flags = sk->sk_socket->file->f_flags;
>>>
>>>   timeo = sock_sndtimeo(sk, f_flags & O_NONBLOCK);
>> But O_NONBLOCK is still passed to sock_sndtimeo() (unless the intention was 
> "f_flags & ~O_NONBLOCK").
>>
> 
> If I am not reading wrong, the O_NONBLOCK doesn't have effective above
> since f_flags is still 0.

OK; I didn't look close enough; you are right.

> 
> So, in this case, timeo is set to sk->sk_sndtimeo. In sctp_wait_for_connect
> you also can see nonblock is only work when timeo is set to 0.
> 
>  if (!*timeo_p)
>  goto do_nonblock;


Another point would be to signal an error for a socket operation with 
unsupported flags, like O_NONBLOCK here. In the end it's probably better to 
remove O_NONBLOCK from the call reather than assuming it would have an effect 
in this case.

The other question (for the kernel developers) is whether the file structure is 
the best place to pass flags like O_NONBLOCK then.

Regards,
Ulrich


___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Antw: Re: [Cluster-devel] DLM connection channel switch take too long time (> 5mins)

2018-03-13 Thread Guoqing Jiang 



On 03/13/2018 03:09 PM, Ulrich Windl wrote:



Guoqing Jiang  schrieb am 13.03.2018 um 05:00 in Nachricht

<87a8e0e7-63f3-2329-6598-1fa80b1b1...@suse.com>:


On 03/08/2018 07:24 PM, Ulrich Windl wrote:

Hi!

What surprises me most is that a connect(...O_NONBLOCK) actually blocks:

EINPROGRESS
The  socket  is  non-blocking  and the connection cannot be

com-

pleted immediately.


Maybe it is because that the socket is created by sock_create_kern, and
O_NONBLOCK flag is not worked  since __sctp_connect has the following
description.

 /* in-kernel sockets don't generally have a file allocated to them
   * if all they do is call sock_create_kern().
   */
  if (sk->sk_socket->file)
  f_flags = sk->sk_socket->file->f_flags;

  timeo = sock_sndtimeo(sk, f_flags & O_NONBLOCK);

But O_NONBLOCK is still passed to sock_sndtimeo() (unless the intention was "f_flags 
& ~O_NONBLOCK").



If I am not reading wrong, the O_NONBLOCK doesn't have effective above
since f_flags is still 0.

So, in this case, timeo is set to sk->sk_sndtimeo. In sctp_wait_for_connect
you also can see nonblock is only work when timeo is set to 0.

            if (!*timeo_p)
    goto do_nonblock;


Thanks,
Guoqing
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Antw: Re: Antw: Re: [Cluster-devel] DLM connection channel switch take too long time (> 5mins)

2018-03-13 Thread Ulrich Windl 


>>> Guoqing Jiang  schrieb am 13.03.2018 um 05:00 in Nachricht
<87a8e0e7-63f3-2329-6598-1fa80b1b1...@suse.com>:

> 
> On 03/08/2018 07:24 PM, Ulrich Windl wrote:
>> Hi!
>>
>> What surprises me most is that a connect(...O_NONBLOCK) actually blocks:
>>
>> EINPROGRESS
>>The  socket  is  non-blocking  and the connection cannot be 
> com-
>>pleted immediately.
>>
> 
> Maybe it is because that the socket is created by sock_create_kern, and
> O_NONBLOCK flag is not worked  since __sctp_connect has the following
> description.
> 
> /* in-kernel sockets don't generally have a file allocated to them
>   * if all they do is call sock_create_kern().
>   */
>  if (sk->sk_socket->file)
>  f_flags = sk->sk_socket->file->f_flags;
> 
>  timeo = sock_sndtimeo(sk, f_flags & O_NONBLOCK);

But O_NONBLOCK is still passed to sock_sndtimeo() (unless the intention was 
"f_flags & ~O_NONBLOCK").

> 
> Thanks,
> Guoqing
> ___
> Users mailing list: Users@clusterlabs.org 
> https://lists.clusterlabs.org/mailman/listinfo/users 
> 
> Project Home: http://www.clusterlabs.org 
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf 
> Bugs: http://bugs.clusterlabs.org

___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org