Sorry for this.
I try to simplify my configuration and delete all unnecessary things and start
only with one LUN, one allowed Initiators, and then step by step I add things..
now I ended up with the following setup:
node 1: ha-test1 \
attributes \
attributes standby=off maintenance=off
node 2: ha-test2 \
attributes \
attributes standby=off
primitive ha-ip IPaddr2 \
params ip=172.16.101.166 cidr_netmask=24 nic=ens192 \
op start interval=0s timeout=20s \
op stop interval=0s timeout=20s \
op monitor interval=10s timeout=20s \
meta target-role=Started
primitive iscsi-lun00 iSCSILogicalUnit \
params implementation=lio-t
target_iqn="iqn.2003-01.org.linux-iscsi.vm-storage.x8664:sn.cf6fa66tgyh3" lun=0
lio_iblock=0 path="/dev/loop1" \
meta
primitive iscsi-lun01 iSCSILogicalUnit \
params implementation=lio-t
target_iqn="iqn.2003-01.org.linux-iscsi.vm-storage.x8664:sn.cf6fa66tgyh3" lun=1
lio_iblock=1 path="/dev/loop2"
primitive iscsi-lun02 iSCSILogicalUnit \
params implementation=lio-t
target_iqn="iqn.2003-01.org.linux-iscsi.vm-storage.x8664:sn.cf6fa66tgyh3" lun=2
lio_iblock=2 path="/dev/loop3" \
meta
primitive iscsi-server iSCSITarget \
params implementation=lio-t
iqn="iqn.2003-01.org.linux-iscsi.vm-storage.x8664:sn.cf6fa66tgyh3"
portals="172.16.101.166:3260"
allowed_initiators="iqn.1998-01.com.vmware:brainslug99-75u88a95
iqn.1998-01.com.vmware:brainslug69-5564u4325
iqn.1993-08.org.debian:01:fee35be01c4d
iqn.1998-01.com.vmware:brainslug14-34a81jd123
iqn.1998-01.com.vmware:brainslug66-778jau77" \
meta
colocation pcs_rsc_colocation_set_ha-ip_vm_storage_iscsi-server inf: ha-ip
iscsi-server iscsi-lun00 iscsi-lun01 iscsi-lun02
order pcs_rsc_order_set_iscsi-server_vm_storage_ha-ip iscsi-server iscsi-lun00
iscsi-lun01 iscsi-lun02 ha-ip
property cib-bootstrap-options: \
have-watchdog=false \
dc-version=1.1.16-94ff4df \
cluster-infrastructure=corosync \
cluster-name=ha-vmstorage \
no-quorum-policy=stop \
stonith-enabled=false \
last-lrm-refresh=1576502457
rsc_defaults rsc_defaults-options: \
resource-stickiness=100
before that I've the following order:
order pcs_rsc_order_set_ha-ip_iscsi-server_vm_storage ha-ip:stop
iscsi-lun00:stop iscsi-lun01:stop iscsi-lun02:stop iscsi-server:stop
symmetrical=false
order pcs_rsc_order_set_iscsi-server_vm_storage_ha-ip iscsi-server:start
iscsi-lun00:start iscsi-lun01:start iscsi-lun02:start ha-ip:start
symmetrical=false
and I thought that this and my current working constraint are the same, but
with the 2line-configuration it doesn't work and with the one-liner it works
fine, so can somebody explain me this?
On Monday, December 16, 2019 6:38:44 PM CET Andrei Borzenkov wrote:
> 16.12.2019 18:26, Stefan K пишет:
> > I thnik I got it..
> >
> > It looks like that (A)
> > order pcs_rsc_order_set_iscsi-server_haip iscsi-server:start
> > iscsi-lun00:start iscsi-lun01:start iscsi-lun02:start ha-ip:start
> > symmetrical=false
>
> It is different from configuration you show originally.
>
> > order pcs_rsc_order_set_haip_iscsi-server ha-ip:stop iscsi-lun02:stop
> > iscsi-lun01:stop iscsi-lun00:stop iscsi-server:stop symmetrical=false
> >
> > and (B)
> > order pcs_rsc_order_set_iscsi-server_vm_storage_ha-ip iscsi-server
> > iscsi-lun00 iscsi-lun01 iscsi-lun02 ha-ip
> >
> > doesn't have the same meaning?!
> > Because with A it doesn't work, but with B it works like expected, can
> > somebody explain me this behavior?
> >
>
> Your original configuration was not symmetrical which may explain it.
> You never said anything about changing configuration so it is unclear
> what you tested - original statement or statement you show now.
>
> > best regards
> > Stefan
> >
> >
> > On Thursday, December 12, 2019 4:19:19 PM CET Stefan K wrote:
> >> So it looks like that it restart the iSCSITarget but not the
> >> iSCSILogicalUnit, that make sense - more or less because I change
> >> something in the iSCSITarget, but it is necessary to have a working
> >> iSCSI.. here a the log output when I change/add the iqn..
> >>
> >> Dec 12 16:08:21 [7051] ha-test1 cib: info: cib_process_request:
> >> Forwarding cib_apply_diff operation for section 'all' to all
> >> (origin=local/cibadmin/2)
> >> Dec 12 16:08:21 [7051] ha-test1 cib: info: cib_perform_op:
> >> Diff: --- 0.58.3 2
> >> Dec 12 16:08:21 [7051] ha-test1 cib: info: cib_perform_op:
> >> Diff: +++ 0.59.0 c5423fcdc276ad43361aeb4c8081f7f4
> >> Dec 12 16:08:21 [7051] ha-test1 cib: info: cib_perform_op: +
> >> /cib: @epoch=59, @num_updates=0
> >> Dec 12 16:08:21 [7051] ha-test1 cib: info: cib_perform_op: +
> >> /cib/configuration/resources/primitive[@id='iscsi-server']/instance_attributes[@id='iscsi-server-instance_attributes']/nvpair[@id='iscsi-server-instance_attributes-allowed_initiators']:
> >> @value=iqn.1998-01.com.vmware:brainslug9-75488e35
> >> iqn.1998-01.com.vmware:brainslug10-5564u4325
> >> iqn.1993-08.org.debian:01:fee35be01c4d
> >> iqn.1998-01.com.vmware:brainslug10-34ad648763
> >> iqn.1998-01.com.vmware:brainslug66-75488e12 iqn.1998-01.com.vmware:brai
> >> Dec 12 16:08:21 [7051] ha-test1 cib: info: cib_process_request:
> >> Completed cib_apply_diff operation for section 'all': OK (rc=0,
> >> origin=ha-test1/cibadmin/2, version=0.59.0)
> >> Dec 12 16:08:21 [7051] ha-test1 cib: info: cib_file_backup:
> >> Archived previous version as /var/lib/pacemaker/cib/cib-69.raw
> >> Dec 12 16:08:21 [7056] ha-test1 crmd: info: do_lrm_rsc_op:
> >> Performing key=12:116:0:9fd4e826-f0ba-4864-8861-2c585d644d1c
> >> op=iscsi-server_stop_0
> >> Dec 12 16:08:21 [7053] ha-test1 lrmd: info: log_execute:
> >> executing - rsc:iscsi-server action:stop call_id:62
> >> Dec 12 16:08:21 [7051] ha-test1 cib: info:
> >> cib_file_write_with_digest: Wrote version 0.59.0 of the CIB to disk
> >> (digest: dcbab759c4d0e7f38234434bfbe7ca8e)
> >> Dec 12 16:08:21 [7051] ha-test1 cib: info:
> >> cib_file_write_with_digest: Reading cluster configuration file
> >> /var/lib/pacemaker/cib/cib.6YgOXO (digest:
> >> /var/lib/pacemaker/cib/cib.gOKlWj)
> >> iSCSITarget(iscsi-server)[4524]: 2019/12/12_16:08:22 INFO: Deleted
> >> Target iqn.2003-01.org.linux-iscsi.vm-storage.x8664:sn.cf6fa66tgyh3.
> >> Dec 12 16:08:22 [7053] ha-test1 lrmd: info: log_finished:
> >> finished - rsc:iscsi-server action:stop call_id:62 pid:4524 exit-code:0
> >> exec-time:293ms queue-time:0ms
> >> Dec 12 16:08:22 [7056] ha-test1 crmd: notice: process_lrm_event:
> >> Result of stop operation for iscsi-server on ha-test1: 0 (ok) |
> >> call=62 key=iscsi-server_stop_0 confirmed=true cib-update=58
> >> Dec 12 16:08:22 [7051] ha-test1 cib: info: cib_process_request:
> >> Forwarding cib_modify operation for section status to all
> >> (origin=local/crmd/58)
> >> Dec 12 16:08:22 [7051] ha-test1 cib: info: cib_perform_op:
> >> Diff: --- 0.59.0 2
> >> Dec 12 16:08:22 [7051] ha-test1 cib: info: cib_perform_op:
> >> Diff: +++ 0.59.1 (null)
> >> Dec 12 16:08:22 [7051] ha-test1 cib: info: cib_perform_op: +
> >> /cib: @num_updates=1
> >> Dec 12 16:08:22 [7051] ha-test1 cib: info: cib_perform_op: +
> >> /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='iscsi-server']/lrm_rsc_op[@id='iscsi-server_last_0']:
> >> @operation_key=iscsi-server_stop_0, @operation=stop,
> >> @transition-key=12:116:0:9fd4e826-f0ba-4864-8861-2c585d644d1c,
> >> @transition-magic=0:0;12:116:0:9fd4e826-f0ba-4864-8861-2c585d644d1c,
> >> @call-id=62, @last-run=1576163301, @last-rc-change=1576163301,
> >> @exec-time=293
> >> Dec 12 16:08:22 [7051] ha-test1 cib: info: cib_process_request:
> >> Completed cib_modify operation for section status: OK (rc=0,
> >> origin=ha-test1/crmd/58, version=0.59.1)
> >> Dec 12 16:08:22 [7056] ha-test1 crmd: info: do_lrm_rsc_op:
> >> Performing key=3:116:0:9fd4e826-f0ba-4864-8861-2c585d644d1c
> >> op=iscsi-server_start_0
> >> Dec 12 16:08:22 [7053] ha-test1 lrmd: info: log_execute:
> >> executing - rsc:iscsi-server action:start call_id:63
> >> iSCSITarget(iscsi-server)[4564]: 2019/12/12_16:08:22 INFO:
> >> Parameter auto_add_default_portal is now 'false'.
> >> iSCSITarget(iscsi-server)[4564]: 2019/12/12_16:08:22 INFO: Created
> >> target iqn.2003-01.org.linux-iscsi.vm-storage.x8664:sn.cf6fa66tgyh3.
> >> Created TPG 1.
> >> iSCSITarget(iscsi-server)[4564]: 2019/12/12_16:08:22 INFO: Using
> >> default IP port 3260 Created network portal 172.16.101.166:3260.
> >> iSCSITarget(iscsi-server)[4564]: 2019/12/12_16:08:22 INFO: Created
> >> Node ACL for iqn.1998-01.com.vmware:brainslug9-75488e35
> >> iSCSITarget(iscsi-server)[4564]: 2019/12/12_16:08:23 INFO: Created
> >> Node ACL for iqn.1998-01.com.vmware:brainslug10-5564u4325
> >> iSCSITarget(iscsi-server)[4564]: 2019/12/12_16:08:23 INFO: Created
> >> Node ACL for iqn.1993-08.org.debian:01:fee35be01c4d
> >> iSCSITarget(iscsi-server)[4564]: 2019/12/12_16:08:23 INFO: Created
> >> Node ACL for iqn.1998-01.com.vmware:brainslug10-34ad648763
> >> iSCSITarget(iscsi-server)[4564]: 2019/12/12_16:08:23 INFO: Created
> >> Node ACL for iqn.1998-01.com.vmware:brainslug66-75488e12
> >> iSCSITarget(iscsi-server)[4564]: 2019/12/12_16:08:23 INFO: Created
> >> Node ACL for iqn.1998-01.com.vmware:brainslug99-5564u4123
> >> iSCSITarget(iscsi-server)[4564]: 2019/12/12_16:08:24 INFO:
> >> Parameter authentication is now '0'.
> >> Dec 12 16:08:24 [7053] ha-test1 lrmd: info: log_finished:
> >> finished - rsc:iscsi-server action:start call_id:63 pid:4564 exit-code:0
> >> exec-time:1781ms queue-time:0ms
> >> Dec 12 16:08:24 [7056] ha-test1 crmd: info: action_synced_wait:
> >> Managed iSCSITarget_meta-data_0 process 4695 exited with rc=0
> >> Dec 12 16:08:24 [7051] ha-test1 cib: info: cib_process_request:
> >> Forwarding cib_modify operation for section status to all
> >> (origin=local/crmd/59)
> >> Dec 12 16:08:24 [7056] ha-test1 crmd: notice: process_lrm_event:
> >> Result of start operation for iscsi-server on ha-test1: 0 (ok) |
> >> call=63 key=iscsi-server_start_0 confirmed=true cib-update=59
> >> Dec 12 16:08:24 [7051] ha-test1 cib: info: cib_perform_op:
> >> Diff: --- 0.59.1 2
> >> Dec 12 16:08:24 [7051] ha-test1 cib: info: cib_perform_op:
> >> Diff: +++ 0.59.2 (null)
> >> Dec 12 16:08:24 [7051] ha-test1 cib: info: cib_perform_op: +
> >> /cib: @num_updates=2
> >> Dec 12 16:08:24 [7051] ha-test1 cib: info: cib_perform_op: +
> >> /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='iscsi-server']/lrm_rsc_op[@id='iscsi-server_last_0']:
> >> @operation_key=iscsi-server_start_0, @operation=start,
> >> @transition-key=3:116:0:9fd4e826-f0ba-4864-8861-2c585d644d1c,
> >> @transition-magic=0:0;3:116:0:9fd4e826-f0ba-4864-8861-2c585d644d1c,
> >> @call-id=63, @last-run=1576163302, @last-rc-change=1576163302,
> >> @exec-time=1781, @op-digest=549b6bf42c2d944da4df2c1d2d675b
> >> Dec 12 16:08:24 [7051] ha-test1 cib: info: cib_process_request:
> >> Completed cib_modify operation for section status: OK (rc=0,
> >> origin=ha-test1/crmd/59, version=0.59.2)
> >> Dec 12 16:08:29 [7051] ha-test1 cib: info: cib_process_ping:
> >> Reporting our current digest to ha-test2: 8bf64451f91add76d89a608b6f51a214
> >> for 0.59.2 (0x55998d145df0 0)
> >>
> >>
> >>
> >>
> >> On Wednesday, December 11, 2019 3:58:51 PM CET Stefan K wrote:
> >>> Hello,
> >>>
> >>> I've a working HA-Setup with iSCSI an ZFS, but last week I add an iSCSI
> >>> allowed initiator, and than it happens - my hole VMware infrastructure
> >>> fails because iSCSI does not working anymore.. today I've time to get a
> >>> closer look into this..
> >>>
> >>> create 2 VMs an put the same (more or less) config into it.
> >>> What I do:
> >>> - I create a iSCSI-target with allowed initiators
> >>> - I create iSCSI Logical units
> >>>
> >>> but I got this:
> >>>
> >>> targetcli
> >>> targetcli shell version 2.1.fb43
> >>> Copyright 2011-2013 by Datera, Inc and others.
> >>> For help on commands, type 'help'.
> >>>
> >>> /> ls
> >>> o- /
> >>> .........................................................................................................................
> >>> [...]
> >>> o- backstores
> >>> ..............................................................................................................
> >>> [...]
> >>> | o- block
> >>> ..................................................................................................
> >>> [Storage Objects: 3]
> >>> | | o- iscsi-lun00
> >>> ..................................................................
> >>> [/dev/loop1 (1.0GiB) write-thru deactivated]
> >>> | | o- iscsi-lun01
> >>> ..................................................................
> >>> [/dev/loop2 (1.0GiB) write-thru deactivated]
> >>> | | o- iscsi-lun02
> >>> .................................................................
> >>> [/dev/loop3 (0 bytes) write-thru deactivated]
> >>> | o- fileio
> >>> .................................................................................................
> >>> [Storage Objects: 0]
> >>> | o- pscsi
> >>> ..................................................................................................
> >>> [Storage Objects: 0]
> >>> | o- ramdisk
> >>> ................................................................................................
> >>> [Storage Objects: 0]
> >>> o- iscsi
> >>> ............................................................................................................
> >>> [Targets: 1]
> >>> | o- iqn.2003-01.org.linux-iscsi.vm-storage.x8664:sn.cf6fa66tgyh3
> >>> ...................................................... [TPGs: 1]
> >>> | o- tpg1
> >>> ...............................................................................................
> >>> [no-gen-acls, no-auth]
> >>> | o- acls
> >>> ..........................................................................................................
> >>> [ACLs: 4]
> >>> | | o- iqn.1993-08.org.debian:01:fee35be01c4d
> >>> ............................................................... [Mapped
> >>> LUNs: 0]
> >>> | | o- iqn.1998-01.com.vmware:brainslug10-34ad648763
> >>> ........................................................ [Mapped LUNs: 0]
> >>> | | o- iqn.1998-01.com.vmware:brainslug10-5564u4325
> >>> ......................................................... [Mapped LUNs: 0]
> >>> | | o- iqn.1998-01.com.vmware:brainslug9-75488e35
> >>> ........................................................... [Mapped LUNs:
> >>> 0]
> >>> | o- luns
> >>> ..........................................................................................................
> >>> [LUNs: 0]
> >>> | o- portals
> >>> ....................................................................................................
> >>> [Portals: 1]
> >>> | o- 172.16.101.166:3260
> >>> ..............................................................................................
> >>> [OK]
> >>> o- loopback
> >>> .........................................................................................................
> >>> [Targets: 0]
> >>> o- vhost
> >>> ............................................................................................................
> >>> [Targets: 0]
> >>>
> >>>
> >>> here you can see there are missing luns, when I move the ressource to the
> >>> other node, it will shown the luns, if I then add/remove/change an
> >>> "allowed_initiators" it will happen again - all luns are gone. And that
> >>> is a very serious problem for us.
> >>>
> >>> So my questions is, do i misconfigure something or is that a bug? My
> >>> pacemaker config looks like the following:
> >>>
> >>> crm conf sh
> >>> node 1: ha-test1 \
> >>> attributes \
> >>> attributes standby=off maintenance=off
> >>> node 2: ha-test2 \
> >>> attributes \
> >>> attributes standby=off
> >>> primitive ha-ip IPaddr2 \
> >>> params ip=172.16.101.166 cidr_netmask=24 nic=ens192 \
> >>> op start interval=0s timeout=20s \
> >>> op stop interval=0s timeout=20s \
> >>> op monitor interval=10s timeout=20s \
> >>> meta target-role=Started
> >>> primitive iscsi-lun00 iSCSILogicalUnit \
> >>> params implementation=lio-t
> >>> target_iqn="iqn.2003-01.org.linux-iscsi.vm-storage.x8664:sn.cf6fa66tgyh3"
> >>> lun=0 lio_iblock=0 path="/dev/loop1" \
> >>> op start interval=0 trace_ra=1 \
> >>> op stop interval=0 trace_ra=1 \
> >>> meta target-role=Started
> >>> primitive iscsi-lun01 iSCSILogicalUnit \
> >>> params implementation=lio-t
> >>> target_iqn="iqn.2003-01.org.linux-iscsi.vm-storage.x8664:sn.cf6fa66tgyh3"
> >>> lun=1 lio_iblock=1 path="/dev/loop2" \
> >>> meta
> >>> primitive iscsi-lun02 iSCSILogicalUnit \
> >>> params implementation=lio-t
> >>> target_iqn="iqn.2003-01.org.linux-iscsi.vm-storage.x8664:sn.cf6fa66tgyh3"
> >>> lun=2 lio_iblock=2 path="/dev/loop3" \
> >>> meta
> >>> primitive iscsi-server iSCSITarget \
> >>> params implementation=lio-t
> >>> iqn="iqn.2003-01.org.linux-iscsi.vm-storage.x8664:sn.cf6fa66tgyh3"
> >>> portals="172.16.101.166:3260"
> >>> allowed_initiators="iqn.1998-01.com.vmware:brainslug9-75488e35
> >>> iqn.1998-01.com.vmware:brainslug10-5564u4325
> >>> iqn.1993-08.org.debian:01:fee35be01c4d
> >>> iqn.1998-01.com.vmware:brainslug10-34ad648763" \
> >>> meta
> >>> colocation pcs_rsc_colocation_set_ha-ip_vm_storage_iscsi-server inf:
> >>> ha-ip iscsi-server iscsi-lun00 iscsi-lun01 iscsi-lun02
> >>> order pcs_rsc_order_set_ha-ip_iscsi-server_vm_storage ha-ip:stop
> >>> iscsi-lun00:stop iscsi-lun01:stop iscsi-lun02:stop iscsi-server:stop
> >>> symmetrical=false
> >>> order pcs_rsc_order_set_iscsi-server_vm_storage_ha-ip iscsi-lun00:start
> >>> iscsi-server:start iscsi-lun01:start iscsi-lun02:start ha-ip:start
> >>> symmetrical=false
> >>> property cib-bootstrap-options: \
> >>> have-watchdog=false \
> >>> dc-version=1.1.16-94ff4df \
> >>> cluster-infrastructure=corosync \
> >>> cluster-name=ha-vmstorage \
> >>> no-quorum-policy=stop \
> >>> stonith-enabled=false \
> >>> last-lrm-refresh=1576056627
> >>> rsc_defaults rsc_defaults-options: \
> >>> resource-stickiness=100
> >>>
> >>>
> >>> The system is running on Debian Stretch.
> >>>
> >>> Thank you very much for you help!
> >>>
> >>> best regards
> >>> Stefan
> >>>
> >>>
> >>> _______________________________________________
> >>> Manage your subscription:
> >>> https://lists.clusterlabs.org/mailman/listinfo/users
> >>>
> >>> ClusterLabs home: https://www.clusterlabs.org/
> >>>
> >>
> >>
> >>
> >> _______________________________________________
> >> Manage your subscription:
> >> https://lists.clusterlabs.org/mailman/listinfo/users
> >>
> >> ClusterLabs home: https://www.clusterlabs.org/
> >>
> >
> >
> >
> > _______________________________________________
> > Manage your subscription:
> > https://lists.clusterlabs.org/mailman/listinfo/users
> >
> > ClusterLabs home: https://www.clusterlabs.org/
> >
>
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
_______________________________________________
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users
ClusterLabs home: https://www.clusterlabs.org/
