date:20180310

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

2018-03-10 Thread Maged Mokhtar

--
From: "Jason Dillaman" 
Sent: Sunday, March 11, 2018 1:46 AM
To: "shadow_lin" 
Cc: "Lazuardi Nasution" ; "Ceph Users" 

Subject: Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive 
Lock

On Sat, Mar 10, 2018 at 10:11 AM, shadow_lin  wrote:

Hi Jason,

As discussed in this thread, for active/passive, upon initiator
failover, we used the RBD exclusive-lock feature to blacklist the old
"active" iSCSI target gateway so that it cannot talk w/ the Ceph
cluster before new writes are accepted on the new target gateway.

I can get during the new active target gateway was talking to rbd the old
active target gateway cannot write because of the RBD exclusive-lock
But after the new target gateway done the writes,if the old target 
gateway

had some blocked io during the failover,cant it then get the lock and
overwrite the new writes?

Negative -- it's blacklisted so it cannot talk to the cluster.

PS:
Petasan say they can do active/active iscsi with patched suse kernel.

I'll let them comment on these corner cases.

We are not currently handling these corner cases. We have not hit this in 
practice but will work on it. We need to account for in-flight time early in 
the target stack before reaching krbd/tcmu.

/Maged

2018-03-10

shadowlin

发件人：Jason Dillaman 
发送时间：2018-03-10 21:40
主题：Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive 
Lock

收件人："shadow_lin"
抄送："Mike Christie","Lazuardi
Nasution","Ceph Users"

On Sat, Mar 10, 2018 at 7:42 AM, shadow_lin  wrote:

Hi Mike,
So for now only suse kernel with target_rbd_core and tcmu-runner can run
active/passive multipath safely?

Negative, the LIO / tcmu-runner implementation documented here [1] is
safe for active/passive.

I am a newbie to iscsi. I think the stuck io get excuted cause overwrite
problem can happen with both active/active and active/passive.
What makes the active/passive safer than active/active?

As discussed in this thread, for active/passive, upon initiator
failover, we used the RBD exclusive-lock feature to blacklist the old
"active" iSCSI target gateway so that it cannot talk w/ the Ceph
cluster before new writes are accepted on the new target gateway.

What mechanism should be implement to avoid the problem with
active/passive
and active/active multipath?

Active/passive it solved as discussed above. For active/active, we
don't have a solution that is known safe under all failure conditions.
If LIO supported MCS (multiple connections per session) instead of
just MPIO (multipath IO), the initiator would provide enough context
to the target to detect IOs from a failover situation.

2018-03-10

shadowlin

发件人：Mike Christie 
发送时间：2018-03-09 00:54
主题：Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive 
Lock

收件人："shadow_lin","Lazuardi
Nasution","Ceph 
Users"

抄送：

On 03/07/2018 09:24 AM, shadow_lin wrote:

Hi Christie,
Is it safe to use active/passive multipath with krbd with exclusive 
lock

for lio/tgt/scst/tcmu?

No. We tried to use lio and krbd initially, but there is a issue where
IO might get stuck in the target/block layer and get executed after new
IO. So for lio, tgt and tcmu it is not safe as is right now. We could
add some code tcmu's file_example handler which can be used with krbd so
it works like the rbd one.

I do know enough about SCST right now.

Is it safe to use active/active multipath If use suse kernel with
target_core_rbd?
Thanks.

2018-03-07

shadowlin

*发件人：*Mike Christie 
*发送时间：*2018-03-07 03:51
*主题：*Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD
Exclusive Lock
*收件人：*"Lazuardi Nasution","Ceph
Users"
*抄送：*

On 03/06/2018 01:17 PM, Lazuardi Nasution wrote:
> Hi,
>
> I want to do load balanced multipathing (multiple iSCSI
gateway/exporter
> nodes) of iSCSI backed with RBD images. Should I disable 
exclusive

lock
> feature? What if I don't disable that feature? I'm using TGT
(manual
> way) since I get so many CPU stuck error messages when I was 
using

LIO.
>

You are using LIO/TGT with krbd right?

You cannot or shouldn't do active/active multipathing. If you have
the
lock enabled then it bounces between paths for each IO and will be
slow.
If you do not

[ceph-users] rbd-nbd not resizing even after kernel tweaks

2018-03-10 Thread Alex Gorbachev

I am running into the problem described in
https://lkml.org/lkml/2018/2/19/565 and
https://tracker.ceph.com/issues/23137

I went ahead and built a custom kernel reverting the change
https://github.com/torvalds/linux/commit/639812a1ed9bf49ae2c026086fbf975339cd1eef

After that a resize shows in lsblk and /sys/block/nbdX/size, but not
in parted for a mounted filesystem.

Unmapping and remapping the NBD device shows the size in parted.

Thank you for any help
--
Alex Gorbachev
Storcium
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

2018-03-10 Thread Jason Dillaman

On Sat, Mar 10, 2018 at 10:11 AM, shadow_lin  wrote:
> Hi Jason,
>
>>As discussed in this thread, for active/passive, upon initiator
>>failover, we used the RBD exclusive-lock feature to blacklist the old
>>"active" iSCSI target gateway so that it cannot talk w/ the Ceph
>>cluster before new writes are accepted on the new target gateway.
>
> I can get during the new active target gateway was talking to rbd the old
> active target gateway cannot write because of the RBD exclusive-lock
> But after the new target gateway done the writes,if the old target gateway
> had some blocked io during the failover,cant it then get the lock and
> overwrite the new writes?

Negative -- it's blacklisted so it cannot talk to the cluster.

> PS:
> Petasan say they can do active/active iscsi with patched suse kernel.

I'll let them comment on these corner cases.

> 2018-03-10
> 
> shadowlin
>
> 
>
> 发件人：Jason Dillaman 
> 发送时间：2018-03-10 21:40
> 主题：Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock
> 收件人："shadow_lin"
> 抄送："Mike Christie","Lazuardi
> Nasution","Ceph Users"
>
> On Sat, Mar 10, 2018 at 7:42 AM, shadow_lin  wrote:
>> Hi Mike,
>> So for now only suse kernel with target_rbd_core and tcmu-runner can run
>> active/passive multipath safely?
>
> Negative, the LIO / tcmu-runner implementation documented here [1] is
> safe for active/passive.
>
>> I am a newbie to iscsi. I think the stuck io get excuted cause overwrite
>> problem can happen with both active/active and active/passive.
>> What makes the active/passive safer than active/active?
>
> As discussed in this thread, for active/passive, upon initiator
> failover, we used the RBD exclusive-lock feature to blacklist the old
> "active" iSCSI target gateway so that it cannot talk w/ the Ceph
> cluster before new writes are accepted on the new target gateway.
>
>> What mechanism should be implement to avoid the problem with
>> active/passive
>> and active/active multipath?
>
> Active/passive it solved as discussed above. For active/active, we
> don't have a solution that is known safe under all failure conditions.
> If LIO supported MCS (multiple connections per session) instead of
> just MPIO (multipath IO), the initiator would provide enough context
> to the target to detect IOs from a failover situation.
>
>> 2018-03-10
>> 
>> shadowlin
>>
>> 
>>
>> 发件人：Mike Christie 
>> 发送时间：2018-03-09 00:54
>> 主题：Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock
>> 收件人："shadow_lin","Lazuardi
>> Nasution","Ceph Users"
>> 抄送：
>>
>> On 03/07/2018 09:24 AM, shadow_lin wrote:
>>> Hi Christie,
>>> Is it safe to use active/passive multipath with krbd with exclusive lock
>>> for lio/tgt/scst/tcmu?
>>
>> No. We tried to use lio and krbd initially, but there is a issue where
>> IO might get stuck in the target/block layer and get executed after new
>> IO. So for lio, tgt and tcmu it is not safe as is right now. We could
>> add some code tcmu's file_example handler which can be used with krbd so
>> it works like the rbd one.
>>
>> I do know enough about SCST right now.
>>
>>
>>> Is it safe to use active/active multipath If use suse kernel with
>>> target_core_rbd?
>>> Thanks.
>>>
>>> 2018-03-07
>>> 
>>> shadowlin
>>>
>>> 
>>>
>>> *发件人：*Mike Christie 
>>> *发送时间：*2018-03-07 03:51
>>> *主题：*Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD
>>> Exclusive Lock
>>> *收件人：*"Lazuardi Nasution","Ceph
>>> Users"
>>> *抄送：*
>>>
>>> On 03/06/2018 01:17 PM, Lazuardi Nasution wrote:
>>> > Hi,
>>> >
>>> > I want to do load balanced multipathing (multiple iSCSI
>>> gateway/exporter
>>> > nodes) of iSCSI backed with RBD images. Should I disable exclusive
>>> lock
>>> > feature? What if I don't disable that feature? I'm using TGT
>>> (manual
>>> > way) since I get so many CPU stuck error messages when I was using
>>> LIO.
>>> >
>>>
>>> You are using LIO/TGT with krbd right?
>>>
>>> You cannot or shouldn't do active/active multipathing. If you have
>>> the
>>> lock enabled then it bounces between paths for each IO and will be
>>> slow.
>>> If you do not have it enabled then you can end up with stale IO
>>> overwriting current data.
>>>
>>>
>>>
>>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>>

Re: [ceph-users] (no subject)

2018-03-10 Thread Oliver Freyermuth

Hi Nathan,

this indeed appears to be a Gentoo-specific issue. 
They install the file at:
/usr/libexec/ceph/ceph-osd-prestart.sh
instead of
/usr/lib/ceph/ceph-osd-prestart.sh

It depends on how you strongly you follow FHS ( 
http://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s07.html )
which is the actual correct place to be used. It seems Gentoo packagers took 
FHS 3.0 seriously to decide where to install things,
but did not patch the code accordingly, so this surely warrants a Gentoo bug. 
Distro packagers should then decide whether they want to change install 
location (and rather follow Ceph upstream than FHS 3.0),
or patch the code to enforce FHS 3.0. 

https://bugs.gentoo.org/632028 is also related to that. 

Cheers,
Oliver

Am 11.03.2018 um 00:34 schrieb Nathan Dehnel:
> Trying to create an OSD:
> 
> gentooserver ~ # ceph-volume lvm create --data /dev/sdb
> Running command: ceph-authtool --gen-print-key
> Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring 
> /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 
> e70500fe-0d51-48c3-a607-414957886726
> Running command: vgcreate --force --yes 
> ceph-a736559a-92d1-483e-9289-d2c7feed510f /dev/sdb
>  stdout: Volume group "ceph-a736559a-92d1-483e-9289-d2c7feed510f" 
> successfully created
> Running command: lvcreate --yes -l 100%FREE -n 
> osd-block-e70500fe-0d51-48c3-a607-414957886726 
> ceph-a736559a-92d1-483e-9289-d2c7feed510f
>  stdout: Logical volume "osd-block-e70500fe-0d51-48c3-a607-414957886726" 
> created.
> Running command: ceph-authtool --gen-print-key
> Running command: mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
> Running command: chown -R ceph:ceph /dev/dm-0
> Running command: ln -s 
> /dev/ceph-a736559a-92d1-483e-9289-d2c7feed510f/osd-block-e70500fe-0d51-48c3-a607-414957886726
>  /var/lib/ceph/osd/ceph-0/block
> Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring 
> /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o 
> /var/lib/ceph/osd/ceph-0/activate.monmap
>  stderr: got monmap epoch 1
> Running command: ceph-authtool /var/lib/ceph/osd/ceph-0/keyring 
> --create-keyring --name osd.0 --add-key 
> AQBEZqRalRoRCBAA03R6VshykLcZjMgQnFKDtg==
>  stdout: creating /var/lib/ceph/osd/ceph-0/keyring
>  stdout: added entity osd.0 auth auth(auid = 18446744073709551615 
> key=AQBEZqRalRoRCBAA03R6VshykLcZjMgQnFKDtg== with 0 caps)
> Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring
> Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/
> Running command: ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs 
> -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data 
> /var/lib/ceph/osd/ceph-0/ --osd-uuid e70500fe-0d51-48c3-a607-414957886726 
> --setuser ceph --setgroup ceph
> --> ceph-volume lvm prepare successful for: /dev/sdb
> Running command: ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev 
> /dev/ceph-a736559a-92d1-483e-9289-d2c7feed510f/osd-block-e70500fe-0d51-48c3-a607-414957886726
>  --path /var/lib/ceph/osd/ceph-0
> Running command: ln -snf 
> /dev/ceph-a736559a-92d1-483e-9289-d2c7feed510f/osd-block-e70500fe-0d51-48c3-a607-414957886726
>  /var/lib/ceph/osd/ceph-0/block
> Running command: chown -R ceph:ceph /dev/dm-0
> Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
> Running command: systemctl enable 
> ceph-volume@lvm-0-e70500fe-0d51-48c3-a607-414957886726
>  stderr: Created symlink 
> /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-e70500fe-0d51-48c3-a607-414957886726.service
>  → /lib/systemd/system/ceph-volume@.service.
> Running command: systemctl start ceph-osd@0
>  stderr: Job for ceph-osd@0.service failed because the control process exited 
> with error code.
> See "systemctl status ceph-osd@0.service" and "journalctl -xe" for details.
> --> Was unable to complete a new OSD, will rollback changes
> --> OSD will be fully purged from the cluster, because the ID was generated
> Running command: ceph osd purge osd.0 --yes-i-really-mean-it
>  stderr: purged osd.0
> -->  RuntimeError: command returned non-zero exit status: 1
> 
> journalctl -xe
> -- Unit ceph-osd@0.service has begun starting up.
> Mar 10 17:14:34 gentooserver systemd[3977]: ceph-osd@0.service: Failed to 
> execute command: No such file or directory
> Mar 10 17:14:34 gentooserver systemd[3977]: ceph-osd@0.service: Failed at 
> step EXEC spawning /usr/lib/ceph/ceph-osd-prestart.sh: No such file or 
> directory
> -- Subject: Process /usr/lib/ceph/ceph-osd-prestart.sh could not be executed
> -- Defined-By: systemd
> -- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
> -- 
> -- The process /usr/lib/ceph/ceph-osd-prestart.sh could not be executed and 
> failed.
> -- 
> -- The error number returned by this process is 2.
> Mar 10 17:14:34 gentooserver systemd[1]: ceph-osd@0.service: Control process 
> exited, code=exited status=203
> Mar 10 17:14:34 gentooserver systemd[1]: ceph-osd@0.service: Failed with 
>

[ceph-users] (no subject)

2018-03-10 Thread Nathan Dehnel

Trying to create an OSD:

gentooserver ~ # ceph-volume lvm create --data /dev/sdb
Running command: ceph-authtool --gen-print-key
Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new
e70500fe-0d51-48c3-a607-414957886726
Running command: vgcreate --force --yes
ceph-a736559a-92d1-483e-9289-d2c7feed510f /dev/sdb
 stdout: Volume group "ceph-a736559a-92d1-483e-9289-d2c7feed510f"
successfully created
Running command: lvcreate --yes -l 100%FREE -n
osd-block-e70500fe-0d51-48c3-a607-414957886726
ceph-a736559a-92d1-483e-9289-d2c7feed510f
 stdout: Logical volume "osd-block-e70500fe-0d51-48c3-a607-414957886726"
created.
Running command: ceph-authtool --gen-print-key
Running command: mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
Running command: chown -R ceph:ceph /dev/dm-0
Running command: ln -s
/dev/ceph-a736559a-92d1-483e-9289-d2c7feed510f/osd-block-e70500fe-0d51-48c3-a607-414957886726
/var/lib/ceph/osd/ceph-0/block
Running command: ceph --cluster ceph --name client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o
/var/lib/ceph/osd/ceph-0/activate.monmap
 stderr: got monmap epoch 1
Running command: ceph-authtool /var/lib/ceph/osd/ceph-0/keyring
--create-keyring --name osd.0 --add-key
AQBEZqRalRoRCBAA03R6VshykLcZjMgQnFKDtg==
 stdout: creating /var/lib/ceph/osd/ceph-0/keyring
 stdout: added entity osd.0 auth auth(auid = 18446744073709551615
key=AQBEZqRalRoRCBAA03R6VshykLcZjMgQnFKDtg== with 0 caps)
Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring
Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/
Running command: ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs
-i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile -
--osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid
e70500fe-0d51-48c3-a607-414957886726 --setuser ceph --setgroup ceph
--> ceph-volume lvm prepare successful for: /dev/sdb
Running command: ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev
/dev/ceph-a736559a-92d1-483e-9289-d2c7feed510f/osd-block-e70500fe-0d51-48c3-a607-414957886726
--path /var/lib/ceph/osd/ceph-0
Running command: ln -snf
/dev/ceph-a736559a-92d1-483e-9289-d2c7feed510f/osd-block-e70500fe-0d51-48c3-a607-414957886726
/var/lib/ceph/osd/ceph-0/block
Running command: chown -R ceph:ceph /dev/dm-0
Running command: chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
Running command: systemctl enable
ceph-volume@lvm-0-e70500fe-0d51-48c3-a607-414957886726
 stderr: Created symlink
/etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-e70500fe-0d51-48c3-a607-414957886726.service
→ /lib/systemd/system/ceph-volume@.service.
Running command: systemctl start ceph-osd@0
 stderr: Job for ceph-osd@0.service failed because the control process
exited with error code.
See "systemctl status ceph-osd@0.service" and "journalctl -xe" for details.
--> Was unable to complete a new OSD, will rollback changes
--> OSD will be fully purged from the cluster, because the ID was generated
Running command: ceph osd purge osd.0 --yes-i-really-mean-it
 stderr: purged osd.0
-->  RuntimeError: command returned non-zero exit status: 1

journalctl -xe
-- Unit ceph-osd@0.service has begun starting up.
Mar 10 17:14:34 gentooserver systemd[3977]: ceph-osd@0.service: Failed to
execute command: No such file or directory
Mar 10 17:14:34 gentooserver systemd[3977]: ceph-osd@0.service: Failed at
step EXEC spawning /usr/lib/ceph/ceph-osd-prestart.sh: No such file or
directory
-- Subject: Process /usr/lib/ceph/ceph-osd-prestart.sh could not be executed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- The process /usr/lib/ceph/ceph-osd-prestart.sh could not be executed and
failed.
-- 
-- The error number returned by this process is 2.
Mar 10 17:14:34 gentooserver systemd[1]: ceph-osd@0.service: Control
process exited, code=exited status=203
Mar 10 17:14:34 gentooserver systemd[1]: ceph-osd@0.service: Failed with
result 'exit-code'.
Mar 10 17:14:34 gentooserver systemd[1]: Failed to start Ceph object
storage daemon osd.0.
-- Subject: Unit ceph-osd@0.service has failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit ceph-osd@0.service has failed.
-- 
-- The result is RESULT.

Why is this file missing? Should I file a bug with my distro's packager?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] New Ceph cluster design

2018-03-10 Thread Vincent Godin

Hi,

As i understand it, you'll have one RAID1 of two SSDs for 12 HDDs. A
WAL is used for all writes on your host. If you have good SSDs, they
can handle 450-550 MBpsc. Your 12 HDDs SATA can handle 12 x 100 MBps
that is to say 1200 GBps. So your RAID 1 will be the bootleneck with
this design. A good design would be to have one SSD for 4 or 5 HDD. In
your case, the best option would be to start with 3 SSDs for 12 HDDs
to have a balances node. Don't forget to choose SSD with a high WDPD
ratio (>10)

The network needs of your node depend of the bandwith of your disks.
As explain over, your 12 HDDs can handle 1200 GBps so you need to have
a public and a private network that can handle it. In your case, a
minimum a two 10 Gbps networks for per node are needed. If you need
redondancy, just use two LACP networks with each having two 10 Gbps
links. The scrub or deep scrub operations will not have a significant
impact on your network but on your disks utilisations. So you need to
plan them during low usage by your clients
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

2018-03-10 Thread shadow_lin

Hi Jason,

>As discussed in this thread, for active/passive, upon initiator 
>failover, we used the RBD exclusive-lock feature to blacklist the old 
>"active" iSCSI target gateway so that it cannot talk w/ the Ceph 
>cluster before new writes are accepted on the new target gateway. 

I can get during the new active target gateway was talking to rbd the old 
active target gateway cannot write because of the RBD exclusive-lock 
But after the new target gateway done the writes,if the old target gateway had 
some blocked io during the failover,cant it then get the lock and overwrite the 
new writes?

PS:
Petasan say they can do active/active iscsi with patched suse kernel.

2018-03-10 



shadowlin




发件人：Jason Dillaman 
发送时间：2018-03-10 21:40
主题：Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock
收件人："shadow_lin"
抄送："Mike Christie","Lazuardi 
Nasution","Ceph Users"

On Sat, Mar 10, 2018 at 7:42 AM, shadow_lin  wrote: 
> Hi Mike, 
> So for now only suse kernel with target_rbd_core and tcmu-runner can run 
> active/passive multipath safely? 

Negative, the LIO / tcmu-runner implementation documented here [1] is 
safe for active/passive. 

> I am a newbie to iscsi. I think the stuck io get excuted cause overwrite 
> problem can happen with both active/active and active/passive. 
> What makes the active/passive safer than active/active? 

As discussed in this thread, for active/passive, upon initiator 
failover, we used the RBD exclusive-lock feature to blacklist the old 
"active" iSCSI target gateway so that it cannot talk w/ the Ceph 
cluster before new writes are accepted on the new target gateway. 

> What mechanism should be implement to avoid the problem with active/passive 
> and active/active multipath? 

Active/passive it solved as discussed above. For active/active, we 
don't have a solution that is known safe under all failure conditions. 
If LIO supported MCS (multiple connections per session) instead of 
just MPIO (multipath IO), the initiator would provide enough context 
to the target to detect IOs from a failover situation. 

> 2018-03-10 
>  
> shadowlin 
> 
>  
> 
> 发件人：Mike Christie  
> 发送时间：2018-03-09 00:54 
> 主题：Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock 
> 收件人："shadow_lin","Lazuardi 
> Nasution","Ceph Users" 
> 抄送： 
> 
> On 03/07/2018 09:24 AM, shadow_lin wrote: 
>> Hi Christie, 
>> Is it safe to use active/passive multipath with krbd with exclusive lock 
>> for lio/tgt/scst/tcmu? 
> 
> No. We tried to use lio and krbd initially, but there is a issue where 
> IO might get stuck in the target/block layer and get executed after new 
> IO. So for lio, tgt and tcmu it is not safe as is right now. We could 
> add some code tcmu's file_example handler which can be used with krbd so 
> it works like the rbd one. 
> 
> I do know enough about SCST right now. 
> 
> 
>> Is it safe to use active/active multipath If use suse kernel with 
>> target_core_rbd? 
>> Thanks. 
>> 
>> 2018-03-07 
>>  
>> shadowlin 
>> 
>>  
>> 
>> *发件人：*Mike Christie  
>> *发送时间：*2018-03-07 03:51 
>> *主题：*Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD 
>> Exclusive Lock 
>> *收件人：*"Lazuardi Nasution","Ceph 
>> Users" 
>> *抄送：* 
>> 
>> On 03/06/2018 01:17 PM, Lazuardi Nasution wrote: 
>> > Hi, 
>> > 
>> > I want to do load balanced multipathing (multiple iSCSI 
>> gateway/exporter 
>> > nodes) of iSCSI backed with RBD images. Should I disable exclusive 
>> lock 
>> > feature? What if I don't disable that feature? I'm using TGT (manual 
>> > way) since I get so many CPU stuck error messages when I was using 
>> LIO. 
>> > 
>> 
>> You are using LIO/TGT with krbd right? 
>> 
>> You cannot or shouldn't do active/active multipathing. If you have the 
>> lock enabled then it bounces between paths for each IO and will be 
>> slow. 
>> If you do not have it enabled then you can end up with stale IO 
>> overwriting current data. 
>> 
>> 
>> 
>> 
> 
> 
> ___ 
> ceph-users mailing list 
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
> 

[1] http://docs.ceph.com/docs/master/rbd/iscsi-overview/ 

--  
Jason ___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

2018-03-10 Thread Jason Dillaman

On Sat, Mar 10, 2018 at 7:42 AM, shadow_lin  wrote:
> Hi Mike,
> So for now only suse kernel with target_rbd_core and tcmu-runner can run
> active/passive multipath safely?

Negative, the LIO / tcmu-runner implementation documented here [1] is
safe for active/passive.

> I am a newbie to iscsi. I think the stuck io get excuted cause overwrite
> problem can happen with both active/active and active/passive.
> What makes the active/passive safer than active/active?

As discussed in this thread, for active/passive, upon initiator
failover, we used the RBD exclusive-lock feature to blacklist the old
"active" iSCSI target gateway so that it cannot talk w/ the Ceph
cluster before new writes are accepted on the new target gateway.

> What mechanism should be implement to avoid the problem with active/passive
> and active/active multipath?

Active/passive it solved as discussed above. For active/active, we
don't have a solution that is known safe under all failure conditions.
If LIO supported MCS (multiple connections per session) instead of
just MPIO (multipath IO), the initiator would provide enough context
to the target to detect IOs from a failover situation.

> 2018-03-10
> 
> shadowlin
>
> 
>
> 发件人：Mike Christie 
> 发送时间：2018-03-09 00:54
> 主题：Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock
> 收件人："shadow_lin","Lazuardi
> Nasution","Ceph Users"
> 抄送：
>
> On 03/07/2018 09:24 AM, shadow_lin wrote:
>> Hi Christie,
>> Is it safe to use active/passive multipath with krbd with exclusive lock
>> for lio/tgt/scst/tcmu?
>
> No. We tried to use lio and krbd initially, but there is a issue where
> IO might get stuck in the target/block layer and get executed after new
> IO. So for lio, tgt and tcmu it is not safe as is right now. We could
> add some code tcmu's file_example handler which can be used with krbd so
> it works like the rbd one.
>
> I do know enough about SCST right now.
>
>
>> Is it safe to use active/active multipath If use suse kernel with
>> target_core_rbd?
>> Thanks.
>>
>> 2018-03-07
>> 
>> shadowlin
>>
>> 
>>
>> *发件人：*Mike Christie 
>> *发送时间：*2018-03-07 03:51
>> *主题：*Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD
>> Exclusive Lock
>> *收件人：*"Lazuardi Nasution","Ceph
>> Users"
>> *抄送：*
>>
>> On 03/06/2018 01:17 PM, Lazuardi Nasution wrote:
>> > Hi,
>> >
>> > I want to do load balanced multipathing (multiple iSCSI
>> gateway/exporter
>> > nodes) of iSCSI backed with RBD images. Should I disable exclusive
>> lock
>> > feature? What if I don't disable that feature? I'm using TGT (manual
>> > way) since I get so many CPU stuck error messages when I was using
>> LIO.
>> >
>>
>> You are using LIO/TGT with krbd right?
>>
>> You cannot or shouldn't do active/active multipathing. If you have the
>> lock enabled then it bounces between paths for each IO and will be
>> slow.
>> If you do not have it enabled then you can end up with stale IO
>> overwriting current data.
>>
>>
>>
>>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>

[1] http://docs.ceph.com/docs/master/rbd/iscsi-overview/

-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

2018-03-10 Thread shadow_lin

Hi Mike,  
So for now only suse kernel with target_rbd_core and tcmu-runner can run 
active/passive multipath safely?
I am a newbie to iscsi. I think the stuck io get excuted cause overwrite 
problem can happen with both active/active and active/passive.
What makes the active/passive safer than active/active? 
What mechanism should be implement to avoid the problem with active/passive and 
active/active multipath?
2018-03-10 

shadowlin

发件人：Mike Christie 
发送时间：2018-03-09 00:54
主题：Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock
收件人："shadow_lin","Lazuardi 
Nasution","Ceph Users"
抄送：

On 03/07/2018 09:24 AM, shadow_lin wrote: 
> Hi Christie, 
> Is it safe to use active/passive multipath with krbd with exclusive lock 
> for lio/tgt/scst/tcmu? 

No. We tried to use lio and krbd initially, but there is a issue where 
IO might get stuck in the target/block layer and get executed after new 
IO. So for lio, tgt and tcmu it is not safe as is right now. We could 
add some code tcmu's file_example handler which can be used with krbd so 
it works like the rbd one. 

I do know enough about SCST right now. 

> Is it safe to use active/active multipath If use suse kernel with 
> target_core_rbd? 
> Thanks. 
>   
> 2018-03-07 
>  
> shadowlin 
>   
>  
>  
> *发件人：*Mike Christie  
> *发送时间：*2018-03-07 03:51 
> *主题：*Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD 
> Exclusive Lock 
> *收件人：*"Lazuardi Nasution","Ceph 
> Users" 
> *抄送：* 
>   
> On 03/06/2018 01:17 PM, Lazuardi Nasution wrote:  
> > Hi,  
> >   
> > I want to do load balanced multipathing (multiple iSCSI 
> gateway/exporter  
> > nodes) of iSCSI backed with RBD images. Should I disable exclusive lock 
>  
> > feature? What if I don't disable that feature? I'm using TGT (manual  
> > way) since I get so many CPU stuck error messages when I was using LIO. 
>  
> >   
>   
> You are using LIO/TGT with krbd right?  
>   
> You cannot or shouldn't do active/active multipathing. If you have the  
> lock enabled then it bounces between paths for each IO and will be slow.  
> If you do not have it enabled then you can end up with stale IO  
> overwriting current data.  
>   
>   
>   
>  ___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [jewel] High fs_apply_latency osds

2018-03-10 Thread shadow_lin

Hi Chris,
The osds are running on arm nodes. Every node has a two core 1.5Ghz arm 32bit 
cpu and 2G ram and runs 2 osds.Hdd is 10TB  and journal  colocate with data on 
the same disk.
Drives are half full now,but the problem I described also happened when the 
hdds are empty. Filesystem is ext4 because I have some problem to run xfs for 
now.
I am trying to better balance the pg distrubition now to see if it can ease the 
high latency problem.

2018-03-10 


shadowlin




发件人：Chris Hoy Poy 
发送时间：2018-03-10 09:44
主题：Re: [ceph-users] [jewel] High fs_apply_latency osds
收件人："shadow_lin","ceph-users"
抄送：

Hi Shadowlin,


Can you describe your hardware ? Cpu/ram/harddrives involved etc


Also how your drives are set up?


How full are the drives? What filesystem is it?


Cheers
/chris






Sent from my SAMSUNG Galaxy S6 on the Telstra Mobile Network




 Original message 
From: shadow_lin  
Date: 10/3/18 1:41 am (GMT+08:00) 
To: ceph-users  
Subject: [ceph-users] [jewel] High fs_apply_latency osds 


Hi list,
During my write test,I find there are always some of the osds have high 
fs_apply_latency（1k-5kms,2-8times more than others）. At first I think it is 
caused by unbalanced pg distribution, but after I reweight the osds the problem 
hasn't gone away.
I looked into the osds with high latency and find one thing in common.There are 
alot of read activities on these osds.Result of iostat shows there are 400-500 
r/s and 2000-3000 rKB/s on these high latnecy osds while the normal osds have 
around 100 r/s 300-400 rKB/s.
I tried to restart osd daemon with high latency.It did became normal sometime 
but there will be another high latency osds come out.And the new high latency 
osd has the same high read activities.

What are these read activities for? Is there a way to lower the latency?

Thanks


2018-03-10



shadowlin




Message protected by MailGuard: e-mail anti-virus, anti-spam and content 
filtering.
http://www.mailguard.com.au

Report this message as spam  
 




Message protected by MailGuard: e-mail anti-virus, anti-spam and content 
filtering.
http://www.mailguard.com.au
 
 ___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

[ceph-users] rbd-nbd not resizing even after kernel tweaks

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

Re: [ceph-users] (no subject)

[ceph-users] (no subject)

Re: [ceph-users] New Ceph cluster design

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

Re: [ceph-users] [jewel] High fs_apply_latency osds

10 matches

Site Navigation

Mail list logo

Footer information