Re: [ceph-users] iSCSI over RBD

[Mike Christie]

On 01/19/2018 02:12 PM, Steven Vacaroaia wrote:
> Hi Joshua,
> 
> I was under the impression that kernel  3.10.0-693 will work with iscsi
>  

That kernel works with RHCS 2.5 and below. You need the rpms from that
or the matching upstream releases. Besides trying to dig out the
versions and matching things up, the problem with those releases is that
they were tech previewed or only supports linux initiators.

It looks like you are using the newer upstream tools or RHCS 3.0 tools.
For them you need the RHEL 7.5 beta or newer kernel or an upstream one.
For upstream all the patches got merged into the target layer
maintainer's tree yesterday. A new tcmu-runner release has been made.
And I just pushed a test kernel with all the patches based on 4.13 (4.14
had a bug in the login code which is being fixed still) to github, so
people do not have to wait for the next-next kernel release to come out.

Just give us a couple days for the kernel build to be done, to make the
needed ceph-iscsi-* release (current version will fail to create rbd
images with the current tcmu-runner release) and get the documentation
updated because some links are incorrect and some version info needs to
be updated.


> Unfortunately  I still cannot create a disk because qfull_time_out is
> not supported 
> 
> What am I missing / do it wrong ?
> 
> 2018-01-19 15:06:45,216 INFO [lun.py:601:add_dev_to_lio()] -
> (LUN.add_dev_to_lio) Adding image 'rbd.disk2' to LIO
> 2018-01-19 15:06:45,295ERROR [lun.py:634:add_dev_to_lio()] - Could
> not set LIO device attribute cmd_time_out/qfull_time_out for device:
> rbd.disk2. Kernel not supported. - error(Cannot find attribute:
> qfull_time_out)
> 2018-01-19 15:06:45,300ERROR [rbd-target-api:731:_disk()] - LUN
> alloc problem - Could not set LIO device attribute
> cmd_time_out/qfull_time_out for device: rbd.disk2. Kernel not supported.
> - error(Cannot find attribute: qfull_time_out)
> 
> 
> Many thanks
> 
> Steven
> 
> On 4 January 2018 at 22:40, Joshua Chen  > wrote:
> 
> Hello Steven,
>   I am using CentOS 7.4.1708 with kernel 3.10.0-693.el7.x86_64
>   and the following packages:
> 
> ceph-iscsi-cli-2.5-9.el7.centos.noarch.rpm
> ceph-iscsi-config-2.3-12.el7.centos.noarch.rpm
> libtcmu-1.3.0-0.4.el7.centos.x86_64.rpm
> libtcmu-devel-1.3.0-0.4.el7.centos.x86_64.rpm
> python-rtslib-2.1.fb64-2.el7.centos.noarch.rpm
> python-rtslib-doc-2.1.fb64-2.el7.centos.noarch.rpm
> targetcli-2.1.fb47-0.1.20170815.git5bf3517.el7.centos.noarch.rpm
> tcmu-runner-1.3.0-0.4.el7.centos.x86_64.rpm
> tcmu-runner-debuginfo-1.3.0-0.4.el7.centos.x86_64.rpm
> 
> 
> Cheers
> Joshua
> 
> 
> On Fri, Jan 5, 2018 at 2:14 AM, Steven Vacaroaia  > wrote:
> 
> Hi Joshua,
> 
> How did you manage to use iSCSI gateway ?
> I would like to do that but still waiting for a patched kernel 
> 
> What kernel/OS did you use and/or how did you patch it ?
> 
> Tahnsk
> Steven
> 
> On 4 January 2018 at 04:50, Joshua Chen
> >
> wrote:
> 
> Dear all,
>   Although I managed to run gwcli and created some iqns, or
> luns,
> but I do need some working config example so that my
> initiator could connect and get the lun.
> 
>   I am familiar with targetcli and I used to do the
> following ACL style connection rather than password, 
> the targetcli setting tree is here:
> 
> (or see this page
> )
> 
> #targetcli ls
> o- /
> 
> .
> [...]
>   o- backstores
> 
> ..
> [...]
>   | o- block
> 
> ..
> [Storage Objects: 1]
>   | | o- vmware_5t
> ..
> [/dev/rbd/rbd/vmware_5t (5.0TiB) write-thru activated]
>   | |   o- alua
> 
> ...
> [ALUA Groups: 1]
>   | | o- default_tg_pt_gp
> 
> ...
> [ALUA state: Active/optimized]
>   | o- fileio
> 
> 

Re: [ceph-users] Removing cache tier for RBD pool

[Mike Lovell]

On Tue, Jan 16, 2018 at 9:25 AM, Jens-U. Mozdzen  wrote:

> Hello Mike,
>
> Zitat von Mike Lovell :
>
>> On Mon, Jan 8, 2018 at 6:08 AM, Jens-U. Mozdzen  wrote:
>>
>>> Hi *,
>>> [...]
>>> 1. Does setting the cache mode to "forward" lead to above situation of
>>> remaining locks on hot-storage pool objects? Maybe the clients' unlock
>>> requests are forwarded to the cold-storage pool, leaving the hot-storage
>>> objects locked? If so, this should be documented and it'd seem impossible
>>> to cleanly remove a cache tier during live operations.
>>>
>>> 2. What is the significant difference between "rados
>>> cache-flush-evict-all" and separate "cache-flush" and "cache-evict"
>>> cycles?
>>> Or is it some implementation error that leads to those "file not found"
>>> errors with "cache-flush-evict-all", while the manual cycles work
>>> successfully?
>>>
>>> Thank you for any insight you might be able to share.
>>>
>>> Regards,
>>> Jens
>>>
>>>
>> i've removed a cache tier in environments a few times. the only locked
>> files i ran into were the rbd_directory and rbd_header objects for each
>> volume. the rbd_headers for each rbd volume are locked as long as the vm
>> is
>> running. every time i've tried to remove a cache tier, i shutdown all of
>> the vms before starting the procedure and there wasn't any problem getting
>> things flushed+evicted. so i can't really give any further insight into
>> what might have happened other than it worked for me. i set the cache-mode
>> to forward everytime before flushing and evicting objects.
>>
>
> while your report doesn't confirm my suspicion expressed in my question 1,
> it at least is another example where removing the cache worked *after
> stopping all instances*, rather than "live". If, OTOH, this limitation is
> confirmed, it should be added to the docs.
>
> Out of curiosity: Do you have any other users for the pool? After stopping
> all VMs (and the image-related services on our Openstack control nodes), my
> pool was without access, so I saw no need to put the caching tier to
> "forward" mode.


the pools were exclusively for rbd and vms. due to other constraints of our
system, i had to shut everything down. i think that if you were to have the
pool in forward mode, shut down a single vm, flush the objects related to
its rbd volumes, then start it that it wont promote objects from those rbd
volumes again. you could, in theory, then power cycle the vms one at a time
and be able to remove the cache tier. thats probably not a practical case
though. the cold tier in our system didn't have the iops capacity to run
all of the vms without the tier so we just shut them all down.

i don't think there really is a significant technical difference between
>> the cache-flush-evict-all command and doing separate cache-flush and
>> cache-evict on individual objects. my understanding is
>> cache-flush-evict-all is just a short cut to getting everything in the
>> cache flushed and evicted. did the cache-flush-evict-all error on some
>> objects where the separate operations succeeded? you're description
>> doesn't
>> say if there was but then you say you used both styles during your second
>> attempt.
>>
>
> It was actually that every run of "cache-flush-evict-all" did report
> errors on all remaining objects, while running the loop manually (issue
> flush for every objects, then issue evict for every remaining object) did
> work flawlessly. That's why my question 2 came up.
>
> The objects I saw seemed related to the images stored in the pool, not any
> "management data" (like the suggested hitset persistence).


 hrm. i don't think i've seen that then. it has been several months since
i've done it though so i might be forgetting.

on a different note, you say that your cluster is on 12.2 but the cache
>> tiers were created on an earlier version. which version was the cache tier
>> created on? how well did the upgrade process work? i am curious since the
>> production clusters i have using a cache tier are still on 10.2 and i'm
>> about to begin the process of testing the upgrade to 12.2. any info on
>> that
>> experience you can share would be helpful.
>>
>
> I *believe* the cache was created on 10.2, but cannot recall for sure. I
> remember having had similar problems in those earlier days with a previous
> instance of that caching tier, but many root causes were "on my side of the
> keyboard". The cache tier I was trying to remove recently was created from
> scratch after those problems, and upgrading to the latest release via the
> recommended intermediate version steps was problem-free. At least when
> focusing on the subject of cache tiers ;)
>

that's good to hear. when we did an upgrade from 0.94.7 to 10.2.5 using
cache tiers that were created on a version prior to 0.94.5 we triggered a
bug causing all of the ssd osds to crash. that led to a 30 hour day. i want
to start the process of upgrading to luminous but 

Re: [ceph-users] ceph df shows 100% used

[QR]

'MAX AVAIL' in the 'ceph df' output represents the amount of data that can be 
used before the first OSD becomes full, and not the sum of all free space 
across a set of OSDs.
 原始邮件 发件人: Webert de 
Souza Lima收件人: 
ceph-users发送时间: 2018年1月19日(周五) 20:20主题: Re: 
[ceph-users] ceph df shows 100% usedWhile it seemed to be solved yesterday, 
today the %USED has grown a lot again. See:
~# ceph osd df tree http://termbin.com/0zhk

~# ceph df detail
http://termbin.com/thox

94% USED while there is about 21TB worth of data, size = 2 menas ~42TB RAW 
Usage, but the OSDs in that root sum ~70TB available space.

Regards,
Webert LimaDevOps Engineer at MAV TecnologiaBelo Horizonte - BrasilIRC NICK - 
WebertRLZ
On Thu, Jan 18, 2018 at 8:21 PM, Webert de Souza Lima  
wrote:
With the help of robbat2 and llua on IRC channel I was able to solve this 
situation by taking down the 2-OSD only hosts.
After crush reweighting OSDs 8 and 23 from host mia1-master-fe02 to 0, ceph df 
showed the expected storage capacity usage (about 70%)


With this in mind, those guys have told me that it is due the cluster beeing 
uneven and unable to balance properly. It makes sense and it worked.
But for me it is still a very unexpected bahaviour for ceph to say that the 
pools are 100% full and Available Space is 0.
There were 3 hosts and repl. size = 2, if the host with only 2 OSDs were full 
(it wasn't), ceph could still use space from OSDs from the other hosts.
Regards,
Webert LimaDevOps Engineer at MAV TecnologiaBelo Horizonte - BrasilIRC NICK - 
WebertRLZ
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] iSCSI over RBD

[Brady Deetz]

I too experienced this with that kernel as well as the elrepo kernel.

On Jan 19, 2018 2:13 PM, "Steven Vacaroaia"  wrote:

Hi Joshua,

I was under the impression that kernel  3.10.0-693 will work with iscsi

Unfortunately  I still cannot create a disk because qfull_time_out is not
supported

What am I missing / do it wrong ?

2018-01-19 15:06:45,216 INFO [lun.py:601:add_dev_to_lio()] -
(LUN.add_dev_to_lio) Adding image 'rbd.disk2' to LIO
2018-01-19 15:06:45,295ERROR [lun.py:634:add_dev_to_lio()] - Could not
set LIO device attribute cmd_time_out/qfull_time_out for device: rbd.disk2.
Kernel not supported. - error(Cannot find attribute: qfull_time_out)
2018-01-19 15:06:45,300ERROR [rbd-target-api:731:_disk()] - LUN alloc
problem - Could not set LIO device attribute cmd_time_out/qfull_time_out
for device: rbd.disk2. Kernel not supported. - error(Cannot find attribute:
qfull_time_out)


Many thanks

Steven

On 4 January 2018 at 22:40, Joshua Chen  wrote:

> Hello Steven,
>   I am using CentOS 7.4.1708 with kernel 3.10.0-693.el7.x86_64
>   and the following packages:
>
> ceph-iscsi-cli-2.5-9.el7.centos.noarch.rpm
> ceph-iscsi-config-2.3-12.el7.centos.noarch.rpm
> libtcmu-1.3.0-0.4.el7.centos.x86_64.rpm
> libtcmu-devel-1.3.0-0.4.el7.centos.x86_64.rpm
> python-rtslib-2.1.fb64-2.el7.centos.noarch.rpm
> python-rtslib-doc-2.1.fb64-2.el7.centos.noarch.rpm
> targetcli-2.1.fb47-0.1.20170815.git5bf3517.el7.centos.noarch.rpm
> tcmu-runner-1.3.0-0.4.el7.centos.x86_64.rpm
> tcmu-runner-debuginfo-1.3.0-0.4.el7.centos.x86_64.rpm
>
>
> Cheers
> Joshua
>
>
> On Fri, Jan 5, 2018 at 2:14 AM, Steven Vacaroaia  wrote:
>
>> Hi Joshua,
>>
>> How did you manage to use iSCSI gateway ?
>> I would like to do that but still waiting for a patched kernel
>>
>> What kernel/OS did you use and/or how did you patch it ?
>>
>> Tahnsk
>> Steven
>>
>> On 4 January 2018 at 04:50, Joshua Chen 
>> wrote:
>>
>>> Dear all,
>>>   Although I managed to run gwcli and created some iqns, or luns,
>>> but I do need some working config example so that my initiator could
>>> connect and get the lun.
>>>
>>>   I am familiar with targetcli and I used to do the following ACL style
>>> connection rather than password,
>>> the targetcli setting tree is here:
>>>
>>> (or see this page
>>> )
>>>
>>> #targetcli ls
>>> o- / 
>>> . [...]
>>>   o- backstores ..
>>> 
>>> [...]
>>>   | o- block ..
>>> 
>>> [Storage Objects: 1]
>>>   | | o- vmware_5t 
>>> ..
>>> [/dev/rbd/rbd/vmware_5t (5.0TiB) write-thru activated]
>>>   | |   o- alua ..
>>> .
>>> [ALUA Groups: 1]
>>>   | | o- default_tg_pt_gp ..
>>> . [ALUA state: Active/optimized]
>>>   | o- fileio ..
>>> ...
>>> [Storage Objects: 0]
>>>   | o- pscsi ..
>>> 
>>> [Storage Objects: 0]
>>>   | o- ramdisk ..
>>> ..
>>> [Storage Objects: 0]
>>>   | o- user:rbd ..
>>> .
>>> [Storage Objects: 0]
>>>   o- iscsi 
>>>  [Targets: 1]
>>>   | o- iqn.2017-12.asiaa.cephosd1:vmware5t
>>> ...
>>> [TPGs: 1]
>>>   |   o- tpg1 ..
>>> 
>>> [gen-acls, no-auth]
>>>   | o- acls ..
>>> ...
>>> [ACLs: 12]
>>>   | | o- iqn.1994-05.com.redhat:15dbed23be9e
>>> ..
>>> [Mapped LUNs: 1]
>>>   | | | o- mapped_lun0 ..
>>> ... [lun0 block/vmware_5t
>>> (rw)]
>>>   | | o- iqn.1994-05.com.redhat:15dbed23be9e-ovirt1
>>> ... [Mapped
>>> LUNs: 1]
>>>   | | | o- mapped_lun0 ..
>>> 

Re: [ceph-users] iSCSI over RBD

[Steven Vacaroaia]

Hi Joshua,

I was under the impression that kernel  3.10.0-693 will work with iscsi

Unfortunately  I still cannot create a disk because qfull_time_out is not
supported

What am I missing / do it wrong ?

2018-01-19 15:06:45,216 INFO [lun.py:601:add_dev_to_lio()] -
(LUN.add_dev_to_lio) Adding image 'rbd.disk2' to LIO
2018-01-19 15:06:45,295ERROR [lun.py:634:add_dev_to_lio()] - Could not
set LIO device attribute cmd_time_out/qfull_time_out for device: rbd.disk2.
Kernel not supported. - error(Cannot find attribute: qfull_time_out)
2018-01-19 15:06:45,300ERROR [rbd-target-api:731:_disk()] - LUN alloc
problem - Could not set LIO device attribute cmd_time_out/qfull_time_out
for device: rbd.disk2. Kernel not supported. - error(Cannot find attribute:
qfull_time_out)


Many thanks

Steven

On 4 January 2018 at 22:40, Joshua Chen  wrote:

> Hello Steven,
>   I am using CentOS 7.4.1708 with kernel 3.10.0-693.el7.x86_64
>   and the following packages:
>
> ceph-iscsi-cli-2.5-9.el7.centos.noarch.rpm
> ceph-iscsi-config-2.3-12.el7.centos.noarch.rpm
> libtcmu-1.3.0-0.4.el7.centos.x86_64.rpm
> libtcmu-devel-1.3.0-0.4.el7.centos.x86_64.rpm
> python-rtslib-2.1.fb64-2.el7.centos.noarch.rpm
> python-rtslib-doc-2.1.fb64-2.el7.centos.noarch.rpm
> targetcli-2.1.fb47-0.1.20170815.git5bf3517.el7.centos.noarch.rpm
> tcmu-runner-1.3.0-0.4.el7.centos.x86_64.rpm
> tcmu-runner-debuginfo-1.3.0-0.4.el7.centos.x86_64.rpm
>
>
> Cheers
> Joshua
>
>
> On Fri, Jan 5, 2018 at 2:14 AM, Steven Vacaroaia  wrote:
>
>> Hi Joshua,
>>
>> How did you manage to use iSCSI gateway ?
>> I would like to do that but still waiting for a patched kernel
>>
>> What kernel/OS did you use and/or how did you patch it ?
>>
>> Tahnsk
>> Steven
>>
>> On 4 January 2018 at 04:50, Joshua Chen 
>> wrote:
>>
>>> Dear all,
>>>   Although I managed to run gwcli and created some iqns, or luns,
>>> but I do need some working config example so that my initiator could
>>> connect and get the lun.
>>>
>>>   I am familiar with targetcli and I used to do the following ACL style
>>> connection rather than password,
>>> the targetcli setting tree is here:
>>>
>>> (or see this page
>>> )
>>>
>>> #targetcli ls
>>> o- / 
>>> . [...]
>>>   o- backstores ..
>>> 
>>> [...]
>>>   | o- block ..
>>> 
>>> [Storage Objects: 1]
>>>   | | o- vmware_5t 
>>> ..
>>> [/dev/rbd/rbd/vmware_5t (5.0TiB) write-thru activated]
>>>   | |   o- alua ..
>>> .
>>> [ALUA Groups: 1]
>>>   | | o- default_tg_pt_gp ..
>>> . [ALUA state: Active/optimized]
>>>   | o- fileio ..
>>> ...
>>> [Storage Objects: 0]
>>>   | o- pscsi ..
>>> 
>>> [Storage Objects: 0]
>>>   | o- ramdisk ..
>>> ..
>>> [Storage Objects: 0]
>>>   | o- user:rbd ..
>>> .
>>> [Storage Objects: 0]
>>>   o- iscsi 
>>>  [Targets: 1]
>>>   | o- iqn.2017-12.asiaa.cephosd1:vmware5t
>>> ...
>>> [TPGs: 1]
>>>   |   o- tpg1 ..
>>> 
>>> [gen-acls, no-auth]
>>>   | o- acls ..
>>> ...
>>> [ACLs: 12]
>>>   | | o- iqn.1994-05.com.redhat:15dbed23be9e
>>> ..
>>> [Mapped LUNs: 1]
>>>   | | | o- mapped_lun0 ..
>>> ... [lun0 block/vmware_5t
>>> (rw)]
>>>   | | o- iqn.1994-05.com.redhat:15dbed23be9e-ovirt1
>>> ... [Mapped
>>> LUNs: 1]
>>>   | | | o- mapped_lun0 ..
>>> ... [lun0 block/vmware_5t
>>> (rw)]
>>>   | | o- iqn.1994-05.com.redhat:2af344ba6ae5-ceph-admin-test
>>> 

[ceph-users] Missing udev rule for FC disks (Re: mkjournal error creating journal ... : (13) Permission denied)

[Fulvio Galeazzi]

Hallo,
 apologies for reviving an old thread, but I just wasted again one 
full day as I had forgotten about this issue...


   To recap, udev rules nowadays do not (at least in my case, I am 
using disks served via FiberChannel) create the links 
/dev/disk/by-partuuid that ceph-disk expects.


I see the "culprit" is this line in (am on CentOS, but Ubuntu has the 
same issue): /usr/lib/udev/rules.d/60-persistent-storage.rules


.
# skip rules for inappropriate block devices
KERNEL=="fd*|mtd*|nbd*|gnbd*|btibm*|dm-*|md*|zram*|mmcblk[0-9]*rpmb", 
GOTO="persistent_storage_end"

.

stating that multipath'ed devices (called dm-*) should be skipped.


I can happily live with the file mentioned below, but was wondering:

- is there any hope that newer kernels may handle multipath devices
  properly?

- as an alternative, could it be possible to update ceph-disk
  such that symlinks for journal use some other
  /dev/disk/by-?

   Thanks!

Fulvio

On 3/16/2017 5:59 AM, Gunwoo Gim wrote:
  Thank you so much Peter. The 'udevadm trigger' after 'partprobe' 
triggered the udev rules and I've found out that even before the udev 
ruleset triggers the owner is already ceph:ceph.


  I've dug into ceph-disk a little more and found out that there is a 
symbolic link of 
/dev/disk/by-partuuid/120c536d-cb30-4cea-b607-dd347022a497 at 
[/dev/mapper/vg--hdd1-lv--hdd1p1(the_filestore_osd)]/journal and the 
source doesn't exist. though it exists in /dev/disk/by-parttypeuuid 
which has been populated by /lib/udev/rules.d/60-ceph-by-parttypeuuid.rules


  So I added this in /lib/udev/rules.d/60-ceph-by-parttypeuuid.rules:
# when ceph-disk prepares a filestore osd it makes a symbolic link by 
disk/by-partuuid but LVM2 doesn't seem to populate /dev/disk/by-partuuid.
ENV{ID_PART_ENTRY_SCHEME}=="gpt", ENV{ID_PART_ENTRY_TYPE}=="?*", 
ENV{ID_PART_ENTRY_UUID}=="?*", 
SYMLINK+="disk/by-partuuid/$env{ID_PART_ENTRY_UUID}"

  And finally got the osds all up and in. :D

  Yeah, It wasn't actually a permission problem, but the link just 
wasn't existing.



~ # ceph-disk -v activate /dev/mapper/vg--hdd1-lv--hdd1p1
...
mount: Mounting /dev/mapper/vg--hdd1-lv--hdd1p1 on 
/var/lib/ceph/tmp/mnt.ECAifr with options noatime,largeio,inode64,swalloc
command_check_call: Running command: /bin/mount -t xfs -o 
noatime,largeio,inode64,swalloc -- /dev/mapper/vg--hdd1-lv--hdd1p1 
/var/lib/ceph/tmp/mnt.ECAifr

mount: DIGGIN ls -al /var/lib/ceph/tmp/mnt.ECAifr
mount: DIGGIN total 36
drwxr-xr-x 3 ceph ceph  174 Mar 14 11:51 .
drwxr-xr-x 6 ceph ceph 4096 Mar 16 11:30 ..
-rw-r--r-- 1 root root  202 Mar 16 11:19 activate.monmap
-rw-r--r-- 1 ceph ceph   37 Mar 14 11:45 ceph_fsid
drwxr-xr-x 3 ceph ceph   39 Mar 14 11:51 current
-rw-r--r-- 1 ceph ceph   37 Mar 14 11:45 fsid
lrwxrwxrwx 1 ceph ceph   58 Mar 14 11:45 journal -> 
/dev/disk/by-partuuid/120c536d-cb30-4cea-b607-dd347022a497

-rw-r--r-- 1 ceph ceph   37 Mar 14 11:45 journal_uuid
-rw-r--r-- 1 ceph ceph   21 Mar 14 11:45 magic
-rw-r--r-- 1 ceph ceph    4 Mar 14 11:51 store_version
-rw-r--r-- 1 ceph ceph   53 Mar 14 11:51 superblock
-rw-r--r-- 1 ceph ceph    2 Mar 14 11:51 whoami
...
ceph_disk.main.Error: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', 
'--mkkey', '-i', u'0', '--monmap', 
'/var/lib/ceph/tmp/mnt.ECAifr/activate.monmap', '-
-osd-data', '/var/lib/ceph/tmp/mnt.ECAifr', '--osd-journal', 
'/var/lib/ceph/tmp/mnt.ECAifr/journal', '--osd-uuid', 
u'377c336b-278d-4caf-b2f5-592ac72cd9b6', '-
-keyring', '/var/lib/ceph/tmp/mnt.ECAifr/keyring', '--setuser', 'ceph', 
'--setgroup', 'ceph'] failed : 2017-03-16 11:30:05.238725 7f918fbc0a40 
-1 filestore(/v
ar/lib/ceph/tmp/mnt.ECAifr) mkjournal error creating journal on 
/var/lib/ceph/tmp/mnt.ECAifr/journal: (13) Permission denied
2017-03-16 11:30:05.238756 7f918fbc0a40 -1 OSD::mkfs: ObjectStore::mkfs 
failed with error -13
2017-03-16 11:30:05.238833 7f918fbc0a40 -1  ** ERROR: error creating 
empty object store in /var/lib/ceph/tmp/mnt.ECAifr: (13) Permission denied



~ # blkid /dev/mapper/vg--*lv-*p* | grep 
'120c536d-cb30-4cea-b607-dd347022a497'
/dev/mapper/vg--ssd1-lv--ssd1p1: PARTLABEL="ceph journal" 
PARTUUID="120c536d-cb30-4cea-b607-dd347022a497"

~ # ls -al /dev/disk/by-id | grep dm-22
lrwxrwxrwx 1 root root   11 Mar 16 11:37 dm-name-vg--ssd1-lv--ssd1p1 -> 
../../dm-22
lrwxrwxrwx 1 root root   11 Mar 16 11:37 
dm-uuid-part1-LVM-n1SH1FvtfjgxJOMWN9aHurFvn2BpIsLZi89GWxA68hLmUQV6l5oyiEOPsFciRbKg 
-> ../../dm-22

~ # ls -al /dev/disk/by-parttypeuuid | grep dm-22
lrwxrwxrwx 1 root root  11 Mar 16 11:37 
45b0969e-9b03-4f30-b4c6-b4b80ceff106.120c536d-cb30-4cea-b607-dd347022a497 -> 
../../dm-22

~ # ls -al /dev/disk/by-uuid | grep dm-22
~ # ls -al /dev/disk/by-partuuid/ | grep dm-22
~ # ls -al /dev/disk/by-path | grep dm-22


Best Regards,
Nicholas Gim.

On Wed, Mar 15, 2017 at 6:46 PM Peter Maloney 
> wrote:


On 03/15/17 08:43, Gunwoo Gim 

Re: [ceph-users] Migrating to new pools

[Jens-U. Mozdzen]

Hi *,

Zitat von ceph-users-requ...@lists.ceph.com:

Hi *,

facing the problem to reduce the number of PGs for a pool, I've found
various information and suggestions, but no "definite guide" to handle
pool migration with Ceph 12.2.x. This seems to be a fairly common
problem when having to deal with "teen-age clusters", so consolidated
information would be a real help. I'm willing to start writing things
up, but don't want to duplicate information. So:

Are there any documented "operational procedures" on how to migrate

- an RBD pool (with snapshots created by Openstack)

- a CephFS data pool

- a CephFS metadata pool

to a different volume, in order to be able to utilize pool settings
that cannot be changed on an existing pool?

---

RBD pools: From what I've read, RBD snapshots are "broken" after using
"rados cppool" to move the content of an "RBD pool" to a new pool.


after having read up on "rbd-mirror" and having had a glimpse at its  
code, it seems that preserving clones, snapshots and their  
relationships has been solved for cluster-to-cluster migration.


Is this really correct? If so, it might be possible to extend the code  
in a fashion that will allow a one-shot, intracluster pool-to-pool  
migration as a spin-off to rbd-mirror.


I was thinking along the following lines:

- run rbd-mirror in a stand-alone fashion, just stating source and  
destination pool


- leave it to the cluster admin to take RDB "offline", so that the  
pool content does not change during the copy (no RBD journaling  
involved)


- check that the destination pool is empty. In a first version,  
cumulative migrates (joining multiple source pools with distinct image  
names) would complicate things ;)


- sync all content from source to destination pool, in a one-shot fashion

- done

Anyone out there who can judge the chances of that approach, better  
than me? I'd be willing to spend development time on this, but  
starting from scratch will be rather hard, so pointers at where to  
look at within the rbd-mirror code would be more than welcome.


Regards,
Jens

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ubuntu 17.10 or Debian 9.3 + Luminous = random OS hang ?

[Youzhong Yang]

I don't think it's hardware issue. All the hosts are VMs. By the way, using
the same set of VMWare hypervisors, I switched back to Ubuntu 16.04 last
night, so far so good, no freeze.

On Fri, Jan 19, 2018 at 8:50 AM, Daniel Baumann 
wrote:

> Hi,
>
> On 01/19/18 14:46, Youzhong Yang wrote:
> > Just wondering if anyone has seen the same issue, or it's just me.
>
> we're using debian with our own backported kernels and ceph, works rock
> solid.
>
> what you're describing sounds more like hardware issues to me. if you
> don't fully "trust"/have confidence in your hardware (and your logs
> don't reveal anything), I'd recommend running some burn-in tests
> (memtest, cpuburn, etc.) on them for 24 hours/machine to rule out
> cpu/ram/etc. issues.
>
> Regards,
> Daniel
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ubuntu 17.10 or Debian 9.3 + Luminous = random OS hang ?

[Daniel Baumann]

Hi,

On 01/19/18 14:46, Youzhong Yang wrote:
> Just wondering if anyone has seen the same issue, or it's just me.

we're using debian with our own backported kernels and ceph, works rock
solid.

what you're describing sounds more like hardware issues to me. if you
don't fully "trust"/have confidence in your hardware (and your logs
don't reveal anything), I'd recommend running some burn-in tests
(memtest, cpuburn, etc.) on them for 24 hours/machine to rule out
cpu/ram/etc. issues.

Regards,
Daniel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ubuntu 17.10 or Debian 9.3 + Luminous = random OS hang ?

[David Turner]

The freeze is likely a kernel panic. Try testing different versions of the
kernel.

On Fri, Jan 19, 2018, 8:46 AM Youzhong Yang  wrote:

> One month ago when I first started evaluating ceph, I chose Debian 9.3 as
> the operating system. I saw random OS hang so I gave up and switched to
> Ubuntu 16.04. Every thing works well using Ubuntu 16.04.
>
> Yesterday I tried Ubuntu 17.10, again I saw random OS hang, no matter it's
> mon, mgr, osd, or rgw. When it hangs, the console won't respond to keyboard
> input, the host is unreachable from the network.
>
> This is the OS vs kernel version list:
> Ubuntu 16.04 -> kernel 4.4
> Debian 9.3  -> kernel 4.9
> Ubuntu 17.10 -> kernel 4.13
>
> Just wondering if anyone has seen the same issue, or it's just me.
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ubuntu 17.10 or Debian 9.3 + Luminous = random OS hang ?

[Youzhong Yang]

One month ago when I first started evaluating ceph, I chose Debian 9.3 as
the operating system. I saw random OS hang so I gave up and switched to
Ubuntu 16.04. Every thing works well using Ubuntu 16.04.

Yesterday I tried Ubuntu 17.10, again I saw random OS hang, no matter it's
mon, mgr, osd, or rgw. When it hangs, the console won't respond to keyboard
input, the host is unreachable from the network.

This is the OS vs kernel version list:
Ubuntu 16.04 -> kernel 4.4
Debian 9.3  -> kernel 4.9
Ubuntu 17.10 -> kernel 4.13

Just wondering if anyone has seen the same issue, or it's just me.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ghost degraded objects

[Sage Weil]

On Fri, 19 Jan 2018, Ugis wrote:
> Running Luminous 12.2.2, noticed strange behavior lately.
> When for example setting "ceph osd out X" closer to the reballancing
> end "degraded" objects still show up, but in "pgs:" section of ceph -s
> no degraded pgs are still recovering, just ramapped and no degraded
> pgs can be found in "ceph pg dump"
> 
>   health: HEALTH_WARN
> 355767/30286841 objects misplaced (1.175%)
> Degraded data redundancy: 28/30286841 objects degraded
> (0.000%), 96 pgs unclean
> 
>   services:
> ...
> osd: 38 osds: 38 up, 37 in; 96 remapped pgs
> 
>   data:
> pools:   19 pools, 4176 pgs
> objects: 9859k objects, 39358 GB
> usage:   114 TB used, 120 TB / 234 TB avail
> pgs: 28/30286841 objects degraded (0.000%)
>  355767/30286841 objects misplaced (1.175%)
>  4080 active+clean
>  81   active+remapped+backfilling
>  15   active+remapped+backfill_wait
> 
> 
> Where those 28 degraded objects come from?

There aren't actually degraded objects.. in this case it's just 
misreporting that there are.

This is a known issue in luminous.  Shortly after release we noticed the 
problem and David has been working on several changes to the stats 
calculation to improve the reporting, but those changes have not been 
backported (and aren't quite complete, either--getting a truly accurate 
number there is nontrivial in some cases it turns out).

> In such cases usually when backfilling is done degraded objects also
> disappear, but normally degraded objects should fix before remapped
> ones by priority.

Yes.

It's unfortunately a scary warning (there shouldn't be degraded 
objects... and generally speaking aren't) that understandably alarms 
users.  We hope to have this sorted out soon!

sage
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] QUEMU - rbd cache - inconsistent documentation?

[Jason Dillaman]

That big fat warning was related to the note under your second quote:

"Note: Prior to QEMU v2.4.0, if you explicitly set RBD Cache settings
in the Ceph configuration file, your Ceph settings override the QEMU
cache settings."

Short story long, starting with QEMU v2.4.0, your QEMU cache settings
100% control/override your librbd cache settings to prevent such
situations.


On Fri, Jan 19, 2018 at 7:48 AM, Wolfgang Lendl
 wrote:
> hi,
>
> I'm a bit confused after reading the official ceph docu regarding QEMU and
> rbd caching.
>
> http://docs.ceph.com/docs/master/rbd/qemu-rbd/?highlight=qemu
>
> there's a big fat warning:
>
> "Important: If you set rbd_cache=true, you must set cache=writeback or risk
> data loss. Without cache=writeback, QEMU will not send flush requests to
> librbd. If QEMU exits uncleanly in this configuration, filesystems on top of
> rbd can be corrupted."
>
> two sections below you can find the following:
>
> "QEMU’s cache settings override Ceph’s cache settings (including settings
> that are explicitly set in the Ceph configuration file)."
>
>
> I find these two statements contradicting and looking for the truth - or did
> i miss something?
>
> ceph/librbd: 12.2
> qemu/kvm: 2.9.0
>
>
> br wolfgang
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] QUEMU - rbd cache - inconsistent documentation?

[Wolfgang Lendl]

hi,

I'm a bit confused after reading the official ceph docu regarding QEMU
and rbd caching.

http://docs.ceph.com/docs/master/rbd/qemu-rbd/?highlight=qemu

there's a big fat warning: 

"Important: If you set rbd_cache=true, you must set cache=writeback or
risk data loss. Without cache=writeback, QEMU will not send flush
requests to librbd. If QEMU exits uncleanly in this configuration,
filesystems on top of rbd can be corrupted."

two sections below you can find the following:

"QEMU’s cache settings override Ceph’s cache settings (including
settings that are explicitly set in the Ceph configuration file)."


I find these two statements contradicting and looking for the truth - or
did i miss something?

ceph/librbd: 12.2
qemu/kvm: 2.9.0


br wolfgang

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph df shows 100% used

[Webert de Souza Lima]

While it seemed to be solved yesterday, today the %USED has grown a lot
again. See:

~# ceph osd df tree
http://termbin.com/0zhk

~# ceph df detail
http://termbin.com/thox

94% USED while there is about 21TB worth of data, size = 2 menas ~42TB RAW
Usage, but the OSDs in that root sum ~70TB available space.



Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
*Belo Horizonte - Brasil*
*IRC NICK - WebertRLZ*

On Thu, Jan 18, 2018 at 8:21 PM, Webert de Souza Lima  wrote:

> With the help of robbat2 and llua on IRC channel I was able to solve this
> situation by taking down the 2-OSD only hosts.
> After crush reweighting OSDs 8 and 23 from host mia1-master-fe02 to 0,
> ceph df showed the expected storage capacity usage (about 70%)
>
>
> With this in mind, those guys have told me that it is due the cluster
> beeing uneven and unable to balance properly. It makes sense and it worked.
> But for me it is still a very unexpected bahaviour for ceph to say that
> the pools are 100% full and Available Space is 0.
>
> There were 3 hosts and repl. size = 2, if the host with only 2 OSDs were
> full (it wasn't), ceph could still use space from OSDs from the other hosts.
>
> Regards,
>
> Webert Lima
> DevOps Engineer at MAV Tecnologia
> *Belo Horizonte - Brasil*
> *IRC NICK - WebertRLZ*
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Fwd: Ceph team involvement in Rook (Deploying Ceph in Kubernetes)

[Marc Roos]

 
Are the guys of apache mesos agreeing to this? I have been looking at 
mesos, dcos and still have to make up my mind which way to go. I like 
that mesos has the unified containerizer that runs the docker images and 
I don’t need to run the dockerd, how the adapt the to the cni standard. 
How is this going with kubernetes? Link maybe to why I should use 
kubernetes?



-Original Message-
From: Kai Wagner [mailto:kwag...@suse.com] 
Sent: vrijdag 19 januari 2018 11:55
To: Ceph Users
Subject: [ceph-users] Fwd: Ceph team involvement in Rook (Deploying Ceph 
in Kubernetes)

Just for those of you who are not subscribed to ceph-users.



 Forwarded Message  
Subject:Ceph team involvement in Rook (Deploying Ceph in 
Kubernetes)  
Date:   Fri, 19 Jan 2018 11:49:05 +0100  
From:   Sebastien Han  
 
To: ceph-users  
 , Squid Cybernetic 
 
 , Dan Mick  
 , Chen, Huamin  
 , John Spray  
 , Sage Weil  
 , bas...@tabbara.com   


Everyone,

Kubernetes is getting bigger and bigger. It has become the platform of 
choice to run microservices applications in containers, just like 
OpenStack did for and Cloud applications in virtual machines.

When it comes to container storage there are three key aspects:

* Providing persistent storage to containers, Ceph has drivers in 
Kuberntes already with kRBD and CephFS
* Containerizing the storage itself, so efficiently running Ceph 
services in Containers. Currently, we have ceph-container
(https://github.com/ceph/ceph-container)
* Deploying the containerized storage in Kubernetes, we wrote ceph-helm 
charts (https://github.com/ceph/ceph-helm)

The third piece although it's working great has a particular goal and 
doesn't aim to run Ceph just like any other applications in Kuberntes.
We were also looking for a better abstraction/ease of use for end-users, 
multi-cluster support, operability, life-cycle management, centralized 
operations, to learn more you can read 
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-October/021918.html.
As a consequence, we decided to look at what the ecosystem had to offer. 
As a result, Rook came out, as a pleasant surprise. For those who are 
not familiar with Rook, please visit https://rook.io but in a nutshell, 
Rook is an open source orchestrator for distributed storage systems 
running in cloud-native environments. Under the hood, Rook is deploying, 
operating and managing Ceph life cycle in Kubernetes. Rook has a vibrant 
community and committed developers.

Even if Rook is not perfect (yet), it has firm foundations, and we are 
planning on helping to make it better. We already opened issues for that 
and started doing work with Rook's core developers. We are looking at 
reconciling what is available today (rook/ceph-container/helm), reduce 
the overlap/duplication and all work together toward a single and common 
goal. With this collaboration, through Rook, we hope to make Ceph the de 
facto Open Source storage solution for Kubernetes.

These are exciting times, so if you're a user, a developer, or merely 
curious, have a look at Rook and send us feedback!

Thanks!
--
Cheers

––
Sébastien Han
Principal Software Engineer, Storage Architect

"Always give 100%. Unless you're giving blood."

Mail: s...@redhat.com
Address: 11 bis, rue Roquépine - 75008 Paris
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
the body of a message to majord...@vger.kernel.org More majordomo info 
at  http://vger.kernel.org/majordomo-info.html


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Fwd: Ceph team involvement in Rook (Deploying Ceph in Kubernetes)

[Kai Wagner]

Just for those of you who are not subscribed to ceph-users.



 Forwarded Message 
Subject:Ceph team involvement in Rook (Deploying Ceph in Kubernetes)
Date:   Fri, 19 Jan 2018 11:49:05 +0100
From:   Sebastien Han 
To: ceph-users , Squid Cybernetic
, Dan Mick , Chen, Huamin
, John Spray , Sage Weil
, bas...@tabbara.com



Everyone,

Kubernetes is getting bigger and bigger. It has become the platform of
choice to run microservices applications in containers, just like
OpenStack did for and Cloud applications in virtual machines.

When it comes to container storage there are three key aspects:

* Providing persistent storage to containers, Ceph has drivers in
Kuberntes already with kRBD and CephFS
* Containerizing the storage itself, so efficiently running Ceph
services in Containers. Currently, we have ceph-container
(https://github.com/ceph/ceph-container)
* Deploying the containerized storage in Kubernetes, we wrote
ceph-helm charts (https://github.com/ceph/ceph-helm)

The third piece although it's working great has a particular goal and
doesn't aim to run Ceph just like any other applications in Kuberntes.
We were also looking for a better abstraction/ease of use for
end-users, multi-cluster support, operability, life-cycle management,
centralized operations, to learn more you can read
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-October/021918.html.
As a consequence, we decided to look at what the ecosystem had to
offer. As a result, Rook came out, as a pleasant surprise. For those
who are not familiar with Rook, please visit https://rook.io but in a
nutshell, Rook is an open source orchestrator for distributed storage
systems running in cloud-native environments. Under the hood, Rook is
deploying, operating and managing Ceph life cycle in Kubernetes. Rook
has a vibrant community and committed developers.

Even if Rook is not perfect (yet), it has firm foundations, and we are
planning on helping to make it better. We already opened issues for
that and started doing work with Rook's core developers. We are
looking at reconciling what is available today
(rook/ceph-container/helm), reduce the overlap/duplication and all
work together toward a single and common goal. With this
collaboration, through Rook, we hope to make Ceph the de facto Open
Source storage solution for Kubernetes.

These are exciting times, so if you're a user, a developer, or merely
curious, have a look at Rook and send us feedback!

Thanks!
-- 
Cheers

––
Sébastien Han
Principal Software Engineer, Storage Architect

"Always give 100%. Unless you're giving blood."

Mail: s...@redhat.com
Address: 11 bis, rue Roquépine - 75008 Paris
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com