Re: [ceph-users] cephfs kernel client blocks when removing large files

Hi Greg, Nowhere in your test procedure do you mention syncing or flushing the files to disk. That is almost certainly the cause of the slowness We have tested performing sync after file creation and the delay still occurs. (See Test3 results below) To clarify, it appears the delay is

Re: [ceph-users] fixing another remapped+incomplete EC 4+2 pg

On 10/09/2018 01:14 PM, Graham Allan wrote: On 10/9/2018 12:19 PM, Gregory Farnum wrote: I think unfortunately the easiest thing for you to fix this will be to set the min_size back to 4 until the PG is recovered (or at least has 5 shards done). This will be fixed in a later version of

[ceph-users] tcmu iscsi (failover not supported)

I'm trying to replace my old single point of failure iscsi gateway with the shiny new tcmu-runner implementation. I've been fighting a Windows initiator all day. I haven't tested any other initiators, as Windows is currently all we use iscsi for. One issue I've considered is our Ceph cluster is

Re: [ceph-users] nfs-ganesha version in Ceph repos

Luminous is also not having an updated librgw that prevents ganesha from using the multi tenancy mounts. Especially with the current issues of mimic, would it be nice if this could be made available in luminous. https://www.mail-archive.com/ceph-users@lists.ceph.com/msg48659.html

Re: [ceph-users] add existing rbd to new tcmu iscsi gateways

On Tue, Oct 9, 2018 at 3:14 PM Brady Deetz wrote: > > I am attempting to migrate to the new tcmu iscsi gateway. Is there a way to > configure gwcli to export an rbd that was created outside gwcli? You should be able to just run "/disks create . " from within "gwcli" to have it add an existing

Re: [ceph-users] ceph-iscsi upgrade issue

Anything in the rbd-target-api.log on osd03 to indicate why it failed? Since you replaced your existing "iscsi-gateway.conf", do your security settings still match between the two hosts (i.e. on the trusted_ip_list, same api_XYZ options)? On Tue, Oct 9, 2018 at 4:25 PM Steven Vacaroaia wrote: >

Re: [ceph-users] ceph-iscsi upgrade issue

so the gateways are up but I have issues adding disks ( i.e. if I do it on one gatway it does not show on the other - however, after I restart the rbd-target services I am seeing the disks ) Thanks in advance for taking the trouble to provide advice / guidance 2018-10-09 16:16:08,968 INFO

Re: [ceph-users] nfs-ganesha version in Ceph repos

On Tue, Oct 9, 2018 at 2:55 PM Erik McCormick wrote: > > > > On Tue, Oct 9, 2018, 2:17 PM Kevin Olbrich wrote: >> >> I had a similar problem: >> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-September/029698.html >> >> But even the recent 2.6.x releases were not working well for me

Re: [ceph-users] ceph-iscsi upgrade issue

It worked. many thanks Steven On Tue, 9 Oct 2018 at 15:36, Jason Dillaman wrote: > Can you try applying [1] and see if that resolves your issue? > > [1] https://github.com/ceph/ceph-iscsi-config/pull/78 > On Tue, Oct 9, 2018 at 3:06 PM Steven Vacaroaia wrote: > > > > Thanks Jason > > > >

Re: [ceph-users] ceph-iscsi upgrade issue

Can you try applying [1] and see if that resolves your issue? [1] https://github.com/ceph/ceph-iscsi-config/pull/78 On Tue, Oct 9, 2018 at 3:06 PM Steven Vacaroaia wrote: > > Thanks Jason > > adding prometheus_host = 0.0.0.0 to iscsi-gateway.cfg does not work - the > error message is > >

[ceph-users] add existing rbd to new tcmu iscsi gateways

I am attempting to migrate to the new tcmu iscsi gateway. Is there a way to configure gwcli to export an rbd that was created outside gwcli? This is necessary for me because I have a lun exported from an old LIO gateway to a Windows host that I need to transition to the new tcmu based cluster.

Re: [ceph-users] ceph-iscsi upgrade issue

Thanks Jason adding prometheus_host = 0.0.0.0 to iscsi-gateway.cfg does not work - the error message is "..rbd-target-gw: ValueError: invalid literal for int() with base 10: '0.0.0.0' " adding prometheus_exporter = false works However I'd like to use prometheus_exporter if possible Any

Re: [ceph-users] nfs-ganesha version in Ceph repos

On Tue, Oct 9, 2018, 2:17 PM Kevin Olbrich wrote: > I had a similar problem: > > http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-September/029698.html > > But even the recent 2.6.x releases were not working well for me (many many > segfaults). I am on the master-branch (2.7.x) and that

Re: [ceph-users] nfs-ganesha version in Ceph repos

On Tue, Oct 9, 2018, 1:48 PM Alfredo Deza wrote: > On Tue, Oct 9, 2018 at 1:39 PM Erik McCormick > wrote: > > > > On Tue, Oct 9, 2018 at 1:27 PM Erik McCormick > > wrote: > > > > > > Hello, > > > > > > I'm trying to set up an nfs-ganesha server with the Ceph FSAL, and > > > running into

Re: [ceph-users] ceph-iscsi upgrade issue

You can try adding "prometheus_exporter = false" in your "/etc/ceph/iscsi-gateway.cfg"'s "config" section if you aren't using "cephmetrics", or try setting "prometheus_host = 0.0.0.0" since it sounds like you have the IPv6 stack disabled. [1]

Re: [ceph-users] nfs-ganesha version in Ceph repos

I had a similar problem: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-September/029698.html But even the recent 2.6.x releases were not working well for me (many many segfaults). I am on the master-branch (2.7.x) and that works well with less crashs. Cluster is 13.2.1/.2 with

Re: [ceph-users] fixing another remapped+incomplete EC 4+2 pg

On 10/9/2018 12:19 PM, Gregory Farnum wrote: On Wed, Oct 3, 2018 at 10:18 AM Graham Allan > wrote: However I have one pg which is stuck in state remapped+incomplete because it has only 4 out of 6 osds running, and I have been unable to bring the missing two

Re: [ceph-users] ceph-iscsi upgrade issue

here is some info from /var/log/messages ..in case someone has the time to take a look Oct 9 13:58:35 osd03 systemd: Started Setup system to export rbd images through LIO. Oct 9 13:58:35 osd03 systemd: Starting Setup system to export rbd images through LIO... Oct 9 13:58:35 osd03 journal:

[ceph-users] HEALTH_WARN 2 osd(s) have {NOUP, NODOWN, NOIN, NOOUT} flags set

Hello everyone, I am getting warning messages regarding 3osd's with noin and noout flags set. The osd are in up state.I have run the ceph osd unset noin on the cluster and it does not seem to clear the flags. I have attached status files for the cluster. The cluster is running

Re: [ceph-users] nfs-ganesha version in Ceph repos

On Tue, Oct 9, 2018 at 1:39 PM Erik McCormick wrote: > > On Tue, Oct 9, 2018 at 1:27 PM Erik McCormick > wrote: > > > > Hello, > > > > I'm trying to set up an nfs-ganesha server with the Ceph FSAL, and > > running into difficulties getting the current stable release running. > > The versions in

Re: [ceph-users] nfs-ganesha version in Ceph repos

On Tue, Oct 9, 2018 at 1:27 PM Erik McCormick wrote: > > Hello, > > I'm trying to set up an nfs-ganesha server with the Ceph FSAL, and > running into difficulties getting the current stable release running. > The versions in the Luminous repo is stuck at 2.6.1, whereas the > current stable

Re: [ceph-users] cephfs kernel client blocks when removing large files

Nowhere in your test procedure do you mention syncing or flushing the files to disk. That is almost certainly the cause of the slowness — the client which wrote the data is required to flush it out before dropping enough file "capabilities" for the other client to do the rm. -Greg On Sun, Oct 7,

[ceph-users] nfs-ganesha version in Ceph repos

Hello, I'm trying to set up an nfs-ganesha server with the Ceph FSAL, and running into difficulties getting the current stable release running. The versions in the Luminous repo is stuck at 2.6.1, whereas the current stable version is 2.6.3. I've seen a couple of HA issues in pre 2.6.3 versions

Re: [ceph-users] fixing another remapped+incomplete EC 4+2 pg

On Wed, Oct 3, 2018 at 10:18 AM Graham Allan wrote: > Following on from my previous adventure with recovering pgs in the face > of failed OSDs, I now have my EC 4+2 pool oeprating with min_size=5 > which is as things should be. > > However I have one pg which is stuck in state

[ceph-users] ceph-iscsi upgrade issue

Hi , I am using Mimic 13.2 and kernel 4.18 Was using gwcli 2.5 and decided to upgrade to latest (2.7) as people reported improved performance What is the proper methodology ? How should I troubleshoot this? What I did ( and it broke it) was cd tcmu-runner; git pull ; make && make install cd

Re: [ceph-users] list admin issues

Without an example of the bounce response itself it's virtually impossible to troubleshoot. Can someone with mailman access please provide an example of a bounce response? All the attachments on those rejected messages are just HTML copies of the message (which are not on the list of filtered

Re: [ceph-users] Can't remove DeleteMarkers in rgw bucket

I would suggest trying to delete the bucket using radosgw-admin. If you can't get that to work, then I would go towards deleting the actual RADOS objects. There are a few threads on the ML that talk about manually deleting a bucket. On Thu, Sep 20, 2018 at 2:04 PM Sean Purdy wrote: > Hi, > >

Re: [ceph-users] bluestore compression enabled but no data compressed

When I've tested compression before there are 2 places you need to configure compression. On the OSDs in the configuration settings that you mentioned, but also on the [1] pools themselves. If you have the compression mode on the pools set to none, then it doesn't matter what the OSDs

Re: [ceph-users] fixing another remapped+incomplete EC 4+2 pg

Oops, by "periods" I do of course mean "intervals"...! On 10/8/2018 4:57 PM, Graham Allan wrote: I'm still trying to find a way to reactivate this one pg which is incomplete. There are a lot of periods in its history based on a combination of a peering storm a couple of weeks ago, with

Re: [ceph-users] Error-code 2002/API 405 S3 REST API. Creating a new bucket

Can you outline the process you're using to access the REST API? It's hard to troubleshoot this without knowing how you were trying to do this. On Mon, Sep 17, 2018 at 7:09 PM Michael Schäfer wrote: > Hi, > > We have a problem with the radosgw using the S3 REST API. > Trying to create a new

Re: [ceph-users] radosgw bucket stats vs s3cmd du

Have you looked at your Garbage Collection. I would guess that your GC is behind and that radosgw-admin is accounting for that space knowing that it hasn't been freed up yet, whiles 3cmd doesn't see it since it no longer shows in the listing. On Tue, Sep 18, 2018 at 4:45 AM Luis Periquito

Re: [ceph-users] MDSs still core dumping

scan_links has finished and now I'm able to start MDS with a bunch of 'failed to open ino' and 'bad backtrace' log entries, but at least MDS no longer segfaults and I can mount the fs. > On 9.10.2018, at 02:22, Sergey Malinin wrote: > > I was able to start MDS 13.2.1 when I had imported

Re: [ceph-users] vfs_ceph ignoring quotas

On Tue, Oct 9, 2018 at 11:51 AM Felix Stolte wrote: > > That's bad news, but maybe there is a workarround. Samba offers the > opportunity to define a custom df command. If I could extract the > current utilization or size of a directory with a quota, i think i > should be able to write a little

Re: [ceph-users] vfs_ceph ignoring quotas

cat /path/to/dir | grep rbytes | awk {'print $2'} assuming cephfs mount has dirstat option. Here is the script I used in my environment: #!/bin/sh if [ -f "$1/.quota" ]; then TOTAL=`cat $1/.quota` USED=`cat $1 | grep rbytes | awk {'print $2'}` USED=$((USED/1000))

Re: [ceph-users] OMAP size on disk

Hi Luis, There are currently open issues with space reclamation after dynamic bucket index resharding, esp. http://tracker.ceph.com/issues/34307 Changes are being worked on to address this, and to permit administratively reclaiming space. Matt On Tue, Oct 9, 2018 at 5:50 AM, Luis Periquito

Re: [ceph-users] list admin issues

Maybe there are some advice here that can help remedy the situation a bit? https://support.google.com/mail/answer/81126?hl=en https://support.google.com/mail/answer/6227174?hl=en /Elias On Tue, Oct 9, 2018 at 2:24 AM Alex Gorbachev wrote: > On Mon, Oct 8, 2018 at 7:48 AM Elias Abacioglu >

Re: [ceph-users] vfs_ceph ignoring quotas

That's bad news, but maybe there is a workarround. Samba offers the opportunity to define a custom df command. If I could extract the current utilization or size of a directory with a quota, i think i should be able to write a little df command. The quota is stored as an extended attribute,

Re: [ceph-users] Cluster broken and ODSs crash with failed assertion in PGLog::merge_log

Yay! I've managed to get the cluster up and running again. Deep scrub is still fixing inconsistencies. I had to do a depth-first-search in the tree of startup errors. My procedure was the already described one: Find and delete PGs from OSDs which trigger the assertion. I've created a script to

[ceph-users] OMAP size on disk

Hi all, I have several clusters, all running Luminous (12.2.7) proving S3 interface. All of them have enabled dynamic resharding and is working. One of the newer clusters is starting to give warnings on the used space for the OMAP directory. The default.rgw.buckets.index pool is replicated with

Re: [ceph-users] vfs_ceph ignoring quotas

On Tue, Oct 9, 2018 at 9:14 AM Felix Stolte wrote: > > Hi folks, > > i'm running a luminous cluster on Ubuntu 18.04 an want to share folders > on cephfs with samba using the vfs_ceph. Sharing works fine, but the > quotas I set on the directories is ignored and every share reports its > size as

[ceph-users] vfs_ceph ignoring quotas

Hi folks, i'm running a luminous cluster on Ubuntu 18.04 an want to share folders on cephfs with samba using the vfs_ceph. Sharing works fine, but the quotas I set on the directories is ignored and every share reports its size as the total size of the cephfs. Anyone got this working? Or is

Re: [ceph-users] Mons are using a lot of disk space and has a lot of old osd maps

On 10/09/2018 09:35 AM, Aleksei Zakharov wrote: > If someone is interested: we've found a workaround in this mailing list: > https://www.spinics.net/lists/ceph-users/msg47963.html > It looks like an old bug. > We fixed the issue by restarting all ceph-mon services one by one. Mon's > store

Re: [ceph-users] Mons are using a lot of disk space and has a lot of old osd maps

If someone is interested: we've found a workaround in this mailing list: https://www.spinics.net/lists/ceph-users/msg47963.html It looks like an old bug. We fixed the issue by restarting all ceph-mon services one by one. Mon's store uses ~500Mb now and osd's removed old osd maps: ~# find