Re: [ceph-users] Pool shard/stripe settings for file too large files?

Yes, I actually changed it back to the default after reading somewhat about it (https://github.com/ceph/ceph/pull/15520). I wanted to store 5GB and 12GB files, that makes recovery not to nice. I thought there was a setting to split them up automatically like with rbd pools. -Original

Re: [ceph-users] Pool shard/stripe settings for file too large files?

Do you know of a rados client that uses this? Maybe a simple 'mount' so I can cp the files on it? -Original Message- From: Christian Wuerdig [mailto:christian.wuer...@gmail.com] Sent: donderdag 9 november 2017 22:01 To: Kevin Hrpcek Cc: Marc Roos; ceph-users Subject: Re:

[ceph-users] Undersized fix for small cluster, other than adding a 4th node?

I added an erasure k=3,m=2 coded pool on a 3 node test cluster and am getting these errors. pg 48.0 is stuck undersized for 23867.00, current state active+undersized+degraded, last acting [9,13,2147483647,7,2147483647] pg 48.1 is stuck undersized for 27479.944212, current state

[ceph-users] Needed help to setup a 3-way replication between 2 datacenters

Hi everyone, Beginner with Ceph, i’m looking for a way to do a 3-way replication between 2 datacenters as mention in ceph docs (but not describe). My goal is to keep access to the data (at least read-only access) even when the link between the 2 datacenters is cut and make sure at least one

[ceph-users] Performance, and how much wiggle room there is with tunables

In my cluster, rados bench shows about 1GB/s bandwidth. I've done some tuning: [osd] osd op threads = 8 osd disk threads = 4 osd recovery max active = 7 I was hoping to get much better bandwidth. My network can handle it, and my disks are pretty fast as well. Are there any major tunables I

Re: [ceph-users] Pool shard/stripe settings for file too large files?

It should be noted that the general advise is to not use such large objects since cluster performance will suffer, see also this thread: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-September/021051.html libradosstriper might be an option which will automatically break the object into

Re: [ceph-users] Pool shard/stripe settings for file too large files?

Marc, If you're running luminous you may need to increase osd_max_object_size. This snippet is from the Luminous change log. "The default maximum size for a single RADOS object has been reduced from 100GB to 128MB. The 100GB limit was completely impractical in practice while the 128MB limit

[ceph-users] Pool shard/stripe settings for file too large files?

I would like store objects with rados -p ec32 put test2G.img test2G.img error putting ec32/test2G.img: (27) File too large Changing the pool application from custom to rgw did not help ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] Fwd: What's the fastest way to try out object classes?

I changed this line into CLS_LOG(0, "loading cls_test"); https://github.com/ceph/ceph/blob/master/src/cls/sdk/cls_sdk.cc#L120 I don't think the test object class is loaded correctly since I don't have the loading information in the log. However I can see "loading cls_sdk" in the osd log. On

Re: [ceph-users] Fwd: What's the fastest way to try out object classes?

On Thu, Nov 9, 2017 at 10:05 AM, Zheyuan Chen wrote: > I installed rados-objclass-dev and objclass.h was installed successfully. > However, I failed to run the objclass following the steps as below: > > 1. copy https://github.com/ceph/ceph/blob/master/src/cls/sdk/cls_sdk.cc

Re: [ceph-users] Fwd: What's the fastest way to try out object classes?

I installed rados-objclass-dev and objclass.h was installed successfully. However, I failed to run the objclass following the steps as below: 1. copy https://github.com/ceph/ceph/blob/master/src/cls/sdk/cls_sdk.cc into my machine. (cls_test.cpp) 2. make some changes on cls_test.cpp: 1) rename all

Re: [ceph-users] how to upgrade CEPH journal?

Hi Caspar, Is this in the [global] or [osd] section of ceph.conf? I am new to ceph so this is all still very vague to me. What is the difference betwen the WAL and the DB? And, lastly, if I want to setup the OSD in Proxmox beforehand and add the journal to it, can I make these changes

Re: [ceph-users] how to upgrade CEPH journal?

Please bear in mind that unless you've got a very good reason for separating the WAL/DB into two partitions (i.e. you are testing/debugging and want to observe their behaviour separately or they're actually going to go on different devices which have different speeds) you should probably stick

Re: [ceph-users] how to upgrade CEPH journal?

2017-11-09 17:02 GMT+01:00 Alwin Antreich : > Hi Rudi, > On Thu, Nov 09, 2017 at 04:09:04PM +0200, Rudi Ahlers wrote: > > Hi, > > > > Can someone please tell me what the correct procedure is to upgrade a > CEPH > > journal? > > > > I'm running ceph: 12.2.1 on Proxmox 5.1,

Re: [ceph-users] how to upgrade CEPH journal?

Rudi, You can set the size of block.db and block.wal partitions in the ceph.conf configuration file using: bluestore_block_db_size = 16106127360 (which is 15GB, just calculate the correct number for your needs) bluestore_block_wal_size = 16106127360 Kind regards, Caspar 2017-11-09 17:19

Re: [ceph-users] Cluster hang

Update: I noticed that there was a pg that remained scrubbing from the first day I found the issue to when I reboot the node and problem disappeared. Can this cause the behaviour I described before? > Il giorno 09 nov 2017, alle ore 15:55, Matteo Dacrema ha > scritto: > >

Re: [ceph-users] how to upgrade CEPH journal?

Hi Alwin, Thanx for the help. I see now that I used the wrong wording in my email. I want to resize the journal, not upgrade. So, following your commands, I still sit with a 1GB journal: oot@virt1:~# ceph-disk prepare --bluestore \ > --block.db /dev/sde --block.wal /dev/sde1 /dev/sda Setting

Re: [ceph-users] how to upgrade CEPH journal?

Hi Rudi, On Thu, Nov 09, 2017 at 04:09:04PM +0200, Rudi Ahlers wrote: > Hi, > > Can someone please tell me what the correct procedure is to upgrade a CEPH > journal? > > I'm running ceph: 12.2.1 on Proxmox 5.1, which runs on Debian 9.1 > > For a journal I have a 400GB Intel SSD drive and it seems

[ceph-users] Cluster hang

Hi all, I’ve experienced a strange issue with my cluster. The cluster is composed by 10 HDDs nodes with 20 nodes + 4 journal each plus 4 SSDs nodes with 5 SSDs each. All the nodes are behind 3 monitors and 2 different crush maps. All the cluster is on 10.2.7 About 20 days ago I started to

[ceph-users] how to upgrade CEPH journal?

Hi, Can someone please tell me what the correct procedure is to upgrade a CEPH journal? I'm running ceph: 12.2.1 on Proxmox 5.1, which runs on Debian 9.1 For a journal I have a 400GB Intel SSD drive and it seems CEPH created a 1GB journal: Disk /dev/sdf: 372.6 GiB, 400088457216 bytes,

Re: [ceph-users] Ceph auth profile definitions

They are currently defined to the following (translated to cap syntax): mon: 'allow service mon r, allow service osd r, allow service pg r, allow command "osd blacklist" with blacklistop=add addr regex "^[^/]+/[0-9]+$"' osd: 'allow class-read object_prefix rbd_children, allow class-read

Re: [ceph-users] who is using nfs-ganesha and cephfs?

The email was not delivered to ceph-de...@vger.kernel.org. So, re-sending it. Few more things regarding the hardware and clients used in our benchmarking setup: - The cephfs benchmark were done using kernel cephfs client. - NFS-Ganesha was mounted using nfs version 4. - Single nfs-ganesha

Re: [ceph-users] bluestore - wal,db on faster devices?

One small point: It's a bit easier to observe distinct WAL and DB behavior when they are on separate partitions. I often do this for benchmarking and testing though I don't know that it would be enough of a benefit to do it in production. Mark On 11/09/2017 04:16 AM, Richard Hesketh wrote:

Re: [ceph-users] bluestore - wal,db on faster devices?

-sorry, wrong address Hi Richard, I have seen a few lectures about bluestore, and they made it abundantly clear, that bluestore is superior to filestore in that manner, that it writes data to the disc only once (this way they could achieve a 2x-3x speed increase). So this is true if there

[ceph-users] Librbd, qemu, libvirt xml

What would be the correct way to convert the xml file rbdmapped images to librbd? I had this: And for librbd this: But this will give me a

Re: [ceph-users] Recovery operations and ioprio options

Hi, Nick Thank you for the answer! It's still unclear for me, do those options have no effect at all? Or disk thread is used for some other operations? 09.11.2017, 04:18, "Nick Fisk" : >>  -Original Message- >>  From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com]

Re: [ceph-users] who is using nfs-ganesha and cephfs?

Hi Sage, As Lars mentioned, at SUSE, we use ganesha 2.5.2/luminous. We did a preliminary performance comparison of cephfs client and nfs-ganesha client. I have attached the results. The results are aggregate bandwidth over 10 clients. 1. Test Setup: We use fio to read/write to a single 5GB

[ceph-users] Ganesha NFS

Hi!I would like to export my cephfs using Ganesha NFS to export a NFSv3 o NFSv4 I am a little lost while doing it. I managed to make it working with NFSv4 but I can't make it work with NFSv3 as the server refuses the connection. Has anyone managed to do it? 

Re: [ceph-users] Ceph auth profile definitions

On Thu, Nov 9, 2017 at 10:12 AM, Marc Roos wrote: > > How/where can I see how eg. 'profile rbd' is defined? > > As in > [client.rbd.client1] > key = xxx== > caps mon = "profile rbd" > caps osd = "profile rbd pool=rbd" The profiles are defined

Re: [ceph-users] bluestore - wal,db on faster devices?

You're correct, if you were going to put the WAL and DB on the same device you should just make one partition and allocate the DB to it, the WAL will automatically be stored with the DB. It only makes sense to specify them separately if they are going to go on different devices, and that itself

[ceph-users] Ceph auth profile definitions

How/where can I see how eg. 'profile rbd' is defined? As in [client.rbd.client1] key = xxx== caps mon = "profile rbd" caps osd = "profile rbd pool=rbd" ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] who is using nfs-ganesha and cephfs?

On 2017-11-08T21:41:41, Sage Weil wrote: > Who is running nfs-ganesha's FSAL to export CephFS? What has your > experience been? > > (We are working on building proper testing and support for this into > Mimic, but the ganesha FSAL has been around for years.) We use it

Re: [ceph-users] Fwd: Luminous RadosGW issue

> On Nov 9, 2017, at 5:25 AM, Sam Huracan wrote: > > root@radosgw system]# ceph --admin-daemon > /var/run/ceph/ceph-client.rgw.radosgw.asok config show | grep log_file > "log_file": "/var/log/ceph/ceph-client.rgw.radosgw.log”, The .asok filename resembles what

Re: [ceph-users] Luminous ceph pool %USED calculation

On Fri, Nov 03, 2017 at 12:09:03PM +0100, Alwin Antreich wrote: > Hi, > > I am confused by the %USED calculation in the output 'ceph df' in luminous. > In the example below the pools use 2.92% "%USED" but with my calculation, > taken from the source code it gives me a 8.28%. On a hammer cluster

Re: [ceph-users] ceph inconsistent pg missing ec object

Hi Greg, Thanks! This seems to have worked for at least 1 of 2 inconsistent pgs: The inconsistency disappeared after a new scrub. Still waiting for the result of the second pg. I tried to force deep-scrub with `ceph pg deep-scrub ` yesterday, but today the last deep scrub is still from a

Re: [ceph-users] Erasure pool

2017-11-08 22:05 GMT+01:00 Marc Roos : > > Can anyone advice on a erasure pool config to store > > - files between 500MB and 8GB, total 8TB > - just for archiving, not much reading (few files a week) > - hdd pool > - now 3 node cluster (4th coming) > - would like to save

Re: [ceph-users] High osd cpu usage

Hi, Yes, im using bluestore. there is no I/O on the ceph cluster. it's totally idle. All the CPU usage are by OSD who don't have any workload on it. Thanks! On Thu, Nov 9, 2017 at 9:37 AM, Vy Nguyen Tan wrote: > Hello, > > I think it not normal behavior in Luminous.

Re: [ceph-users] bluestore - wal,db on faster devices?

I get confused there because on the documentation:http://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/ "If there is more, provisioning a DB device makes more sense. The BlueStore journal will always be placed on the fastest device available, so using a DB device will