[ceph-users] Ceph REST API

Hi, As I referring the below link for Ceph REST API http://docs.ceph.com/docs/mimic/mgr/restful/ Some of the other endpoints implemented in the restful module include:

Re: [ceph-users] How to use STS Lite correctly?

When I changed the rgw sts key to 16 bits, it started working as expected. Thank you very much to my friend. myxingkong ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 13.2.4 odd memory leak?

Yeah, there's a bug in 13.2.4. You need to set it to at least ~1.2GB. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Tue, Mar 5, 2019 at 9:00 AM Steffen Winther

Re: [ceph-users] Ceph cluster on AMD based system.

Not with this particular server, but we've played around with two EPYCs system with 10 NVMe in each and 100 Gbit/s network between them. Make sure to use a recent Linux kernel, but other than that it works fine. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at

Re: [ceph-users] 13.2.4 odd memory leak?

> On 4 Mar 2019, at 16.09, Paul Emmerich wrote: > > Bloated to ~4 GB per OSD and you are on HDDs? Something like that yes. > > 13.2.3 backported the cache auto-tuning which targets 4 GB memory > usage by default. > > See https://ceph.com/releases/13-2-4-mimic-released/ >

[ceph-users] optimize bluestore for random write i/o

Hello list, while the performance of sequential writes 4k on bluestore is very high and even higher than filestore i was wondering what i can do to optimize random pattern as well. While using: fio --rw=write --iodepth=32 --ioengine=libaio --bs=4k --numjobs=4 --filename=/tmp/test --size=10G

Re: [ceph-users] optimize bluestore for random write i/o

This workload is probably bottlenecked by rocksdb (since the small writes are buffered there), so that's probably what needs tuning here. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel:

Re: [ceph-users] 14.1.0, No dashboard module

As a follow up seems the dashboard is a separate package not installed by default called "ceph-mgr-dashboard" Seems this is currently missing off the RC notes, and the master doc for ceph dashboard. Cheers On Tue, Mar 5, 2019 at 10:54 AM Ashley Merrick wrote: > I have just spun up a small

[ceph-users] chown -R on every osd activating

Hi, we recently updated to ceph luminous 12.2.11 after running in this bug: http://tracker.ceph.com/issues/37784. But this is a other story. Now after rebooting a host I see there is a chown -R ceph:ceph running on each OSD before the OSD daemon starts. This takes a lot of time (-> millions of

[ceph-users] Ceph cluster on AMD based system.

Hello, I was thinking of using AMD based system for my new nvme based cluster. In particular I'm looking at https://www.supermicro.com/Aplus/system/1U/1113/AS-1113S-WN10RT.cfm and https://www.amd.com/en/products/cpu/amd-epyc-7451 CPU's. Have anyone tried running it on this particular hardware?

Re: [ceph-users] optimize bluestore for random write i/o

Am 05.03.19 um 10:05 schrieb Paul Emmerich: > This workload is probably bottlenecked by rocksdb (since the small > writes are buffered there), so that's probably what needs tuning here. while reading:

Re: [ceph-users] chown -R on every osd activating

On Tue, 5 Mar 2019 11:04:16 +0100 Paul Emmerich wrote: > On Tue, Mar 5, 2019 at 10:51 AM Manuel Lausch > wrote: > > Now after rebooting a host I see there is a chown -R ceph:ceph > > running on each OSD before the OSD daemon starts. > > > > This takes a lot of time (-> millions of objects per

Re: [ceph-users] Ceph cluster on AMD based system.

If your crushmap is set to replicate by host you would only ever have one copy on a single host, no matter how many OSD’s you placed on a single NVME/disk. But yes you would not want to mix OSD based rules and multiple OSD per a physical disk. On Tue, 5 Mar 2019 at 7:54 PM, Marc Roos wrote: >

Re: [ceph-users] chown -R on every osd activating

On Tue, Mar 5, 2019 at 10:51 AM Manuel Lausch wrote: > Now after rebooting a host I see there is a chown -R ceph:ceph running > on each OSD before the OSD daemon starts. > > This takes a lot of time (-> millions of objects per OSD) and I think > this is unneccessary on each startup. In my opinion

Re: [ceph-users] Experiences with the Samsung SM/PM883 disk?

Well, tried a different disk and basically got the same results as Jacob. So I've just had a bad disk there (server was fine as other disks work in it). Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München

Re: [ceph-users] Ceph cluster on AMD based system.

I see indeed lately people writing about putting 2 osd on a nvme, but does this not undermine the idea of having 3 copies on different osds/drives? In theory you could loose 2 copies when one disk fails??? -Original Message- From: Darius Kasparaviius [mailto:daz...@gmail.com]

Re: [ceph-users] How to just delete PGs stuck incomplete on EC pool

Last time I had to do this, I used the command outlined here: https://tracker.ceph.com/issues/10098 On Mon, Mar 4, 2019 at 11:05 AM Daniel K wrote: > > Thanks for the suggestions. > > I've tried both -- setting osd_find_best_info_ignore_history_les = true and > restarting all OSDs, as well as

Re: [ceph-users] Mounting image from erasure-coded pool without tiering in KVM

It looks like your trying to mount using krbd unless I am reading your XML wrong. Does the image that you create in the replicated pool have the same features as the above "layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool" Also the user cephx user your using has full

[ceph-users] Mounting image from erasure-coded pool without tiering in KVM

Hello, all! I have a problem with adding image volumes to my KVM VM. I prepared erasure coded pool (named data01) on full-bluestore OSDs and allowed ec_overwrites on it. Also i created replicated pool for image volumes metadata named ssd-repl. Pools were prepared by: ceph osd pool create data01

Re: [ceph-users] Ceph cluster on AMD based system.

Hi, I've got a ryzen7 1700 box that I regularly run tests on along with the upstream community performance test nodes that have Intel Xeon E5-2650v3 processors in them.  The Ryzen is 3.0GHz/3.7GHz turbo while the Xeons are 2.3GHz/3.0GHz.  The Xeons are quite a bit faster clock/clock in the

Re: [ceph-users] 14.1.0, No dashboard module

Hi Ashley, thanks for pointing this out! I've created a tracker issue [1] and we will take care of updating the documentation accordingly. Thanks, Laura [1] https://tracker.ceph.com/issues/38584 On 05.03.19 10:16, Ashley Merrick wrote: > As a follow up seems the dashboard is a separate

Re: [ceph-users] radosgw sync falling behind regularly

Hi Matthew, I work with Christian. Thanks so much for looking at this. We have a huge stale-instances list from that command. Our periods are all the same, I redirected them to a file on each node and checksummed them. Here's the period: { "id": "3d0d40ef-90de-40ea-8c44-caa20ea8dc53",

Re: [ceph-users] radosgw sync falling behind regularly

Matthew, first of all, let me say we very much appreciate your help! So I don’t think we turned dynamic resharding on, nor did we manually reshard buckets. Seems like it defaults to on for luminous but the mimic docs say it’s not supported in multisite. So do we need to disable it manually

[ceph-users] Deploy Cehp in multisite setup

Hi. I'm looking for a distributed filesystem that would distribute a single namespace with POSIX access. It should be scaleable upwards to be easily to add more physical drives as storage fills up with time. The deepscrub and snapshotting are essential. I have been studying different options

Re: [ceph-users] radosgw sync falling behind regularly

Hi Christian, To be on the safe side and future proof yourself will want to go ahead and set the following in your ceph.conf file, and then issue a restart to your RGW instances. rgw_dynamic_resharding = false There are a number of issues with dynamic resharding, multisite rgw problems being

Re: [ceph-users] Deploy Cehp in multisite setup

Choose two: * POSIX filesystem with a reliable storage underneath * Multiple sites with poor or high-latency connection between them * Performance -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io

Re: [ceph-users] Ceph cluster on AMD based system.

Thank you for your response. I was planning to use a 100GbE or 45GbE bond for this cluster. It was acceptable for our use case to lose sequential/larger I/O speed for it. Dual socket would be and option, but I do not want to touch numa, cgroups and the rest settings. Most of the time is just

Re: [ceph-users] How to use STS Lite correctly?

05T024604Z > Authorization: AWS4-HMAC-SHA256 Credential=O966WM2NEUB232Z53VYG/20190305 > //sts/aws4_request, SignedHeaders=content-type;host;x-amz-date, > Signature=dfb51d46ca561fa7bf763ceaededf58afd17b3fe6293c4cc6dc4fccba24c95d1 > User-Agent: Boto3/1.9.106 Python/2.7.15 Windows/7 B

Re: [ceph-users] Ceph cluster on AMD based system.

On 3/5/19 10:20 AM, Darius Kasparavičius wrote: Thank you for your response. I was planning to use a 100GbE or 45GbE bond for this cluster. It was acceptable for our use case to lose sequential/larger I/O speed for it. Dual socket would be and option, but I do not want to touch numa, cgroups

Re: [ceph-users] optimize bluestore for random write i/o

Hi Stefan, Could you try running your random write workload against bluestore and then take a wallclock profile of an OSD using gdbpmp? It's available here: https://github.com/markhpc/gdbpmp Thanks, Mark On 3/5/19 2:29 AM, Stefan Priebe - Profihost AG wrote: Hello list, while the

Re: [ceph-users] RBD poor performance

Yes (I mean yes, it's real). Ceph's tiering works by moving whole (4MB) objects to the cache pool, updating them there (with 4K random writes?) and evicting them back when cache is full. I.e. the bad part here is that it can't do "write-through". Also there are some configuration options

Re: [ceph-users] radosgw sync falling behind regularly

Hi Christian, I think you've correctly intuited that the issues are related to the use of 'bucket sync disable'. There was a bug fix for that feature in http://tracker.ceph.com/issues/26895, and I recently found that a block of code was missing from its luminous backport. That missing code is

Re: [ceph-users] radosgw sync falling behind regularly

Casey, Thanks very much for the reply! We definitely have lots of errors on sync-disabled buckets and the workaround for that is obvious (most of them are empty anyway). Our second form of error is stale buckets. We had dynamic resharding enabled but have now disabled it (having discovered it

Re: [ceph-users] optimize bluestore for random write i/o

Testing -rw=write without -sync=1 or -fsync=1 (or -fsync=32 for batch IO, or just fio -ioengine=rbd from outside a VM) is rather pointless - you're benchmarking the RBD cache, not Ceph itself. RBD cache is coalescing your writes into big sequential writes. Of course bluestore is faster in

[ceph-users] mount cephfs on ceph servers

Hi, I'm gonna mount cephfs from my ceph servers for some reason, including monitors, metadata servers and osd servers. I know it's not a best practice. But what is the exact potential danger if I mount cephfs from its own server? Thanks ___ ceph-users

Re: [ceph-users] ceph bug#2445 hitting version-12.2.4

Hi - we are using ceph 12.2.4 and bug#24445 hitting, which caused 10 min IO pause on ceph cluster.. Is this bug fixed? bug:https://tracker.ceph.com/issues/24445/ Seems this is a network issue, not ceph. Reporter of this ticket was never come backs. k

Re: [ceph-users] Ceph cluster on AMD based system.

On Tue, 5 Mar 2019 10:39:14 -0600 Mark Nelson wrote: > On 3/5/19 10:20 AM, Darius Kasparavičius wrote: > > Thank you for your response. > > > > I was planning to use a 100GbE or 45GbE bond for this cluster. It was > > acceptable for our use case to lose sequential/larger I/O speed for > > it.