Re: [ceph-users] Uneven pg distribution cause high fs_apply_latency on osds with more pgs

Thanks for your advice. I will try to reweight osds of my cluster. Why ceph is so sensitive to unblanced pg distribution during high load? ceph osd df result is: https://pastebin.com/ur4Q9jsA. ceph osd perf result is: https://pastebin.com/87DitPhV There is no osd with very high pg count

Re: [ceph-users] Problem with UID starting with underscores

because one our script misbehaved, new user with bad UID was created via API, and now we can't remove, view or modify it. I believe, it's because it has three underscores at the beginning: Same problem here http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-February/024578.html You

Re: [ceph-users] /var/lib/ceph/osd/ceph-xxx/current/meta shows "Structure needs cleaning"

Thank you for your suggestions. We will upgrade ubuntu distro and linux kernel to see if the problem still exists or not. > 在 2018年3月8日，下午5:51，Brad Hubbard 写道： > > On Thu, Mar 8, 2018 at 7:33 PM, 赵赵贺东 > wrote: >> Hi

Re: [ceph-users] Civetweb log format

Hi Yehuda, I did add support for logging arbitrary headers, but not a configurable log record a-la webservers. To level set, David, are you speaking about a file or pipe log sync on the RGW host? Matt On Thu, Mar 8, 2018 at 7:55 PM, Yehuda Sadeh-Weinraub wrote: > On Thu,

Re: [ceph-users] Civetweb log format

On Thu, Mar 8, 2018 at 2:22 PM, David Turner wrote: > I remember some time ago Yehuda had commented on a thread like this saying > that it would make sense to add a logging/auditing feature like this to RGW. > I haven't heard much about it since then, though. Yehuda, do

Re: [ceph-users] OSD crash with segfault Luminous 12.2.4

On Fri, Mar 9, 2018 at 3:54 AM, Subhachandra Chandra wrote: > I noticed a similar crash too. Unfortunately, I did not get much info in the > logs. > > *** Caught signal (Segmentation fault) ** > > Mar 07 17:58:26 data7 ceph-osd-run.sh[796380]: in thread 7f63a0a97700 >

Re: [ceph-users] Uneven pg distribution cause high fs_apply_latency on osds with more pgs

PGs being unevenly distributed is a common occurrence in Ceph. Luminous started making some steps towards correcting this, but you're in Jewel. There are a lot of threads in the ML archives about fixing PG distribution. Generally every method comes down to increasing the weight on OSDs with too

[ceph-users] set pg_num on pools with different size

Hi, we have a ceph cluster with 3 cluster nodes and 20 OSD's, with 6-7-7 2 TB HDD/s per node. In long term we want to use 7-9 pools, and for 20 OSD and 8 pools I calculate that the ideal pg_num was 250 (20 * 100 / 8). In this case normally each OSD store 100 pg's, that is the recommanded. I

Re: [ceph-users] Civetweb log format

I remember some time ago Yehuda had commented on a thread like this saying that it would make sense to add a logging/auditing feature like this to RGW. I haven't heard much about it since then, though. Yehuda, do you remember that and/or think that logging like this might become viable. On Thu,

Re: [ceph-users] Civetweb log format

Yea thats what I was afraid of. I'm looking at possibly patching to add it, but i really dont want to support my own builds. I suppose other alternatives are to use proxies to log stuff, but that makes me sad. Aaron On Mar 8, 2018, at 12:36 PM, David Turner

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

Hi Jason, I understand. Thank you for your explanation. Best regards, On Mar 9, 2018 3:45 AM, "Jason Dillaman" wrote: > On Thu, Mar 8, 2018 at 3:41 PM, Lazuardi Nasution > wrote: > > Hi Jason, > > > > If there is the case that the gateway cannot

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

On Thu, Mar 8, 2018 at 3:41 PM, Lazuardi Nasution wrote: > Hi Jason, > > If there is the case that the gateway cannot access the Ceph, I think you > are right. Anyway, I put iSCSI Gateway on MON node. It's connectivity to the specific OSD associated to the IO operation

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

Hi Jason, If there is the case that the gateway cannot access the Ceph, I think you are right. Anyway, I put iSCSI Gateway on MON node. Best regards, On Mar 9, 2018 1:41 AM, "Jason Dillaman" wrote: On Thu, Mar 8, 2018 at 12:47 PM, Lazuardi Nasution

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

On Thu, Mar 8, 2018 at 2:11 PM, Ashish Samant wrote: > > > On 03/08/2018 10:44 AM, Mike Christie wrote: >> >> On 03/08/2018 10:59 AM, Lazuardi Nasution wrote: >>> >>> Hi Mike, >>> >>> Since I have moved from LIO to TGT, I can do full ALUA (active/active) >>> of multiple

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

On 03/08/2018 10:44 AM, Mike Christie wrote: On 03/08/2018 10:59 AM, Lazuardi Nasution wrote: Hi Mike, Since I have moved from LIO to TGT, I can do full ALUA (active/active) of multiple gateways. Of course I have to disable any write back cache at any level (RBD cache and TGT cache). It seem

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

On 03/08/2018 12:44 PM, Mike Christie wrote: > stuck/queued then your osd_request_timeout value might be too short. For Sorry, I meant too long. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

On 03/08/2018 10:59 AM, Lazuardi Nasution wrote: > Hi Mike, > > Since I have moved from LIO to TGT, I can do full ALUA (active/active) > of multiple gateways. Of course I have to disable any write back cache > at any level (RBD cache and TGT cache). It seem to be safe to disable > exclusive lock

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

On Thu, Mar 8, 2018 at 12:47 PM, Lazuardi Nasution wrote: > Jason, > > As long you don't activate any cache and single image for single client > only, it seem impossible to have old data overwrite. May be, it is related > to I/O pattern too. Anyway, maybe other Ceph users

Re: [ceph-users] OSD crash with segfault Luminous 12.2.4

I noticed a similar crash too. Unfortunately, I did not get much info in the logs. *** Caught signal (Segmentation fault) ** Mar 07 17:58:26 data7 ceph-osd-run.sh[796380]: in thread 7f63a0a97700 thread_name:safe_timer Mar 07 17:58:28 data7 ceph-osd-run.sh[796380]: docker_exec.sh: line 56:

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

Jason, As long you don't activate any cache and single image for single client only, it seem impossible to have old data overwrite. May be, it is related to I/O pattern too. Anyway, maybe other Ceph users have different experience. It can be different result with different case. Best regards,

Re: [ceph-users] Civetweb log format

Setting radosgw debug logging to 10/10 is the only way I've been able to get the access key in the logs for requests. It's very unfortunate as it DRASTICALLY increases the amount of log per request, but it's what we needed to do to be able to have the access key in the logs along with the

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

On Thu, Mar 8, 2018 at 11:59 AM, Lazuardi Nasution wrote: > Hi Mike, > > Since I have moved from LIO to TGT, I can do full ALUA (active/active) of > multiple gateways. Of course I have to disable any write back cache at any > level (RBD cache and TGT cache). It seem to be

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

Hi Mike, Since I have moved from LIO to TGT, I can do full ALUA (active/active) of multiple gateways. Of course I have to disable any write back cache at any level (RBD cache and TGT cache). It seem to be safe to disable exclusive lock since each RBD image is accessed only by single client and as

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

On 03/07/2018 09:24 AM, shadow_lin wrote: > Hi Christie, > Is it safe to use active/passive multipath with krbd with exclusive lock > for lio/tgt/scst/tcmu? No. We tried to use lio and krbd initially, but there is a issue where IO might get stuck in the target/block layer and get executed after

[ceph-users] Bluestore bluestore_prefer_deferred_size and WAL size

Dear all, I'm reading about the bluestore_prefer_deferred_size parameter for Bluestore. Are there any hints about its size when using a dedicated SSD for bock.wal and block.db ? Thank you in advance! Laszlo ___ ceph-users mailing list

Re: [ceph-users] change radosgw object owner

Hi Robin, Thanks for the pointer! My one concern though is that it didn’t seem to update the original object owner’s quota however, which is a bit of a sticking point. Is this expected (and is there a workaround)? I will admit to being a bit naive to how radosgw’s quota system works under the

[ceph-users] OSD crash with segfault Luminous 12.2.4

Hi, I noticed in my client (using cephfs) logs that an osd was unexpectedly going down. While checking the osd logs for the affected OSD I found that the osd was seg faulting: [] 2018-03-07 06:01:28.873049 7fd9af370700 -1 *** Caught signal (Segmentation fault) ** in thread 7fd9af370700

[ceph-users] Object Gateway - Server Side Encryption

Hi, I am trying to configure server side encryption using Key Management Service as per documentation http://docs.ceph.com/docs/master/radosgw/encryption/ Configured Keystone/Barbican integration and its working, tested using curl commands. After I configure RadosGW and use

Re: [ceph-users] pg inconsistent

Hi Brad Thank you very much for your attention. On 07.03.2018 23:46, Brad Hubbard wrote: On Thu, Mar 8, 2018 at 1:22 AM, Harald Staub wrote: "ceph pg repair" leads to: 5.7bd repair 2 errors, 0 fixed Only an empty list from: rados list-inconsistent-obj 5.7bd

Re: [ceph-users] /var/lib/ceph/osd/ceph-xxx/current/meta shows "Structure needs cleaning"

On Thu, Mar 8, 2018 at 7:33 PM, 赵赵贺东 wrote: > Hi Brad, > > Thank you for your attention. > >> 在 2018年3月8日，下午4:47，Brad Hubbard 写道： >> >> On Thu, Mar 8, 2018 at 5:01 PM, 赵贺东 wrote: >>> Hi All, >>> >>> Every time after we activate

Re: [ceph-users] /var/lib/ceph/osd/ceph-xxx/current/meta shows "Structure needs cleaning"

Hi Wido, Thank you for attention! > 在 2018年3月8日，下午4:21，Wido den Hollander 写道： > > > > On 03/08/2018 08:01 AM, 赵贺东 wrote: >> Hi All, >> Every time after we activate osd, we got “Structure needs cleaning” in >> /var/lib/ceph/osd/ceph-xxx/current/meta. >>

Re: [ceph-users] /var/lib/ceph/osd/ceph-xxx/current/meta shows "Structure needs cleaning"

Hi Brad, Thank you for your attention. > 在 2018年3月8日，下午4:47，Brad Hubbard 写道： > > On Thu, Mar 8, 2018 at 5:01 PM, 赵贺东 wrote: >> Hi All, >> >> Every time after we activate osd, we got “Structure needs cleaning” in >>

[ceph-users] 19th April 2018: Ceph/Apache CloudStack day in London

Hello Ceph (and CloudStack ;-) ) people! Together with the Apache CloudStack [0] project we are organizing a Ceph Day in London on April 19th this year. As there are many users using Apache CloudStack with Ceph as the storage behind their Virtual Machines or using Ceph as a object store in

Re: [ceph-users] improve single job sequencial read performance.

Hi Alex... thank you for the tips! Yesterday I've made a lot of testing and it seems that my network is really what is holding the speed down. I just like to confirm if this is not really a problem or misconfigurantion in my cluster that would be masked by the network upgrade. The cache make

Re: [ceph-users] /var/lib/ceph/osd/ceph-xxx/current/meta shows "Structure needs cleaning"

On 03/08/2018 08:01 AM, 赵贺东 wrote: Hi All, Every time after we activate osd, we got “Structure needs cleaning” in /var/lib/ceph/osd/ceph-xxx/current/meta. /var/lib/ceph/osd/ceph-xxx/current/meta # ls -l ls: reading directory .: Structure needs cleaning total 0 Could Anyone say something

Re: [ceph-users] /var/lib/ceph/osd/ceph-xxx/current/meta shows "Structure needs cleaning"

On Thu, Mar 8, 2018 at 5:01 PM, 赵贺东 wrote: > Hi All, > > Every time after we activate osd, we got “Structure needs cleaning” in > /var/lib/ceph/osd/ceph-xxx/current/meta. > > > /var/lib/ceph/osd/ceph-xxx/current/meta > # ls -l > ls: reading directory .: Structure needs

Re: [ceph-users] BlueStore questions

Hi Frank, 2018-03-04 1:40 GMT+01:00 Frank Ritchie : > Hi all, > > I have a few questions on using BlueStore. > > With FileStore it is not uncommon to see 1 nvme device being used as the > journal device for up to 12 OSDs. > > Can an adequately sized nvme device also be