Re: [ceph-users] /var/lib/ceph/osd/ceph-xxx/current/meta shows "Structure needs cleaning"

2018-03-08 Thread 赵贺东
Hi Wido, Thank you for attention! > 在 2018年3月8日,下午4:21,Wido den Hollander 写道: > > > > On 03/08/2018 08:01 AM, 赵贺东 wrote: >> Hi All, >> Every time after we activate osd, we got “Structure needs cleaning” in >> /var/lib/ceph/osd/ceph-xxx/current/meta. >>

Re: [ceph-users] /var/lib/ceph/osd/ceph-xxx/current/meta shows "Structure needs cleaning"

2018-03-08 Thread Brad Hubbard
On Thu, Mar 8, 2018 at 5:01 PM, 赵贺东 wrote: > Hi All, > > Every time after we activate osd, we got “Structure needs cleaning” in > /var/lib/ceph/osd/ceph-xxx/current/meta. > > > /var/lib/ceph/osd/ceph-xxx/current/meta > # ls -l > ls: reading directory .: Structure needs

Re: [ceph-users] improve single job sequencial read performance.

2018-03-08 Thread Cassiano Pilipavicius
Hi Alex... thank you for the tips! Yesterday I've made a lot of testing and it seems that my network is really what is holding the speed down. I just like to confirm if this is not really a problem or misconfigurantion in my cluster that would be masked by the network upgrade. The cache make

[ceph-users] 19th April 2018: Ceph/Apache CloudStack day in London

2018-03-08 Thread Wido den Hollander
Hello Ceph (and CloudStack ;-) ) people! Together with the Apache CloudStack [0] project we are organizing a Ceph Day in London on April 19th this year. As there are many users using Apache CloudStack with Ceph as the storage behind their Virtual Machines or using Ceph as a object store in

Re: [ceph-users] /var/lib/ceph/osd/ceph-xxx/current/meta shows "Structure needs cleaning"

2018-03-08 Thread 赵赵贺东
Hi Brad, Thank you for your attention. > 在 2018年3月8日,下午4:47,Brad Hubbard 写道: > > On Thu, Mar 8, 2018 at 5:01 PM, 赵贺东 wrote: >> Hi All, >> >> Every time after we activate osd, we got “Structure needs cleaning” in >>

Re: [ceph-users] /var/lib/ceph/osd/ceph-xxx/current/meta shows "Structure needs cleaning"

2018-03-08 Thread Wido den Hollander
On 03/08/2018 08:01 AM, 赵贺东 wrote: Hi All, Every time after we activate osd, we got “Structure needs cleaning” in /var/lib/ceph/osd/ceph-xxx/current/meta. /var/lib/ceph/osd/ceph-xxx/current/meta # ls -l ls: reading directory .: Structure needs cleaning total 0 Could Anyone say something

Re: [ceph-users] /var/lib/ceph/osd/ceph-xxx/current/meta shows "Structure needs cleaning"

2018-03-08 Thread Brad Hubbard
On Thu, Mar 8, 2018 at 7:33 PM, 赵赵贺东 wrote: > Hi Brad, > > Thank you for your attention. > >> 在 2018年3月8日,下午4:47,Brad Hubbard 写道: >> >> On Thu, Mar 8, 2018 at 5:01 PM, 赵贺东 wrote: >>> Hi All, >>> >>> Every time after we activate

Re: [ceph-users] BlueStore questions

2018-03-08 Thread Caspar Smit
Hi Frank, 2018-03-04 1:40 GMT+01:00 Frank Ritchie : > Hi all, > > I have a few questions on using BlueStore. > > With FileStore it is not uncommon to see 1 nvme device being used as the > journal device for up to 12 OSDs. > > Can an adequately sized nvme device also be

[ceph-users] Object Gateway - Server Side Encryption

2018-03-08 Thread Amardeep Singh
Hi, I am trying to configure server side encryption using Key Management Service as per documentation http://docs.ceph.com/docs/master/radosgw/encryption/ Configured Keystone/Barbican integration and its working, tested using curl commands. After I configure RadosGW and use

Re: [ceph-users] pg inconsistent

2018-03-08 Thread Harald Staub
Hi Brad Thank you very much for your attention. On 07.03.2018 23:46, Brad Hubbard wrote: On Thu, Mar 8, 2018 at 1:22 AM, Harald Staub wrote: "ceph pg repair" leads to: 5.7bd repair 2 errors, 0 fixed Only an empty list from: rados list-inconsistent-obj 5.7bd

[ceph-users] OSD crash with segfault Luminous 12.2.4

2018-03-08 Thread Dietmar Rieder
Hi, I noticed in my client (using cephfs) logs that an osd was unexpectedly going down. While checking the osd logs for the affected OSD I found that the osd was seg faulting: [] 2018-03-07 06:01:28.873049 7fd9af370700 -1 *** Caught signal (Segmentation fault) ** in thread 7fd9af370700

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

2018-03-08 Thread Mike Christie
On 03/07/2018 09:24 AM, shadow_lin wrote: > Hi Christie, > Is it safe to use active/passive multipath with krbd with exclusive lock > for lio/tgt/scst/tcmu? No. We tried to use lio and krbd initially, but there is a issue where IO might get stuck in the target/block layer and get executed after

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

2018-03-08 Thread Mike Christie
On 03/08/2018 10:59 AM, Lazuardi Nasution wrote: > Hi Mike, > > Since I have moved from LIO to TGT, I can do full ALUA (active/active) > of multiple gateways. Of course I have to disable any write back cache > at any level (RBD cache and TGT cache). It seem to be safe to disable > exclusive lock

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

2018-03-08 Thread Mike Christie
On 03/08/2018 12:44 PM, Mike Christie wrote: > stuck/queued then your osd_request_timeout value might be too short. For Sorry, I meant too long. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

2018-03-08 Thread Jason Dillaman
On Thu, Mar 8, 2018 at 11:59 AM, Lazuardi Nasution wrote: > Hi Mike, > > Since I have moved from LIO to TGT, I can do full ALUA (active/active) of > multiple gateways. Of course I have to disable any write back cache at any > level (RBD cache and TGT cache). It seem to be

Re: [ceph-users] OSD crash with segfault Luminous 12.2.4

2018-03-08 Thread Subhachandra Chandra
I noticed a similar crash too. Unfortunately, I did not get much info in the logs. *** Caught signal (Segmentation fault) ** Mar 07 17:58:26 data7 ceph-osd-run.sh[796380]: in thread 7f63a0a97700 thread_name:safe_timer Mar 07 17:58:28 data7 ceph-osd-run.sh[796380]: docker_exec.sh: line 56:

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

2018-03-08 Thread Ashish Samant
On 03/08/2018 10:44 AM, Mike Christie wrote: On 03/08/2018 10:59 AM, Lazuardi Nasution wrote: Hi Mike, Since I have moved from LIO to TGT, I can do full ALUA (active/active) of multiple gateways. Of course I have to disable any write back cache at any level (RBD cache and TGT cache). It seem

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

2018-03-08 Thread Jason Dillaman
On Thu, Mar 8, 2018 at 12:47 PM, Lazuardi Nasution wrote: > Jason, > > As long you don't activate any cache and single image for single client > only, it seem impossible to have old data overwrite. May be, it is related > to I/O pattern too. Anyway, maybe other Ceph users

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

2018-03-08 Thread Lazuardi Nasution
Hi Mike, Since I have moved from LIO to TGT, I can do full ALUA (active/active) of multiple gateways. Of course I have to disable any write back cache at any level (RBD cache and TGT cache). It seem to be safe to disable exclusive lock since each RBD image is accessed only by single client and as

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

2018-03-08 Thread Lazuardi Nasution
Jason, As long you don't activate any cache and single image for single client only, it seem impossible to have old data overwrite. May be, it is related to I/O pattern too. Anyway, maybe other Ceph users have different experience. It can be different result with different case. Best regards,

Re: [ceph-users] Civetweb log format

2018-03-08 Thread David Turner
Setting radosgw debug logging to 10/10 is the only way I've been able to get the access key in the logs for requests. It's very unfortunate as it DRASTICALLY increases the amount of log per request, but it's what we needed to do to be able to have the access key in the logs along with the

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

2018-03-08 Thread Jason Dillaman
On Thu, Mar 8, 2018 at 2:11 PM, Ashish Samant wrote: > > > On 03/08/2018 10:44 AM, Mike Christie wrote: >> >> On 03/08/2018 10:59 AM, Lazuardi Nasution wrote: >>> >>> Hi Mike, >>> >>> Since I have moved from LIO to TGT, I can do full ALUA (active/active) >>> of multiple

[ceph-users] set pg_num on pools with different size

2018-03-08 Thread Nagy Ákos
Hi, we have a ceph cluster with 3 cluster nodes and 20 OSD's, with 6-7-7 2 TB HDD/s per node. In long term we want to use 7-9 pools, and for 20 OSD and 8 pools I calculate that the ideal pg_num was 250 (20 * 100 / 8). In this case normally each OSD store 100 pg's, that is the recommanded. I

Re: [ceph-users] Uneven pg distribution cause high fs_apply_latency on osds with more pgs

2018-03-08 Thread David Turner
PGs being unevenly distributed is a common occurrence in Ceph. Luminous started making some steps towards correcting this, but you're in Jewel. There are a lot of threads in the ML archives about fixing PG distribution. Generally every method comes down to increasing the weight on OSDs with too

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

2018-03-08 Thread Jason Dillaman
On Thu, Mar 8, 2018 at 3:41 PM, Lazuardi Nasution wrote: > Hi Jason, > > If there is the case that the gateway cannot access the Ceph, I think you > are right. Anyway, I put iSCSI Gateway on MON node. It's connectivity to the specific OSD associated to the IO operation

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

2018-03-08 Thread Lazuardi Nasution
Hi Jason, I understand. Thank you for your explanation. Best regards, On Mar 9, 2018 3:45 AM, "Jason Dillaman" wrote: > On Thu, Mar 8, 2018 at 3:41 PM, Lazuardi Nasution > wrote: > > Hi Jason, > > > > If there is the case that the gateway cannot

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

2018-03-08 Thread Lazuardi Nasution
Hi Jason, If there is the case that the gateway cannot access the Ceph, I think you are right. Anyway, I put iSCSI Gateway on MON node. Best regards, On Mar 9, 2018 1:41 AM, "Jason Dillaman" wrote: On Thu, Mar 8, 2018 at 12:47 PM, Lazuardi Nasution

Re: [ceph-users] Civetweb log format

2018-03-08 Thread Aaron Bassett
Yea thats what I was afraid of. I'm looking at possibly patching to add it, but i really dont want to support my own builds. I suppose other alternatives are to use proxies to log stuff, but that makes me sad. Aaron On Mar 8, 2018, at 12:36 PM, David Turner

Re: [ceph-users] Civetweb log format

2018-03-08 Thread David Turner
I remember some time ago Yehuda had commented on a thread like this saying that it would make sense to add a logging/auditing feature like this to RGW. I haven't heard much about it since then, though. Yehuda, do you remember that and/or think that logging like this might become viable. On Thu,

Re: [ceph-users] /var/lib/ceph/osd/ceph-xxx/current/meta shows "Structure needs cleaning"

2018-03-08 Thread 赵贺东
Thank you for your suggestions. We will upgrade ubuntu distro and linux kernel to see if the problem still exists or not. > 在 2018年3月8日,下午5:51,Brad Hubbard 写道: > > On Thu, Mar 8, 2018 at 7:33 PM, 赵赵贺东 > wrote: >> Hi

Re: [ceph-users] Uneven pg distribution cause high fs_apply_latency on osds with more pgs

2018-03-08 Thread shadow_lin
Thanks for your advice. I will try to reweight osds of my cluster. Why ceph is so sensitive to unblanced pg distribution during high load? ceph osd df result is: https://pastebin.com/ur4Q9jsA. ceph osd perf result is: https://pastebin.com/87DitPhV There is no osd with very high pg count

Re: [ceph-users] OSD crash with segfault Luminous 12.2.4

2018-03-08 Thread Brad Hubbard
On Fri, Mar 9, 2018 at 3:54 AM, Subhachandra Chandra wrote: > I noticed a similar crash too. Unfortunately, I did not get much info in the > logs. > > *** Caught signal (Segmentation fault) ** > > Mar 07 17:58:26 data7 ceph-osd-run.sh[796380]: in thread 7f63a0a97700 >

Re: [ceph-users] Civetweb log format

2018-03-08 Thread Yehuda Sadeh-Weinraub
On Thu, Mar 8, 2018 at 2:22 PM, David Turner wrote: > I remember some time ago Yehuda had commented on a thread like this saying > that it would make sense to add a logging/auditing feature like this to RGW. > I haven't heard much about it since then, though. Yehuda, do

Re: [ceph-users] Problem with UID starting with underscores

2018-03-08 Thread Konstantin Shalygin
because one our script misbehaved, new user with bad UID was created via API, and now we can't remove, view or modify it. I believe, it's because it has three underscores at the beginning: Same problem here http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-February/024578.html You

Re: [ceph-users] Civetweb log format

2018-03-08 Thread Matt Benjamin
Hi Yehuda, I did add support for logging arbitrary headers, but not a configurable log record a-la webservers. To level set, David, are you speaking about a file or pipe log sync on the RGW host? Matt On Thu, Mar 8, 2018 at 7:55 PM, Yehuda Sadeh-Weinraub wrote: > On Thu,

[ceph-users] Bluestore bluestore_prefer_deferred_size and WAL size

2018-03-08 Thread Budai Laszlo
Dear all, I'm reading about the bluestore_prefer_deferred_size parameter for Bluestore. Are there any hints about its size when using a dedicated SSD for bock.wal and block.db ? Thank you in advance! Laszlo ___ ceph-users mailing list

Re: [ceph-users] change radosgw object owner

2018-03-08 Thread Ryan Leimenstoll
Hi Robin, Thanks for the pointer! My one concern though is that it didn’t seem to update the original object owner’s quota however, which is a bit of a sticking point. Is this expected (and is there a workaround)? I will admit to being a bit naive to how radosgw’s quota system works under the