Re: [ceph-users] Troubleshooting rgw bucket list

Hi, see inline On 01-09-15 20:14, Yehuda Sadeh-Weinraub wrote: > I assume you filtered the log by thread? I don't see the response > messages. For the bucket check you can run radosgw-admin with > --log-to-stderr. nothing is logged to the console when I do that > > Can you also set 'debug

Re: [ceph-users] Troubleshooting rgw bucket list

not sure where I can find the logs for the bucket check, I can't really filter them out in the radosgw log. -Sam On 01-09-15 19:25, Sam Wouters wrote: > It looks like it, this is what shows in the logs after bumping the debug > and requesting a bucket list. > > 2015-09-01 17:14:53.008620

Re: [ceph-users] Accelio & Ceph

Hi Roy, I understand, we are looking for using accelio with an starting small cluster of 3 mon and 8 osd servers: 3x MON servers 2x Intel Xeon E5-2630v3 @2.40Ghz (32C with HT) 24x 16GB DIMM DDR3 1333Mhz (384GB) 2x 120GB Intel SSD DC S3500 (RAID-1 for OS) 1x ConnectX-3 VPI FDR

Re: [ceph-users] Accelio & Ceph

Thanks Roy, we're planning to grow on this cluster if can get the performance that we need, the idea is to run non-relational databases here, so it would be high-io intensive. We are talking in grow terms of about 40-50 OSD servers with no more than 6 OSD daemons per server. If you got some hints

Re: [ceph-users] Accelio & Ceph

Thanks ! I think you should try installing from the ceph mainstream..There are some bug fixes went on after Hammer (not sure if it is backported).. I would say try with 1 drive -> 1 OSD first since presently we have seen some stability issues (mainly due to resource constraint) with more OSDs in

Re: [ceph-users] Accelio & Ceph

Thanks ! 6 OSD daemons per server should be good. Vu, Could you please send out the doc you are maintaining ? Regards Somnath From: German Anders [mailto:gand...@despegar.com] Sent: Tuesday, September 01, 2015 11:36 AM To: Somnath Roy Cc: Robert LeBlanc; ceph-users Subject: Re: [ceph-users]

Re: [ceph-users] Troubleshooting rgw bucket list

Sorry, forgot to mention: - yes, filtered by thread - the "is not valid" line occurred when performing the bucket --check - when doing a bucket listing, I also get an "is not valid", but on a different object: 7fe4f1d5b700 20 cls/rgw/cls_rgw.cc:460: entry

Re: [ceph-users] Troubleshooting rgw bucket list

It looks like it, this is what shows in the logs after bumping the debug and requesting a bucket list. 2015-09-01 17:14:53.008620 7fccb17ca700 10 cls_bucket_list aws-cmis-prod(@{i=.be-east.rgw.buckets.index}.be-east.rgw.buckets[be-east.5436.1]) start

Re: [ceph-users] Troubleshooting rgw bucket list

I assume you filtered the log by thread? I don't see the response messages. For the bucket check you can run radosgw-admin with --log-to-stderr. Can you also set 'debug objclass = 20' on the osds? You can do it by: $ ceph tell osd.\* injectargs --debug-objclass 20 Also, it'd be interesting to

Re: [ceph-users] Ceph Performance Questions with rbd images access by qemu-kvm

Be selective with the SSDs you choose. I personally have tried Micron M500DC, Intel S3500, and some PCIE cards that would all suffice. There are MANY that do not work well at all. A shockingly large list, in fact. Intel 3500/3700 are the gold standards. Warren From: ceph-users

Re: [ceph-users] Accelio & Ceph

Thanks a lot for the quick response Robert, any idea when it's going to be ready for production? any alternative solution for similar-performance? Best regards, *German * 2015-09-01 13:42 GMT-03:00 Robert LeBlanc : > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > >

Re: [ceph-users] Accelio & Ceph

Hi German, We are working on to make it production ready ASAP. As you know RDMA is very resource constrained and at the same time will outperform TCP as well. There will be some definite tradeoff between cost Vs Performance. We are lacking on ideas on how big the RDMA deployment could be and it

Re: [ceph-users] radosgw secret_key

Look at this: https://github.com/ncw/rclone/issues/47 Because this is a json dump, it is encoding the / as \/. It was source of confusion also for me. Best regards Saverio 2015-08-24 16:58 GMT+02:00 Luis Periquito : > When I create a new user using radosgw-admin most

Re: [ceph-users] Append data via librados C API in erasure coded pool

I think this could be misaligned writes. Is it multiple of 4k ?? Its just a wild guess. thanks, Shylesh On Tue, Sep 1, 2015 at 9:17 AM, Hercules wrote: > Hello, > > I use librados C API rados_append() to append object data in erasure coded > pool, it always return -95

Re: [ceph-users] any recommendation of using EnhanceIO?

> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Wang, Zhiqiang > Sent: 01 September 2015 02:48 > To: Nick Fisk ; 'Samuel Just' > Cc: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] any

[ceph-users] libvirt rbd issue

Hi ceph-users, Hoping to get some help with a tricky problem. I have a rhel7.1 VM guest (host machine also rhel7.1) with root disk presented from ceph 0.94.2-0 (rbd) using libvirt. The VM also has a second rbd for storage presented from the same ceph cluster, also using libvirt. The VM boots

Re: [ceph-users] Accelio & Ceph

Hi Vu, Thanks a lot for the link Best regards, *German* 2015-09-01 19:02 GMT-03:00 Vu Pham : > Hi German, > > > > You can try this small wiki to setup ceph/accelio > > > > https://community.mellanox.com/docs/DOC-2141 > > > > thanks, > > -vu > > > > > > *From:*

Re: [ceph-users] Rados: Undefined symbol error

- Original Message - > From: "Aakanksha Pudipeddi-SSI" > To: "Brad Hubbard" > Sent: Wednesday, 2 September, 2015 6:25:49 AM > Subject: RE: [ceph-users] Rados: Undefined symbol error > > Hello Brad, > > I wanted to clarify the "make

Re: [ceph-users] Ceph SSD CPU Frequency Benchmarks

-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Nick, I've been trying to replicate your results without success. Can you help me understand what I'm doing that is not the same as your test? My setup is two boxes, one is a client and the other is a server. The server has Intel(R) Atom(TM) CPU

Re: [ceph-users] Moving/Sharding RGW Bucket Index

We also run RGW buckets with many millions of objects and had to shard our existing buckets. We did have to delete the old ones first, unfortunately. I haven't tried moving the index pool to an SSD ruleset - would also be interested in folks' experiences with this. Thanks for the information on

[ceph-users] librados application consultant needed

We have an application built on top of librados that has barely acceptable performance and in need of optimizations. Since the code is functionally correct, we have a hard time freeing up the resources to fully investigate where the bottlenecks occur and fix them. We would like to hire a

Re: [ceph-users] Ceph SSD CPU Frequency Benchmarks

> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Robert LeBlanc > Sent: 01 September 2015 21:48 > To: Nick Fisk > Cc: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] Ceph SSD CPU Frequency Benchmarks > > -BEGIN

Re: [ceph-users] Accelio & Ceph

Hi German, You can try this small wiki to setup ceph/accelio https://community.mellanox.com/docs/DOC-2141 thanks, -vu From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of German Anders Sent: Tuesday, September 01, 2015 12:00 PM To: Somnath Roy Cc: ceph-users Subject: Re:

[ceph-users] cephfs read-only setting doesn't work?

Hi, I tried to set up a read-only permission for a client but it looks always writable. I did the following: ==Server end== [client.cephfs_data_ro] key = AQxx== caps mon = "allow r" caps osd = "allow r pool=cephfs_data, allow r pool=cephfs_metadata"

Re: [ceph-users] Append data via librados C API in erasure coded pool

Hi, Like Shylesh said: you need to obey alignment constraints. See rados_ioctx_pool_requires_alignment in http://ceph.com/docs/hammer/rados/api/librados/ Cheers On 01/09/2015 08:49, shylesh kumar wrote: > I think this could be misaligned writes. > Is it multiple of 4k ?? Its just a wild

Re: [ceph-users] any recommendation of using EnhanceIO?

> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Wang, Zhiqiang > Sent: 01 September 2015 09:48 > To: Nick Fisk ; 'Samuel Just' > Cc: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] any

Re: [ceph-users] any recommendation of using EnhanceIO?

> -Original Message- > From: Nick Fisk [mailto:n...@fisk.me.uk] > Sent: Tuesday, September 1, 2015 3:55 PM > To: Wang, Zhiqiang; 'Nick Fisk'; 'Samuel Just' > Cc: ceph-users@lists.ceph.com > Subject: RE: [ceph-users] any recommendation of using EnhanceIO? > > > > > > > -Original

Re: [ceph-users] Troubleshooting rgw bucket list

Hi, I've started the bucket --check --fix on friday evening and it's still running. 'ceph -s' shows the cluster health as OK, I don't know if there is anything else I could check? Is there a way of finding out if its actually doing something? We only have this issue on the one bucket with

Re: [ceph-users] any recommendation of using EnhanceIO?

> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Wang, Zhiqiang > Sent: 01 September 2015 09:18 > To: Nick Fisk ; 'Samuel Just' > Cc: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] any

Re: [ceph-users] Testing CephFS

> On Sep 1, 2015, at 16:13, Simon Hallam wrote: > > Hi Greg, Zheng, > > Is this fixed in a later version of the kernel client? Or would it be wise > for us to start using the fuse client? > > Cheers, I just wrote a fix

Re: [ceph-users] Testing CephFS

Hi Greg, Zheng, Is this fixed in a later version of the kernel client? Or would it be wise for us to start using the fuse client? Cheers, Simon > -Original Message- > From: Gregory Farnum [mailto:gfar...@redhat.com] > Sent: 31 August 2015 13:02 > To: Yan, Zheng > Cc: Simon Hallam;

Re: [ceph-users] any recommendation of using EnhanceIO?

> -Original Message- > From: Nick Fisk [mailto:n...@fisk.me.uk] > Sent: Tuesday, September 1, 2015 4:37 PM > To: Wang, Zhiqiang; 'Samuel Just' > Cc: ceph-users@lists.ceph.com > Subject: RE: [ceph-users] any recommendation of using EnhanceIO? > > > > > > > -Original Message- >

Re: [ceph-users] OSD won't go up after node reboot

Data lives in another container attached to OSD container as Docker volume. According to `deis ps -a`, this volume was created two weeks ago, though all files in `current` are very recent. I suspect that something removed files in the data volume after reboot. As reboot was caused by CoreOS

Re: [ceph-users] ceph distributed osd

Hi Robert, We are going to use ceph with ocfs2 in production. Here my doubt is rbd mounted in 12 clients using ocfs2 clustering and network for server client will be 1 Gig. Is the throughput performance is ok for this setup? Regards Prabu On Thu, 20 Aug 2015 02:15:53

[ceph-users] How should I deal with placement group numbers when reducing number of OSDs

Hi, we're in the process of changing 480G drives for 1200G drives, which should cut the number of OSDs I have roughly to 1/3. My largest "volumes" pool for OpenStack volumes has 16384 PGs at the moment and I have 36K PGs in total. That equals to ~180 PGs/OSD and would become ~500 PG/s OSD. I

Re: [ceph-users] Firefly to Hammer Upgrade -- HEALTH_WARN; too many PGs per OSD (480 > max 300)

Hi Greg, Thanks for the update.. I think the documentation on Ceph should be reworded. --snip-- http://ceph.com/docs/master/rados/operations/placement-groups/#choosing-the-number-of-placement-groups * Less than 5 OSDs set pg_num to 128 * Between 5 and 10 OSDs set pg_num to 512 * Between 10 and

[ceph-users] How to add a slave zone to rgw

Hi, ceph users: I have a ceph cluster for rgw service in production, which was setup according to the simple configuration tutorial, with only one deafult region and one default zone. Even worse, I didn't enable neither the meta logging nor the data logging in the master zone. Now i want to add a

Re: [ceph-users] Ceph Performance Questions with rbd images access by qemu-kvm

Hello, On Tue, 1 Sep 2015 11:50:07 -0500 Kenneth Van Alstyne wrote: > Got it — I’ll keep that in mind. That may just be what I need to “get > by” for now. Ultimately, we’re looking to buy at least three nodes of > servers that can hold 40+ OSDs backed by 2TB+ SATA disks, > As mentioned, pick

Re: [ceph-users] How should I deal with placement group numbers when reducing number of OSDs

Unfortunately we are not in control of the VMs using this pool, so something like "sync -> stop VM -> incremental sync -> start VM on new pool" would be extremely complicated. I _think_ it's possible to misuse a cache tier to do this (add a cache tier, remove the underlying tier, add a new pool

Re: [ceph-users] Moving/Sharding RGW Bucket Index

I added sharding to our busiest RGW sites, but it will not shard existing bucket indexes, only applies to new buckets. Even with that change, I'm still considering moving the index pool to SSD. The main factor being the rate of writes. We are looking at a project that will have extremely high

Re: [ceph-users] How should I deal with placement group numbers when reducing number of OSDs

-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 I'm not convinced that a backing pool can be removed from a caching tier. I just haven't been able to get around to trying it. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Tue, Sep 1, 2015

Re: [ceph-users] Ceph Performance Questions with rbd images access by qemu-kvm

-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Just swapping out spindles for SSD will not give you orders of magnitude performance gains as it does in regular cases. This is because Ceph has a lot of overhead for each I/O which limits the performance of the SSDs. In my testing, two Intel S3500

[ceph-users] Accelio & Ceph

Hi cephers, I would like to know the status for production-ready of Accelio & Ceph, does anyone had a home-made procedure implemented with Ubuntu? recommendations, comments? Thanks in advance, Best regards, *German* ___ ceph-users mailing list

Re: [ceph-users] Accelio & Ceph

-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Accelio and Ceph are still in heavy development and not ready for production. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Tue, Sep 1, 2015 at 10:31 AM, German Anders wrote: Hi cephers,

Re: [ceph-users] ceph distributed osd

-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 You will be the one best equipped to answer the performance question. You will have to figure out what minimal performance your application will need. Then you have to match the disks to that (disk random IOPs * # disks) / replicas will get you in

[ceph-users] how to improve ceph cluster capacity usage

hi,all Recently, i did some experiments on OSD data distribution, we set up a cluster with 72 OSDs,all 2TB sata disk, and ceph version is v0.94.3 and linux kernel version is 3.18, and set "ceph osd crush tunables optimal". There are 3 pools: pool 0 'rbd' replicated size 2 min_size 1 crush_ruleset

Re: [ceph-users] Troubleshooting rgw bucket list

Can you bump up debug (debug rgw = 20, debug ms = 1), and see if the operations (bucket listing and bucket check) go into some kind of infinite loop? Yehuda On Tue, Sep 1, 2015 at 1:16 AM, Sam Wouters wrote: > Hi, I've started the bucket --check --fix on friday evening and it's

[ceph-users] Moving/Sharding RGW Bucket Index

Hello, I have two large buckets in my RGW and I think the performance is being impacted by the bucket index. One bucket contains 9 million objects and the other one has 22 million. I'd like to shard the bucket index and also change the ruleset of the .rgw.buckets.index pool to put it on

Re: [ceph-users] How to disable object-map and exclusive features ?

Hi Jason, I have a coredump with the size of 1200M compressed . Where shall i put the dump ? I think the crashes are often caused when I do a snapshot backup of the vm-images. Then somwething happens with locking which causes the cm to crash Thanks Christoph On Mon, Aug 31, 2015 at

Re: [ceph-users] Ceph Performance Questions with rbd images access by qemu-kvm

Thanks for the awesome advice folks. Until I can go larger scale (50+ SATA disks), I’m thinking my best option here is to just swap out these 1TB SATA disks with 1TB SSDs. Am I oversimplifying the short term solution? Thanks, -- Kenneth Van Alstyne Systems Architect Knight Point Systems, LLC

[ceph-users] Appending to an open file - O_APPEND flag

Hi! open( ... O_APPEND) works fine in a single system. If many processes write to the same file, their output will never overwrite each other. On NFS overwriting is possible, as appending is only emulated - each write is preceded by a seek to the current file size and race condition may occur.

Re: [ceph-users] Appending to an open file - O_APPEND flag

On Sep 1, 2015 4:41 PM, "Janusz Borkowski" wrote: > > Hi! > > open( ... O_APPEND) works fine in a single system. If many processes write to the same file, their output will never overwrite each other. > > On NFS overwriting is possible, as appending is only

Re: [ceph-users] Appending to an open file - O_APPEND flag

Hi guy's, I am totally new to Ceph deploy and I have succefully install ceph cluster on a Admin node and able to active it by One monitor and two OSD. After creation of Ceph Cluster I checked the Ceph Health Status and The output was OK. With that success I started to move to next stage for

Re: [ceph-users] SSD test results with Plextor M6 Pro, HyperX Fury, Kingston V300, ADATA SP90

Hi Jan, I am building two new clusters for testing. I been reading your messages on the mailing list for a while now and want to thank you for your support. I can redo all the numbers, but is your question to run all the test again with [hdparm -W 1 /dev/sdc]? Please tell me what else you would

Re: [ceph-users] How should I deal with placement group numbers when reducing number of OSDs

-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 We are in a situation where we need to decrease PG for a pool as well. One thought is to live migrate with block copy to a new pool with the right number of PGs and then once they are all moved delete the old pool. We don't have a lot of data in