[ceph-users] radosgw_agent sync issues

HI i setup 2 clusters and in using radosgw_agent to sync them last week the sync stop working if on runinig the agent from command line i see its stuck on 2 files in the console im geting : 2016-03-17 21:11:57,391 14323 [radosgw_agent.worker][DEBUG ] op state is [] 2016-03-17 21:11:57,391 14323

Re: [ceph-users] [cephfs] About feature 'snapshot'

Hi John, How to set this feature on? Thank you 2016-03-17 21:41 GMT+08:00 Gregory Farnum : > On Thu, Mar 17, 2016 at 3:49 AM, John Spray wrote: > > Snapshots are disabled by default: > > >

Re: [ceph-users] ceph-disk from jewel has issues on redhat 7

Hi, It's true, partprobe works intermittently. I extracted the key commands to show the problem: [18:44]# /usr/sbin/sgdisk --new=2:0:20480M --change-name=2:'ceph journal' --partition-guid=2:aa23e07d-e6b3-4261-a236-c0565971d88d --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt --

Re: [ceph-users] SSDs for journals vs SSDs for a cache tier, which is better?

Thanks all for your suggestions and advice. I'll let you know how it goes :) Stephen On 2016-03-16 16:58, Heath Albritton wrote: The rule of thumb is to match the journal throughput to the OSD throughout. I'm seeing ~180MB/s sequential write on my OSDs and I'm using one of the P3700 400GB

[ceph-users] RBD/Ceph as Physical boot volume

I posted about this a while ago, and someone else has since inquired, but I am seriously wanting to know if anybody has figured out how to boot from a RBD device yet using ipxe or similar. Last I read. loading the kernel and initrd from object storage would be theoretically easy, and would only

Re: [ceph-users] Does object map feature lock snapshots ?

Hi, I had no special logging activated. Today I re-enabled exclusive-lock object-map and fast-diff on an image in 9.2.1 As soon as I ran an rbd export-diff I had lots of these error messages on the console of the rbd export process: 2016-03-18 11:18:21.546658 7f77245d1700 1 heartbeat_map

Re: [ceph-users] ceph-disk from jewel has issues on redhat 7

Hi, Is there a tracker for this? We just hit the same problem on 10.0.5. Cheers, Dan # rpm -q ceph ceph-10.0.5-0.el7.x86_64 # cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) # ceph-disk -v prepare /dev/sdc DEBUG:ceph-disk:get_dm_uuid /dev/sdc uuid path is

Re: [ceph-users] inconsistent PG -> unfound objects on an erasure coded system

Basically, the lookup process is: try DIR_9/DIR_5/DIR_4/DIR_D/DIR_E/DIR_C/DIR_9/DIR_7...doesn't exist try DIR_9/DIR_5/DIR_4/DIR_D/DIR_E/DIR_C/DIR_9/...doesn't exist try DIR_9/DIR_5/DIR_4/DIR_D/DIR_E/DIR_C/...doesn't exist try DIR_9/DIR_5/DIR_4/DIR_D/DIR_E/...does exist, object must be here If

[ceph-users] Single key delete performance against increasing bucket size

On Wed, Mar 16, 2016 at 06:36:33AM +, Pavan Rallabhandi wrote: > I find this to be discussed here before, but couldn¹t find any solution > hence the mail. In RGW, for a bucket holding objects in the range of ~ > millions, one can find it to take for ever to delete the bucket(via >

Re: [ceph-users] v10.0.4 released

Hi, Because of a tiny mistake preventing deb packages to be built, v10.0.5 was released shortly after v10.0.4 and is now the current development release. The Stable release team[0] collectively decided to help by publishing development packages[1], starting with v10.0.5. The packages for

[ceph-users] RBD hanging on some volumes of a pool

Hi, I am facing issues with some of my rbd volumes since yesterday. Some of them completely hang at some point before eventually resuming IO, may it be a few minutes or several hours later. First and foremost, my setup : I already detailed it on the mailing list [0][1]. Some changes have been

[ceph-users] [cephfs] About feature 'snapshot'

Hi all, I encounter a trouble about cephfs sanpshot. It seems that the folder '.snap' is exist. But I use 'll -a' can't let it show up. And I enter that folder and create folder in it, it showed something wrong to use snapshot. Please check : http://imgur.com/elZhQvD

Re: [ceph-users] data corruption with hammer

Hi,All. I confirm the problem. When min_read_recency_for_promote> 1 data failure. С уважением, Фасихов Ирек Нургаязович Моб.: +79229045757 2016-03-17 15:26 GMT+03:00 Sage Weil : > On Thu, 17 Mar 2016, Nick Fisk wrote: > > There is got to be something else going on here. All

Re: [ceph-users] data corruption with hammer

Also, is this ceph_test_rados rewriting objects quickly? I think that the issue is with rewriting objects so if we can tailor the ceph_test_rados to do that, it might be easier to reproduce. Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On

Re: [ceph-users] ZFS or BTRFS for performance?

Neither of these file systems is recommended for production use underlying an OSD. The general direction for ceph is to move away from having a file system at all. That effort is called "bluestore" and is supposed to show up in the jewel release. -H > On Mar 18, 2016, at 11:15, Schlacta,

Re: [ceph-users] SSDs for journals vs SSDs for a cache tier, which is better?

The rule of thumb is to match the journal throughput to the OSD throughout. I'm seeing ~180MB/s sequential write on my OSDs and I'm using one of the P3700 400GB units per six OSDs. The 400GB P3700 yields around 1200MB/s* and has around 1/10th the latency of any SATA SSD I've tested. I put a

Re: [ceph-users] inconsistent PG -> unfound objects on an erasure coded system

Great, I just recovered the first placement group from this error. To be sure, I ran a deep-scrub and that comes back clean. Thanks for all your help. Regards, Jeff On Thu, Mar 17, 2016 at 11:58 AM, Samuel Just wrote: > Oh, it's getting a stat mismatch. I think what

Re: [ceph-users] [cephfs] About feature 'snapshot'

Snapshots are disabled by default: http://docs.ceph.com/docs/hammer/cephfs/early-adopters/#most-stable-configuration John On Thu, Mar 17, 2016 at 10:02 AM, 施柏安 wrote: > Hi all, > I encounter a trouble about cephfs sanpshot. It seems that the folder > '.snap' is exist.

Re: [ceph-users] inconsistent PG -> unfound objects on an erasure coded system

Yep, thanks for all the help tracking down the root cause! -Sam On Thu, Mar 17, 2016 at 10:50 AM, Jeffrey McDonald wrote: > Great, I just recovered the first placement group from this error. To be > sure, I ran a deep-scrub and that comes back clean. > > Thanks for all your

Re: [ceph-users] cephfs infernalis (ceph version 9.2.1) - bonnie++

Hi, on ubuntu 14.04 client and centos 7.2 client with centos 7 Hammer its working without problems. -- Mit freundlichen Gruessen / Best regards Oliver Dzombic IP-Interactive mailto:i...@ip-interactive.de Anschrift: IP Interactive UG ( haftungsbeschraenkt ) Zum Sonnenberg 1-3 63571

Re: [ceph-users] data corruption with hammer

Hi, Nick I switched between forward and writeback. (forward -> writeback) С уважением, Фасихов Ирек Нургаязович Моб.: +79229045757 2016-03-17 16:10 GMT+03:00 Nick Fisk : > > -Original Message- > > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of

Re: [ceph-users] inconsistent PG -> unfound objects on an erasure coded system

Hi Sam, In the 70.459 logs from the deep-scrub, there is an error: $ zgrep "= \-2$" ceph-osd.307.log.1.gz 2016-03-07 16:11:41.828332 7ff30cdad700 10 filestore(/var/lib/ceph/osd/ceph-307) remove

Re: [ceph-users] data corruption with hammer

I'll miss the Ceph community as well. There was a few things I really wanted to work in with Ceph. I got this: update_object_version oid 13 v 1166 (ObjNum 1028 snap 0 seq_num 1028) dirty exists 1038: left oid 13 (ObjNum 1028 snap 0 seq_num 1028) 1040: finishing write tid 1 to nodez23350-256

Re: [ceph-users] data corruption with hammer

Cherry-picking that commit onto v0.94.6 wasn't clean so I'm just building your branch. I'm not sure what the difference between your branch and 0.94.6 is, I don't see any commits against osd/ReplicatedPG.cc in the last 5 months other than the one you did today. Robert LeBlanc PGP

Re: [ceph-users] RBD/Ceph as Physical boot volume

On 03/17/2016 03:51 AM, Schlacta, Christ wrote: I posted about this a while ago, and someone else has since inquired, but I am seriously wanting to know if anybody has figured out how to boot from a RBD device yet using ipxe or similar. Last I read. loading the kernel and initrd from object

Re: [ceph-users] v0.94.6 Hammer released

Hi Stable Release Team for v0.94, On Thu, Mar 10, 2016 at 11:00:06AM +1100, Chris Dunlop wrote: > On Wed, Mar 02, 2016 at 06:32:18PM +0700, Loic Dachary wrote: >> I think you misread what Sage wrote : "The intention was to >> continue building stable releases (0.94.x) on the old list of >>

[ceph-users] ceph-deploy rgw

For clusters that were created pre-hammer and want to use ceph-deploy to create additional rgw instances is there a way to create the bootstrap-rgw keyring? http://docs.ceph.com/ceph-deploy/docs/rgw.html -- Derek T. Yarnell University of Maryland Institute for Advanced Computer Studies

Re: [ceph-users] data corruption with hammer

> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Irek Fasikhov > Sent: 17 March 2016 13:00 > To: Sage Weil > Cc: Robert LeBlanc ; ceph-users us...@lists.ceph.com>; Nick Fisk

Re: [ceph-users] data corruption with hammer

just got done with a test against a build of 0.94.6 minus the two commits that were backported in PR 7207. everything worked as it should with the cache-mode set to writeback and the min_read_recency_for_promote set to 2. assuming it works properly on master, there must be a commit that we're

Re: [ceph-users] SSDs for journals vs SSDs for a cache tier, which is better?

On 2016-02-17 11:07, Christian Balzer wrote: On Wed, 17 Feb 2016 10:04:11 +0100 Piotr Wachowicz wrote: > > Let's consider both cases: > > Journals on SSDs - for writes, the write operation returns right > > after data lands on the Journal's SSDs, but before it's written to > > the backing

[ceph-users] Radosgw (civetweb) hangs once around 850 established connections

I have a cluster of around 630 OSDs with 3 dedicated monitors and 2 dedicated gateways. The entire cluster is running hammer (0.94.5 (9764da52395923e0b32908d83a9f7304401fee43)). (Both of my gateways have stopped responding to curl right now. root@host:~# timeout 5 curl localhost ; echo $? 124

Re: [ceph-users] RGW quota

On Wednesday, 16 March 2016, Derek Yarnell wrote: > Hi, > > We have a user with a 50GB quota and has now a single bucket with 20GB > of files. They had previous buckets created and removed but the quota > has not decreased. I understand that we do garbage collection but

Re: [ceph-users] data corruption with hammer

-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Possible, it looks like all the messages comes from a test suite. Is there some logging that would expose this or an assert that could be added? We are about ready to do some testing in our lab to see if we can replicate it and workaround the issue.

Re: [ceph-users] data corruption with hammer

We are trying to figure out how to use rados bench to reproduce. Ceph itself doesn't seem to think there is any corruption, but when you do a verify inside the RBD, there is. Can rados bench verify the objects after they are written? It also seems to be primarily the filesystem metadata that is

Re: [ceph-users] RBD hanging on some volumes of a pool

> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Adrien Gillard > Sent: 17 March 2016 10:23 > To: ceph-users > Subject: [ceph-users] RBD hanging on some volumes of a pool > > Hi, > > I am facing issues with some of my

Re: [ceph-users] RBD hanging on some volumes of a pool

Hi Nick, Thank you for your feedback. The cache tiers was fine. We identified some packet loss between two switches. As usual with network, relatively easy to identify but not something that comes to mind at first :) Adrien On Thu, Mar 17, 2016 at 2:32 PM, Nick Fisk wrote: >

Re: [ceph-users] Local SSD cache for ceph on each compute node.

Hi Nick, Your solution requires manual configuration for each VM and cannot be setup as part of an automated OpenStack deployment. It would be really nice if it was a hypervisor based setting as opposed to a VM based setting. Thanks Daniel -Original Message- From: Nick Fisk

Re: [ceph-users] v10.0.4 released

Thank you for doing this. It will make testing 10.0.x easier for all of us in the field, and will make it easier to report bugs, as we will know that the problems we find were not caused by our build process. Eric On Wed, Mar 16, 2016 at 7:14 AM, Loic Dachary wrote: > Hi, >

Re: [ceph-users] data corruption with hammer

On Thu, 17 Mar 2016, Nick Fisk wrote: > There is got to be something else going on here. All that PR does is to > potentially delay the promotion to hit_set_period*recency instead of > just doing it on the 2nd read regardless, it's got to be uncovering > another bug. > > Do you see the same

Re: [ceph-users] SSDs for journals vs SSDs for a cache tier, which is better?

> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Stephen Harker > Sent: 16 March 2016 16:22 > To: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] SSDs for journals vs SSDs for a cache tier, which is > better? > > On 2016-02-17 11:07,

Re: [ceph-users] ssd only storage and ceph

> On 17 Mar 2016, at 17:28, Erik Schwalbe wrote: > > Hi, > > at the moment I do some tests with SSD's and ceph. > My Question is, how to mount an SSD OSD? With or without discard option? I recommend running without discard but running "fstrim" command every now and

Re: [ceph-users] v0.94.6 Hammer released

Hi Chen, On Thu, Mar 17, 2016 at 12:40:28AM +, Chen, Xiaoxi wrote: > It’s already there, in > http://download.ceph.com/debian-hammer/pool/main/c/ceph/. I can only see ceph*_0.94.6-1~bpo80+1_amd64.deb there. Debian wheezy would be bpo70. Cheers, Chris > On 3/17/16, 7:20 AM, "Chris Dunlop"

Re: [ceph-users] inconsistent PG -> unfound objects on an erasure coded system

OK, I think I have it now. I do have one more question, in this case, the hash indicates the directory structure but how do I know from the hash how many levels I should go down.If the hash is a 32-bit hex integer, *how do I know how many should be included as part of the hash for the

[ceph-users] Cannot remove rbd locks

Hi, some of my rbds show they have an exclusive lock. I think the lock can be stale or weeks old. We have also once added feature exclusive lock and later removed that feature I can see the lock: root@machine:~# rbd lock list vm-114-disk-1 There is 1 exclusive lock on this image. Locker

Re: [ceph-users] Radosgw (civetweb) hangs once around 850 established connections

What OS are you using? I have a lot more open connections than that. (though i have some other issues, where rgw sometimes returns 500 errors, it doesn't stop like yours) You might try tuning civetweb's num_threads and 'rgw num rados handles': rgw frontends = civetweb num_threads=125

Re: [ceph-users] inconsistent PG -> unfound objects on an erasure coded system

Hi Sam, I've written a script but i'm a little leary of unleasing it until I find a few more cases to test. The script successfully removed the file mentioned above. I took the next pg which was marked inconsistent and ran the following command over those pg directory structures: find . -name

Re: [ceph-users] RGW quota

On 3/17/16 1:41 PM, Marius Vaitiekunas wrote: > It's possible that somebody changed the owner of some bucket. But all > objects in that bucket still belongs to this user. That way you can get > quota exceeded. We had the same situation. Well the user says he didn't write to any other buckets than

[ceph-users] Infernalis: chown ceph:ceph at runtime ?

Hi, we have upgraded our ceph-cluster to infernalis from hammer. Ceph is still running as root and we are using the "setuser match path = /var/lib/ceph/$type/$cluster-$id" directive in ceph.conf Now we would like to change the ownership of data-files and devices to ceph at runtime. What ist

Re: [ceph-users] ceph-disk from jewel has issues on redhat 7

Thanks Dan, I have raised the tracker for this issue http://tracker.ceph.com/issues/15176 On Thu, Mar 17, 2016 at 10:47 AM, Dan van der Ster wrote: > Hi, > > It's true, partprobe works intermittently. I extracted the key > commands to show the problem: > > [18:44]#

Re: [ceph-users] ZFS or BTRFS for performance?

FWIW, from purely a performance perspective Ceph usually looks pretty fantastic on a fresh BTRFS filesystem. In fact it will probably continue to look great until you do small random writes to large objects (like say to blocks in an RBD volume). Then COW starts fragmenting the objects into

Re: [ceph-users] rgw bucket deletion woes

On Tue, Mar 15, 2016 at 11:36 PM, Pavan Rallabhandi wrote: > Hi, > > I find this to be discussed here before, but couldn¹t find any solution > hence the mail. In RGW, for a bucket holding objects in the range of ~ > millions, one can find it to take for ever to

Re: [ceph-users] Local SSD cache for ceph on each compute node.

> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Daniel Niasoff > Sent: 16 March 2016 21:02 > To: Nick Fisk ; 'Van Leeuwen, Robert' > ; 'Jason Dillaman' > Cc:

Re: [ceph-users] ZFS or BTRFS for performance?

Le 19/03/2016 18:38, Heath Albritton a écrit : > If you google "ceph bluestore" you'll be able to find a couple slide > decks on the topic. One of them by Sage is easy to follow without the > benefit of the presentation. There's also the " Redhat Ceph Storage > Roadmap 2016" deck. > > In any

Re: [ceph-users] ZFS or BTRFS for performance?

On 20/03/2016 3:38 AM, Heath Albritton wrote: Ceph protects against bitrot at a much higher level by validating the checksum of the entire placement group during a deep scrub. Ceph has checksums? I didn't think it did. Its my understanding that it just compares blocks between replications

[ceph-users] CfP 11th Workshop on Virtualization in High-Performance Cloud Computing (VHPC '16)

CfP 11th Workshop on Virtualization in High-Performance Cloud Computing (VHPC '16) CALL FOR PAPERS 11th Workshop on Virtualization in High­-Performance Cloud Computing (VHPC '16) held in conjunction with the International

Re: [ceph-users] ZFS or BTRFS for performance?

On Mar 18, 2016 4:31 PM, "Lionel Bouton" > > Will bluestore provide the same protection against bitrot than BTRFS? > Ie: with BTRFS the deep-scrubs detect inconsistencies *and* the OSD(s) > with invalid data get IO errors when trying to read corrupted data and >

Re: [ceph-users] [cephfs] About feature 'snapshot'

On Fri, Mar 18, 2016 at 1:33 AM, 施柏安 wrote: > Hi John, > How to set this feature on? ceph mds set allow_new_snaps true --yes-i-really-mean-it John > Thank you > > 2016-03-17 21:41 GMT+08:00 Gregory Farnum : >> >> On Thu, Mar 17, 2016 at 3:49 AM,

Re: [ceph-users] [cephfs] About feature 'snapshot'

Hi John, Really thank you for your help, and sorry about that I ask such a stupid question of setting... So isn't this feature ready in Jewel? I found something info says that the features(snapshot, quota...) become stable in Jewel Thank you 2016-03-18 21:07 GMT+09:00 John Spray

Re: [ceph-users] ZFS or BTRFS for performance?

Hello, On Sun, 20 Mar 2016 00:45:47 +0100 Lionel Bouton wrote: > Le 19/03/2016 18:38, Heath Albritton a écrit : > > If you google "ceph bluestore" you'll be able to find a couple slide > > decks on the topic. One of them by Sage is easy to follow without the > > benefit of the presentation.

Re: [ceph-users] rgw bucket deletion woes

We would be a big user of this. We delete large buckets often and it takes forever. Though didn't I read that 'object expiration' support is on the near-term RGW roadmap? That may do what we want.. we're creating thousands of objects a day, and thousands of objects a day will be expiring, so RGW

Re: [ceph-users] data corruption with hammer

On Thu, 17 Mar 2016, Robert LeBlanc wrote: > We are trying to figure out how to use rados bench to reproduce. Ceph > itself doesn't seem to think there is any corruption, but when you do a > verify inside the RBD, there is. Can rados bench verify the objects after > they are written? It also seems

Re: [ceph-users] inconsistent PG -> unfound objects on an erasure coded system

Ok, like I said, most files with _long at the end are *not orphaned*. The generation number also is *not* an indication of whether the file is orphaned -- some of the orphaned files will have as the generation number and others won't. For each long filename object in a pg you

Re: [ceph-users] data corruption with hammer

Yep, let me pull and build that branch. I tried installing the dbg packages and running it in gdb, but it didn't load the symbols. Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Thu, Mar 17, 2016 at 11:36 AM, Sage Weil

Re: [ceph-users] inconsistent PG -> unfound objects on an erasure coded system

There is a directory structure hash, it's just that it's at the end of the name and you'll have to check the xattr I mentioned to find it. I think that file is actually the one we are talking about removing.

Re: [ceph-users] ZFS or BTRFS for performance?

Yes, I`m missing protection from Ceph too. http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-February/007680.html ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cannot remove rbd locks

Try the following: # rbd lock remove vm-114-disk-1 "auto 140454012457856" client.71260575 -- Jason Dillaman - Original Message - > From: "Christoph Adomeit" > To: ceph-us...@ceph.com > Sent: Friday, March 18, 2016 11:14:00 AM > Subject: [ceph-users]

Re: [ceph-users] ZFS or BTRFS for performance?

If you google "ceph bluestore" you'll be able to find a couple slide decks on the topic. One of them by Sage is easy to follow without the benefit of the presentation. There's also the " Redhat Ceph Storage Roadmap 2016" deck. In any case, bluestore is not intended to address bitrot. Given

[ceph-users] Upgrade from .94 to 10.0.5

Is there documentation on all the steps showing how to upgrade from .94 to 10.0.5? Thanks Rick ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] data corruption with hammer

There is got to be something else going on here. All that PR does is to potentially delay the promotion to hit_set_period*recency instead of just doing it on the 2nd read regardless, it's got to be uncovering another bug. Do you see the same problem if the cache is in writeback mode before you

Re: [ceph-users] inconsistent PG -> unfound objects on an erasure coded system

Oh, it's getting a stat mismatch. I think what happened is that on one of the earlier repairs it reset the stats to the wrong value (the orphan was causing the primary to scan two objects twice, which matches the stat mismatch I see here). A pg repair repair will clear that up. -Sam On Thu, Mar

[ceph-users] reallocate when OSD down

Hello, I have a problem with the following crushmap : # begin crush map tunable choose_local_tries 0 tunable choose_local_fallback_tries 0 tunable choose_total_tries 50 tunable chooseleaf_descend_once 1 tunable straw_calc_version 1 # devices device 0 osd.0 device 1 osd.1 device 2 osd.2 device 3

Re: [ceph-users] [cephfs] About feature 'snapshot'

On Thu, Mar 17, 2016 at 3:49 AM, John Spray wrote: > Snapshots are disabled by default: > http://docs.ceph.com/docs/hammer/cephfs/early-adopters/#most-stable-configuration Which makes me wonder if we ought to be hiding the .snaps directory entirely in that case. I haven't