Re: [ceph-users] ceph warning

2016-09-01 Thread Ishmael Tsoaela
Thanks, I'll keep that in mind. I appreciate the assitance. everything looks good this morning. cluster df3f96d8-3889-4baa-8b27-cc2839141425 health HEALTH_OK monmap e7: 3 mons at {Monitors} election epoch 118, quorum 0,1,2 nodeB,nodeC,nodeD osdmap e5246: 18 osds: 18 u

Re: [ceph-users] cephfs page cache

2016-09-01 Thread Yan, Zheng
I think about this again. This issue could be caused by stale session. Could you check kernel logs of your servers. Are there any ceph related kernel message (such as "ceph: mds0 caps stale") Regards Yan, Zheng On Thu, Sep 1, 2016 at 11:02 PM, Sean Redmond wrote: > Hi, > > It seems to be using

Re: [ceph-users] Slow Request on OSD

2016-09-01 Thread Dan Jakubiec
Thanks you for all the help Wido: > On Sep 1, 2016, at 14:03, Wido den Hollander wrote: > > You have to mark those OSDs as lost and also force create the incomplete PGs. > This might be the root of our problems. We didn't mark the parent OSD as "lost" before we removed it. Now ceph won't le

Re: [ceph-users] ceph warning

2016-09-01 Thread Christian Balzer
Hello, On Thu, 1 Sep 2016 16:24:28 +0200 Ishmael Tsoaela wrote: > I did set configure the following during my initial setup: > > osd pool default size = 3 > Ah yes, so not this. (though the default "rbd" pool that's initially created tended to ignore that parameter and would default to 3 in an

Re: [ceph-users] vmware + iscsi + tgt + reservations

2016-09-01 Thread Brad Hubbard
On Fri, Sep 2, 2016 at 7:41 AM, Oliver Dzombic wrote: > Hi, > > i know, this is not really ceph related anymore. But i guess it could be > helpful for others too. > > I was using: > > https://ceph.com/dev-notes/adding-support-for-rbd-to-stgt/ > > and i am currently running in a problem, where > >

[ceph-users] vmware + iscsi + tgt + reservations

2016-09-01 Thread Oliver Dzombic
Hi, i know, this is not really ceph related anymore. But i guess it could be helpful for others too. I was using: https://ceph.com/dev-notes/adding-support-for-rbd-to-stgt/ and i am currently running in a problem, where ONE LUN is connected to TWO Nodes ( esxi 6.0 ) And the 2nd node is unab

Re: [ceph-users] Strange copy errors in osd log

2016-09-01 Thread Samuel Just
If it's bluestore, this is pretty likely to be a bluestore bug. If you are interested in experimenting with bluestore, you probably want to watch developements on the master branch, it's undergoing a bunch of changes right now. -Sam On Thu, Sep 1, 2016 at 1:54 PM, Виталий Филиппов wrote: > Hi! I

Re: [ceph-users] Auto recovering after loosing all copies of a PG(s)

2016-09-01 Thread Wido den Hollander
> Op 1 september 2016 om 17:37 schreef Iain Buclaw : > > > On 16 August 2016 at 17:13, Wido den Hollander wrote: > > > >> Op 16 augustus 2016 om 15:59 schreef Iain Buclaw : > >> > >> > >> The desired behaviour for me would be for the client to get an instant > >> "not found" response from stat(

Re: [ceph-users] Slow Request on OSD

2016-09-01 Thread Wido den Hollander
> Op 1 september 2016 om 18:55 schreef Dan Jakubiec : > > > Thanks Wido. Reed and I have been working together to try to restore this > cluster for about 3 weeks now. I have been accumulating a number of failure > modes that I am hoping to share with the Ceph group soon, but have been > hol

[ceph-users] Strange copy errors in osd log

2016-09-01 Thread Виталий Филиппов
Hi! I'm playing with a test setup of ceph jewel with bluestore and cephfs over erasure-coded pool with replicated pool as a cache tier. After writing some number of small files to cephfs I begin seeing the following error messages during the migration of data from cache to EC pool: 2016-09-

Re: [ceph-users] Slow Request on OSD

2016-09-01 Thread Dan Jakubiec
Thanks Wido. Reed and I have been working together to try to restore this cluster for about 3 weeks now. I have been accumulating a number of failure modes that I am hoping to share with the Ceph group soon, but have been holding off a bit until we see the full picture clearly so that we can p

[ceph-users] CDM Reminder

2016-09-01 Thread Patrick McGarry
Hey cephers, Just a reminder that this month’s Ceph Developer Monthly meeting will be next Wed 07 Sep @ 9p EDT (it’s an APAC-friendly month). Please submit your blueprints to: http://wiki.ceph.com/CDM_07-SEP-2016 If you have any questions or concerns, please feel free to send them my way. Thanks

Re: [ceph-users] Turn snapshot of a flattened snapshot into regular image

2016-09-01 Thread Steve Taylor
Something isn't right. Ceph won't delete RBDs that have existing snapshots, even when those snapshots aren't protected. You can't delete a snapshot that's protected, and you can't unprotect a snapshot if there is a COW clone that depends on it. I'm not intimately familiar with OpenStack, but it

Re: [ceph-users] Auto recovering after loosing all copies of a PG(s)

2016-09-01 Thread Iain Buclaw
On 16 August 2016 at 17:13, Wido den Hollander wrote: > >> Op 16 augustus 2016 om 15:59 schreef Iain Buclaw : >> >> >> The desired behaviour for me would be for the client to get an instant >> "not found" response from stat() operations. For write() to recreate >> unfound objects. And for missin

Re: [ceph-users] Turn snapshot of a flattened snapshot into regular image

2016-09-01 Thread Eugen Block
Thanks for the quick response, but I don't believe I'm there yet ;-) cloned the glance image to a cinder device I have configured these three services (nova, glance, cinder) to use ceph as storage backend, but cinder is not involved in this process I'm referring to. Now I wanted to repro

Re: [ceph-users] cephfs page cache

2016-09-01 Thread Sean Redmond
Hi, It seems to be using syscall mmap() from what I read this indicates it is using memory-mapped IO. Please see a strace here: http://pastebin.com/6wjhSNrP Thanks On Wed, Aug 31, 2016 at 5:51 PM, Sean Redmond wrote: > I am not sure how to tell? > > Server1 and Server2 mount the ceph file sys

Re: [ceph-users] [Board] Ceph at OpenStack Barcelona

2016-09-01 Thread Dan Van Der Ster
Hi Patrick, > On 01 Sep 2016, at 16:29, Patrick McGarry wrote: > > Hey cephers, > > Now that our APAC roadshow has concluded I’m starting to look forward > to upcoming events like OpenStack Barcelona. There were a ton of talks > submitted this time around, so many of you did not get your talk >

[ceph-users] Ceph at OpenStack Barcelona

2016-09-01 Thread Patrick McGarry
Hey cephers, Now that our APAC roadshow has concluded I’m starting to look forward to upcoming events like OpenStack Barcelona. There were a ton of talks submitted this time around, so many of you did not get your talk accepted. You can see the 8 accepted talks here: https://www.openstack.org/sum

Re: [ceph-users] ceph warning

2016-09-01 Thread Ishmael Tsoaela
I did set configure the following during my initial setup: osd pool default size = 3 root@nodeC:/mnt/vmimages# ceph osd dump | grep "replicated size" pool 0 'rbd' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 217 flags hashpspool stripe_width

Re: [ceph-users] ceph warning

2016-09-01 Thread Christian Balzer
Hello, On Thu, 1 Sep 2016 14:00:53 +0200 Ishmael Tsoaela wrote: > more questions and I hope you don;t mind: > > > > My understanding is that if I have 3 hosts with 5 osd each, 1 host > goes down, Ceph should not replicate to the osd that are down. > How could it replicate to something that i

Re: [ceph-users] Turn snapshot of a flattened snapshot into regular image

2016-09-01 Thread Steve Taylor
You're already there. When you booted ONE you cloned the glance image to a cinder device (A', separate RBD) that was a COW clone of A. That's why you can't delete A until you flatten SNAP1. A' isn't a full copy until that flatten is complete, at which point you're able to delete A. SNAP2 is a s

[ceph-users] Turn snapshot of a flattened snapshot into regular image

2016-09-01 Thread Eugen Block
Hi all, I'm trying to understand the idea behind rbd images and their clones/snapshots. I have tried this scenario: 1. upload image A to glance 2. boot instance ONE from image A 3. make changes to instance ONE (install new package) 4. create snapshot SNAP1 from ONE 5. delete instance ONE 6. d

Re: [ceph-users] ceph warning

2016-09-01 Thread Ishmael Tsoaela
more questions and I hope you don;t mind: My understanding is that if I have 3 hosts with 5 osd each, 1 host goes down, Ceph should not replicate to the osd that are down. When the host comes up, only then the replication will commence right? If only 1 osd out of 5 comes up, then only data mea

Re: [ceph-users] ceph warning

2016-09-01 Thread Ishmael Tsoaela
Thank you again. I will add 3 more osd today and leave untouched, maybe over weekend. On Thu, Sep 1, 2016 at 1:16 PM, Christian Balzer wrote: > > Hello, > > On Thu, 1 Sep 2016 11:20:33 +0200 Ishmael Tsoaela wrote: > >> thanks for the response >> >> >> >> > You really will want to spend more time

Re: [ceph-users] Slow Request on OSD

2016-09-01 Thread Cloud List
On Thu, Sep 1, 2016 at 3:50 PM, Nick Fisk wrote: > > > Op 31 augustus 2016 om 23:21 schreef Reed Dier >: > > > > > > > > > Multiple XFS corruptions, multiple leveldb issues. Looked to be result > of write cache settings which have been adjusted now. > > Reed, I realise that you are probably very

Re: [ceph-users] ceph warning

2016-09-01 Thread Christian Balzer
Hello, On Thu, 1 Sep 2016 11:20:33 +0200 Ishmael Tsoaela wrote: > thanks for the response > > > > > You really will want to spend more time reading documentation and this ML, > > as well as using google to (re-)search things. > > > I did do some reading on the error but cannot understand w

Re: [ceph-users] ceph journal system vs filesystem journal system

2016-09-01 Thread huang jun
2016-09-01 17:25 GMT+08:00 한승진 : > Hi all. > > I'm very confused about ceph journal system > > Some people said ceph journal system works like linux journal filesystem. > > Also some people said all data are written journal first and then written to > OSD data. > > Journal of Ceph storage also writ

[ceph-users] ceph journal system vs filesystem journal system

2016-09-01 Thread 한승진
Hi all. I'm very confused about ceph journal system Some people said ceph journal system works like linux journal filesystem. Also some people said all data are written journal first and then written to OSD data. Journal of Ceph storage also write just metadata of object or write all data of ob

Re: [ceph-users] ceph warning

2016-09-01 Thread Ishmael Tsoaela
thanks for the response > You really will want to spend more time reading documentation and this ML, > as well as using google to (re-)search things. I did do some reading on the error but cannot understand why they do not clear even after so long. > In your previous mail you already mention

[ceph-users] RadosGW zonegroup id error

2016-09-01 Thread Yoann Moulin
Hello, I have an issue with the default zonegroup on my cluster (Jewel 10.2.2), I don't know when this occured, but I think I did a wrong command during the manipulation of zones and regions. Now the ID of my zonegroup is "default" instead of "4d982760-7853-4174-8c05-cec2ef148cf0", I cannot update

Re: [ceph-users] ceph warning

2016-09-01 Thread Christian Balzer
Hello, On Thu, 1 Sep 2016 10:18:39 +0200 Ishmael Tsoaela wrote: > Hi All, > > Can someone please decipher this errors for me, after all nodes rebooted in > my cluster on Monday. the warning has not gone. > You really will want to spend more time reading documentation and this ML, as well as usi

[ceph-users] ceph warning

2016-09-01 Thread Ishmael Tsoaela
Hi All, Can someone please decipher this errors for me, after all nodes rebooted in my cluster on Monday. the warning has not gone. Will the warning ever clear? cluster df3f96d8-3889-4baa-8b27-cc2839141425 health HEALTH_WARN 2 pgs backfill_toofull 532 pgs backfill

Re: [ceph-users] Slow Request on OSD

2016-09-01 Thread Nick Fisk
> -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Wido > den Hollander > Sent: 01 September 2016 08:19 > To: Reed Dier > Cc: ceph-users@lists.ceph.com > Subject: Re: [ceph-users] Slow Request on OSD > > > > Op 31 augustus 2016 om 23:21 schre

Re: [ceph-users] Slow Request on OSD

2016-09-01 Thread Wido den Hollander
> Op 31 augustus 2016 om 23:21 schreef Reed Dier : > > > Multiple XFS corruptions, multiple leveldb issues. Looked to be result of > write cache settings which have been adjusted now. > That is bad news, really bad. > You’ll see below that there are tons of PG’s in bad states, and it was slo