Re: Ceph version 0.56.1, data loss on power failure

2013-01-16 Thread Dino Yancey
Hi Marcin, Not sure if anyone asked, but are your OSD journals on actual disk or are you using tmpfs? Dino On Wed, Jan 16, 2013 at 4:53 AM, Wido den Hollander w...@widodh.nl wrote: On 01/16/2013 11:50 AM, Marcin Szukala wrote: Hi all, Any ideas how can I resolve my issue? Or where the

Re: Ceph version 0.56.1, data loss on power failure

2013-01-16 Thread Yann Dupont
Le 16/01/2013 11:53, Wido den Hollander a écrit : On 01/16/2013 11:50 AM, Marcin Szukala wrote: Hi all, Any ideas how can I resolve my issue? Or where the problem is? Let me describe the issue. Host boots up and maps RBD image with XFS filesystems Host mounts the filesystems from the RBD

Re: Ceph version 0.56.1, data loss on power failure

2013-01-16 Thread Marcin Szukala
Hi Dino, journals are on dedicated SSD Regard, Marcin 2013/1/16 Dino Yancey dino2...@gmail.com: Hi Marcin, Not sure if anyone asked, but are your OSD journals on actual disk or are you using tmpfs? Dino On Wed, Jan 16, 2013 at 4:53 AM, Wido den Hollander w...@widodh.nl wrote: On

Re: Ceph version 0.56.1, data loss on power failure

2013-01-16 Thread Marcin Szukala
2013/1/16 Yann Dupont yann.dup...@univ-nantes.fr: Le 16/01/2013 11:53, Wido den Hollander a écrit : On 01/16/2013 11:50 AM, Marcin Szukala wrote: Hi all, Any ideas how can I resolve my issue? Or where the problem is? Let me describe the issue. Host boots up and maps RBD image with XFS

Re: REMINDER: all argonaut users should upgrade to v0.48.3argonaut

2013-01-16 Thread Sébastien Han
Can we use this doc as a reference for the upgrade? https://github.com/ceph/ceph/blob/eb02eaede53c03579d015ca00a888a48dbab739a/doc/install/upgrading-ceph.rst Thanks. -- Regards, Sébastien Han. On Tue, Jan 15, 2013 at 10:49 PM, Sage Weil s...@inktank.com wrote: That there are some critical

Re: [PATCH] libceph: for chooseleaf rules, retry CRUSH map descent from root if leaf is failed

2013-01-16 Thread Jim Schutt
Hi Sage, On 01/15/2013 07:55 PM, Sage Weil wrote: Hi Jim- I just realized this didn't make it into our tree. It's now in testing, and will get merged in the next window. D'oh! That's great news - thanks for the update. -- Jim sage -- To unsubscribe from this list: send the line

OSD don't start after upgrade form 0.47.2 to 0.56.1

2013-01-16 Thread Michael Menge
Hi List, i tried to upgrade my ceph cluster from 0.47.2 (openSuSE buildservice for SLES 11 SP2) to 0.56.1 (ceph.com/rpm/sles11/) At first I updated only one server (mon.b / osd.2) and restartet ceph on this server. After a short time /etc/init.d/ceph -a status showed not running for

8 out of 12 OSDs died after expansion on 0.56.1 (void OSD::do_waiters())

2013-01-16 Thread Wido den Hollander
Hi, I'm testing a small Ceph cluster with Asus C60M1-1 mainboards. The setup is: - AMD Fusion C60 CPU - 8GB DDR3 - 1x Intel 520 120GB SSD (OS + Journaling) - 4x 1TB disk I had two of these systems running, but yesterday I wanted to add a third one. So I had 8 OSDs (one per disk) running on

Re: REMINDER: all argonaut users should upgrade to v0.48.3argonaut

2013-01-16 Thread Sage Weil
On Wed, 16 Jan 2013, S?bastien Han wrote: Can we use this doc as a reference for the upgrade? https://github.com/ceph/ceph/blob/eb02eaede53c03579d015ca00a888a48dbab739a/doc/install/upgrading-ceph.rst Yeah. It's pretty simple in this case (since it's a point release upgrade): - install new

Re: Ceph version 0.56.1, data loss on power failure

2013-01-16 Thread Jeff Mitchell
FWIW, my ceph data dirs (for e.g. mons) are all on XFS. I've experienced a lot of corruption on these on power loss to the node -- and in some cases even when power wasn't lost, and the box was simply rebooted. This is on Ubuntu 12.04 with the ceph-provied 3.6.3 kernel (as I'm using RBD on these).

Re: Ceph slow request unstable issue

2013-01-16 Thread Sage Weil
Hi, On Wed, 16 Jan 2013, Andrey Korolyov wrote: On Wed, Jan 16, 2013 at 4:58 AM, Chen, Xiaoxi xiaoxi.c...@intel.com wrote: Hi list, We are suffering from OSD or OS down when there is continuing high pressure on the Ceph rack. Basically we are on Ubuntu 12.04+ Ceph

Re: Ceph version 0.56.1, data loss on power failure

2013-01-16 Thread Sage Weil
On Wed, 16 Jan 2013, Wido den Hollander wrote: On 01/16/2013 11:50 AM, Marcin Szukala wrote: Hi all, Any ideas how can I resolve my issue? Or where the problem is? Let me describe the issue. Host boots up and maps RBD image with XFS filesystems Host mounts the filesystems from

Re: REMINDER: all argonaut users should upgrade to v0.48.3argonaut

2013-01-16 Thread Sébastien Han
Thanks Sage! -- Regards, Sébastien Han. On Wed, Jan 16, 2013 at 5:39 PM, Sage Weil s...@inktank.com wrote: On Wed, 16 Jan 2013, S?bastien Han wrote: Can we use this doc as a reference for the upgrade?

Re: Ceph version 0.56.1, data loss on power failure

2013-01-16 Thread Wido den Hollander
Op 16 jan. 2013 om 18:00 heeft Sage Weil s...@inktank.com het volgende geschreven: On Wed, 16 Jan 2013, Wido den Hollander wrote: On 01/16/2013 11:50 AM, Marcin Szukala wrote: Hi all, Any ideas how can I resolve my issue? Or where the problem is? Let me describe the issue. Host

Re: flashcache

2013-01-16 Thread Mark Nelson
On 01/16/2013 03:46 PM, Sage Weil wrote: On Wed, 16 Jan 2013, Gandalf Corvotempesta wrote: 2013/1/16 Sage Weil s...@inktank.com: This sort of configuration effectively bundles the disk and SSD into a single unit, where the failure of either results in the loss of both. From Ceph's

Re: Ceph slow request unstable issue

2013-01-16 Thread Andrey Korolyov
On Wed, Jan 16, 2013 at 10:35 PM, Andrey Korolyov and...@xdel.ru wrote: On Wed, Jan 16, 2013 at 8:58 PM, Sage Weil s...@inktank.com wrote: Hi, On Wed, 16 Jan 2013, Andrey Korolyov wrote: On Wed, Jan 16, 2013 at 4:58 AM, Chen, Xiaoxi xiaoxi.c...@intel.com wrote: Hi list, We are

Re: [PATCH REPOST 0/2] libceph: embed r_trail struct in ceph_osd_request()

2013-01-16 Thread Josh Durgin
On 01/03/2013 03:34 PM, Alex Elder wrote: This series simplifies some handling of osd client message handling by using an initialized ceph_pagelist structure to refer to the trail portion of a ceph_osd_request rather than using a null pointer to represent not there.

Re: [PATCH REPOST] rbd: separate layout init

2013-01-16 Thread Josh Durgin
On 01/03/2013 02:55 PM, Alex Elder wrote: Pull a block of code that initializes the layout structure in an osd request into its own function so it can be reused. Signed-off-by: Alex Elder el...@inktank.com --- Reviewed-by: Josh Durgin josh.dur...@inktank.com drivers/block/rbd.c | 23

Re: [PATCH REPOST 0/6] libceph: parameter cleanup

2013-01-16 Thread Josh Durgin
On 01/04/2013 06:31 AM, Alex Elder wrote: This series mostly cleans up parameters used by functions in libceph, in the osd client code. -Alex [PATCH REPOST 1/6] libceph: pass length to ceph_osdc_build_request() [PATCH REPOST 2/6] libceph: pass length to

Re: mds: first stab at lookup-by-ino problem/soln description

2013-01-16 Thread Yan, Zheng
On Thu, Jan 17, 2013 at 5:52 AM, Gregory Farnum g...@inktank.com wrote: My biggest concern with this was how it worked on cluster with multiple data pools, and Sage's initial response was to either 1) create an object for each inode that lives in the metadata pool, and holds the backtraces

Re: mds: first stab at lookup-by-ino problem/soln description

2013-01-16 Thread Gregory Farnum
On Wed, Jan 16, 2013 at 3:54 PM, Sam Lang sam.l...@inktank.com wrote: On Wed, Jan 16, 2013 at 3:52 PM, Gregory Farnum g...@inktank.com wrote: My biggest concern with this was how it worked on cluster with multiple data pools, and Sage's initial response was to either 1) create an object for

Re: [PATCH 00/29] Various fixes for MDS

2013-01-16 Thread Sage Weil
Hi Yan, I reviewed these on the plane last night and they look good. There was one small cleanup I pushed on top of wip-mds (in ceph.git). I'll run this through our (still limited) fs suite and then merge into master. Thanks! sage On Fri, 4 Jan 2013, Yan, Zheng wrote: From: Yan, Zheng

RE: Ceph slow request unstable issue

2013-01-16 Thread Sage Weil
On Thu, 17 Jan 2013, Chen, Xiaoxi wrote: Hi Sage? Both CPU and Memory utilization are very low. CPU is ~ 20% (with 60% IOWAIT), Memory is far more less . I have 32 Core Sandybridege CPU(64 Core for HT), together with 128GB RAM per node. Hmm! -Original Message- From: Sage

Re: [PATCH REPOST 0/4] rbd: explicitly support only one osd op

2013-01-16 Thread Josh Durgin
On 01/04/2013 06:43 AM, Alex Elder wrote: An osd request can be made up of multiple ops, all of which are completed (or not) transactionally. There is partial support for multiple ops in an rbd request in the rbd code, but it's incomplete and not even supported by the osd client or the

Re: [PATCH REPOST 0/3] rbd: no need for file mapping calculation

2013-01-16 Thread Josh Durgin
On 01/04/2013 06:51 AM, Alex Elder wrote: Currently every osd request submitted by the rbd code undergoes a file mapping operation, which is common with what the ceph file system uses. But some analysis shows that there is no need to do this for rbd, because it already takes care of its own

Re: [PATCH REPOST] rbd: kill ceph_osd_req_op-flags

2013-01-16 Thread Josh Durgin
On 01/04/2013 06:46 AM, Alex Elder wrote: The flags field of struct ceph_osd_req_op is never used, so just get rid of it. Signed-off-by: Alex Elder el...@inktank.com --- Reviewed-by: Josh Durgin josh.dur...@inktank.com include/linux/ceph/osd_client.h |1 - 1 file changed, 1

Re: [PATCH REPOST] rbd: use a common layout for each device

2013-01-16 Thread Josh Durgin
On 01/04/2013 06:54 AM, Alex Elder wrote: Each osd message includes a layout structure, and for rbd it is always the same (at least for osd's in a given pool). Initialize a layout structure when an rbd_dev gets created and just copy that into osd requests for the rbd image. Replace an

Re: [PATCH REPOST] rbd: combine rbd sync watch/unwatch functions

2013-01-16 Thread Josh Durgin
On 01/04/2013 06:55 AM, Alex Elder wrote: The rbd_req_sync_watch() and rbd_req_sync_unwatch() functions are nearly identical. Combine them into a single function with a flag indicating whether a watch is to be initiated or torn down. Signed-off-by: Alex Elder el...@inktank.com ---

Re: [PATCH REPOST 6/6] rbd: move remaining osd op setup into rbd_osd_req_op_create()

2013-01-16 Thread Josh Durgin
On 01/04/2013 07:07 AM, Alex Elder wrote: The two remaining osd ops used by rbd are CEPH_OSD_OP_WATCH and CEPH_OSD_OP_NOTIFY_ACK. Move the setup of those operations into rbd_osd_req_op_create(), and get rid of rbd_create_rw_op() and rbd_destroy_op(). Signed-off-by: Alex Elder el...@inktank.com

Re: [PATCH REPOST 0/6] rbd: consolidate osd request setup

2013-01-16 Thread Josh Durgin
On 01/04/2013 07:03 AM, Alex Elder wrote: This series consolidates and encapsulates the setup of all osd requests into a single function which takes variable arguments appropriate for the type of request. The result groups together common code idioms and I think makes the spots that build these

Re: [PATCH REPOST] rbd: assign watch request more directly

2013-01-16 Thread Josh Durgin
On 01/04/2013 07:07 AM, Alex Elder wrote: Both rbd_req_sync_op() and rbd_do_request() have a linger parameter, which is the address of a pointer that should refer to the osd request structure used to issue a request to an osd. Only one case ever supplies a non-null linger argument: an

Re: [PATCH REPOST 6/6] rbd: move remaining osd op setup into rbd_osd_req_op_create()

2013-01-16 Thread Alex Elder
On 01/16/2013 10:23 PM, Josh Durgin wrote: On 01/04/2013 07:07 AM, Alex Elder wrote: The two remaining osd ops used by rbd are CEPH_OSD_OP_WATCH and CEPH_OSD_OP_NOTIFY_ACK. Move the setup of those operations into rbd_osd_req_op_create(), and get rid of rbd_create_rw_op() and

Re: flashcache

2013-01-16 Thread Stefan Priebe - Profihost AG
Hi Mark, Am 16.01.2013 um 22:53 schrieb Mark With only 2 SSDs for 12 spinning disks, you'll need to make sure the SSDs are really fast. I use Intel 520s for testing which are great, but I wouldn't use them in production. Why not? I use them for a ssd only ceph cluster. Stefan-- To

RE: Ceph slow request unstable issue

2013-01-16 Thread Chen, Xiaoxi
Some update summary for tested case till now: Ceph is v0.56.1 1. RBD:Ubuntu 13.04 + 3.7Kernel OSD:Ubuntu 13.04 + 3.7Kernel XFS Result: Kernel Panic on both RBD and OSD sides 2. RBD:Ubuntu 13.04 +3.2Kernel OSD:Ubuntu 13.04 +3.2Kernel XFS

Re: code coverage and teuthology

2013-01-16 Thread Loic Dachary
On 01/15/2013 06:21 PM, Josh Durgin wrote: On 01/15/2013 02:10 AM, Loic Dachary wrote: On 01/14/2013 06:26 PM, Josh Durgin wrote: Looking at how it's run automatically might help: https://github.com/ceph/teuthology/blob/master/teuthology/coverage.py#L88 You should also add 'coverage:

HOWTO: teuthology and code coverage

2013-01-16 Thread Loic Dachary
Hi, I'm happy to report that running teuthology to get a lcov code coverage report worked for me. http://dachary.org/wp-uploads/2013/01/teuthology/total/mon/Monitor.cc.gcov.html It took me a while to figure out the logic (thanks Josh for the help :-). I wrote a HOWTO explaining the steps in