Re: Correct usage of rbd_aio_release

2013-08-12 Thread Josh Durgin
On 08/12/2013 06:36 AM, Sylvain Munaut wrote: Hi, When should / can rbd_aio_release be called exactly ? It should be called whenever you're certain you won't be using the rbd_completion_t anymore (and an rbd_completion_t should not be reused after rbd_aio_{read,write,discard,flush} is called

Re: teuthology and code coverage

2013-08-14 Thread Josh Durgin
On 08/13/2013 10:01 AM, Loic Dachary wrote: Hi, When running teuthology from a laptop with the configuration below and ./virtualenv/bin/teuthology --archive=/tmp/teuthology try.yaml it then fails on ./virtualenv/bin/teuthology-coverage -v --html-output /tmp/html -o /tmp/lcov

Re: Show outdated diffs on github

2013-08-16 Thread Josh Durgin
On 08/15/2013 11:45 PM, Loic Dachary wrote: Hi Josh, From https://github.com/ceph/ceph/pull/504 I see that your comments persisted despite a rebase and show as jdurgin discussed an outdated diff 8 hours ago I would like to do the same but when I comment inline, it shows differently and

Re: libvirt: Removing RBD volumes with snapshots, auto purge or not?

2013-08-20 Thread Josh Durgin
On 08/20/2013 08:36 AM, Wido den Hollander wrote: Hi, The current [0] libvirt storage pool code simply calls rbd_remove without anything else. As far as I know rbd_remove will fail if the image still has snapshots, you have to remove those snapshots first before you can remove the image. The

[PATCH] rbd: fix I/O error propagation for reads

2013-08-26 Thread Josh Durgin
incomplete. Fixes: http://tracker.ceph.com/issues/5647 Signed-off-by: Josh Durgin josh.dur...@inktank.com --- drivers/block/rbd.c | 14 +++--- 1 files changed, 7 insertions(+), 7 deletions(-) diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c index 0d669ae..f8fd7d3 100644

Re: [PATCH] rbd: fix I/O error propagation for reads

2013-08-27 Thread Josh Durgin
. If you look at the admin socket's dump_historic_ops do you see higher op durations around the dips in performance? What about any correlation with underlying disk stats from iostat -x? Thanks, Mike Dawson On 8/26/2013 9:34 PM, Josh Durgin wrote: When a request returns an error, the driver needs

[PATCH] rbd: fix buffer size for writes to images with snapshots

2013-08-27 Thread Josh Durgin
: libceph: BUG_ON(p msg-front.iov_base + msg-front.iov_len); Fix this by associating the obj_request with its img_request immediately after it's created, before the osd request is created. Fixes: http://tracker.ceph.com/issues/5760 Suggested-by: Alex Elder alex.el...@linaro.org Signed-off-by: Josh

[PATCH 1/3] rbd: fix null dereference in dout

2013-08-29 Thread Josh Durgin
The order parameter is sometimes NULL in _rbd_dev_v2_snap_size(), but the dout() always derefences it. Move this to another dout() protected by a check that order is non-NULL. Signed-off-by: Josh Durgin josh.dur...@inktank.com --- drivers/block/rbd.c |8 +--- 1 files changed, 5

[PATCH 2/3] libceph: add function to ensure notifies are complete

2013-08-29 Thread Josh Durgin
for the event. If the queue is flushed after the event is unregistered, the caller can be sure no more watch callbacks will occur for the canceled watch. Signed-off-by: Josh Durgin josh.dur...@inktank.com --- include/linux/ceph/osd_client.h |2 ++ net/ceph/osd_client.c | 11

[PATCH 0/3] shutdown race and debug fix

2013-08-29 Thread Josh Durgin
These patches fix a race in rbd shutdown and a null deref when debugging is enabled. They're also available in the wip-rbd-bugs branch of ceph-client.git. Josh Durgin (3): rbd: fix null dereference in dout libceph: add function to ensure notifies are complete rbd: close remove vs. notify

[PATCH 3/3] rbd: close remove vs. notify race leading to use-after-free

2013-08-29 Thread Josh Durgin
the queue at this point since the watch has been canceled. Fixes: http://tracker.ceph.com/issues/5636 Signed-off-by: Josh Durgin josh.dur...@inktank.com --- drivers/block/rbd.c | 25 ++--- 1 files changed, 22 insertions(+), 3 deletions(-) diff --git a/drivers/block/rbd.c b

Re: [PATCH 3/3] rbd: close remove vs. notify race leading to use-after-free

2013-08-29 Thread Josh Durgin
On 08/29/2013 08:21 AM, Alex Elder wrote: On 08/29/2013 01:24 AM, Josh Durgin wrote: Removing a device deallocates the disk, unschedules the watch, and finally cleans up the rbd_dev structure. rbd_dev_refresh(), called from the watch callback, updates the disk size and rbd_dev structure

Re: [PATCH 3/3] rbd: close remove vs. notify race leading to use-after-free

2013-08-29 Thread Josh Durgin
On 08/29/2013 07:46 AM, Sage Weil wrote: On Wed, 28 Aug 2013, Josh Durgin wrote: Removing a device deallocates the disk, unschedules the watch, and finally cleans up the rbd_dev structure. rbd_dev_refresh(), called from the watch callback, updates the disk size and rbd_dev structure

[PATCH v2 2/3] rbd: complete notifies before cleaning up osd_client and rbd_dev

2013-08-29 Thread Josh Durgin
To ensure rbd_dev is not used after it's released, flush all pending notify callbacks before calling rbd_dev_image_release(). No new notifies can be added to the queue at this point because the watch has already be unregistered with the osd_client. Signed-off-by: Josh Durgin josh.dur

[PATCH v2 1/3] rbd: fix use-after free of rbd_dev-disk

2013-08-29 Thread Josh Durgin
been freed. To fix this, add a shutting_down flag and a mutex protecting it to rbd_dev. Take the mutex and check whether the flag is set before using rbd_dev-disk. Move this disk-updating to a separate function as well. Fixes: http://tracker.ceph.com/issues/5636 Signed-off-by: Josh Durgin josh.dur

[PATCH v2 3/3] rbd: make rbd_obj_notify_ack() synchronous

2013-08-29 Thread Josh Durgin
, waiting for the notify ack to complete before returning from the watch callback ensures there are no notify acks in flight during shutdown. Signed-off-by: Josh Durgin josh.dur...@inktank.com --- drivers/block/rbd.c |4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/drivers

[PATCH] rbd: ignore unmapped snapshots that no longer exist

2013-08-29 Thread Josh Durgin
This prevents erroring out while adding a device when a snapshot unrelated to the current mapping is deleted between reading the snapshot context and reading the snapshot names. If the mapped snapshot name is not found an error still occurs as usual. Signed-off-by: Josh Durgin josh.dur

Re: libvirt: Using rbd_create3 to create format 2 images

2013-08-30 Thread Josh Durgin
On 08/30/2013 02:42 AM, Wido den Hollander wrote: Hi, I created the attached patch to have libvirt create images with format 2 by default, this would simplify the CloudStack code and could also help other projects. The problem with libvirt is that there is no mechanism to supply information

Re: libvirt: Using rbd_create3 to create format 2 images

2013-09-03 Thread Josh Durgin
On 09/02/2013 07:58 AM, Wido den Hollander wrote: On 08/30/2013 08:38 PM, Wido den Hollander wrote: On 08/30/2013 05:26 PM, Josh Durgin wrote: On 08/30/2013 02:42 AM, Wido den Hollander wrote: Hi, I created the attached patch to have libvirt create images with format 2 by default, this would

Re: Add Metavariable $imagename

2013-09-05 Thread Josh Durgin
On 09/04/2013 09:22 AM, Sage Weil wrote: Hi Mike, On Wed, 4 Sep 2013, Mike Dawson wrote: Thanks for the feature request Mark! To illustrate where my understanding ends, here is a patch to stub in this functionality: From 137ebc2d5326ca710a5e99e2899fd851b02c10f7 Mon Sep 17 00:00:00 2001

Re: Add Metavariable $imagename

2013-09-05 Thread Josh Durgin
On 09/05/2013 07:59 AM, Sage Weil wrote: On Wed, 4 Sep 2013, Josh Durgin wrote: On 09/04/2013 09:22 AM, Sage Weil wrote: Hi Mike, On Wed, 4 Sep 2013, Mike Dawson wrote: Thanks for the feature request Mark! To illustrate where my understanding ends, here is a patch to stub

[PATCH v3 2/5] rbd: make rbd_obj_notify_ack() synchronous

2013-09-09 Thread Josh Durgin
. Signed-off-by: Josh Durgin josh.dur...@inktank.com --- drivers/block/rbd.c | 11 ++- 1 files changed, 6 insertions(+), 5 deletions(-) diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c index bf89e34..9e5f07f 100644 --- a/drivers/block/rbd.c +++ b/drivers/block/rbd.c @@ -2808,7 +2808,7

[PATCH v3 1/5] rbd: complete notifies before cleaning up osd_client and rbd_dev

2013-09-09 Thread Josh Durgin
To ensure rbd_dev is not used after it's released, flush all pending notify callbacks before calling rbd_dev_image_release(). No new notifies can be added to the queue at this point because the watch has already be unregistered with the osd_client. Signed-off-by: Josh Durgin josh.dur

[PATCH v3 0/5] fix shutdown races and snapshot error handling

2013-09-09 Thread Josh Durgin
. It also restricts where it holds the lock to avoid an inversion with bdev-lock. Patch 4 is the same. Patch 5 fixes an inconsistency noticed by Alex in his review of patch 4. Josh Durgin (5): rbd: complete notifies before cleaning up osd_client and rbd_dev rbd: make rbd_obj_notify_ack

[PATCH v3 4/5] rbd: ignore unmapped snapshots that no longer exist

2013-09-09 Thread Josh Durgin
This prevents erroring out while adding a device when a snapshot unrelated to the current mapping is deleted between reading the snapshot context and reading the snapshot names. If the mapped snapshot name is not found an error still occurs as usual. Signed-off-by: Josh Durgin josh.dur

[PATCH v3 5/5] rbd: fix error handling from rbd_snap_name()

2013-09-09 Thread Josh Durgin
an ERR_PTR in rbd_dev_v1_snap_name() so that the specific error can be propagated, and it is consistent with rbd_dev_v2_snap_name(). Handle the ERR_PTR in the only rbd_snap_name() caller. Suggested-by: Alex Elder alex.el...@linaro.org Signed-off-by: Josh Durgin josh.dur...@inktank.com --- drivers

Re: [PATCH v2 1/3] rbd: fix use-after free of rbd_dev-disk

2013-09-09 Thread Josh Durgin
On 09/03/2013 05:41 AM, Alex Elder wrote: On 08/29/2013 07:57 PM, Josh Durgin wrote: Removing a device deallocates the disk, unschedules the watch, and finally cleans up the rbd_dev structure. rbd_dev_refresh(), called from the watch callback, updates the disk size and rbd_dev structure

Re: [PATCH] rbd: ignore unmapped snapshots that no longer exist

2013-09-09 Thread Josh Durgin
On 09/09/2013 12:31 AM, Josh Durgin wrote: On 09/03/2013 06:29 AM, Alex Elder wrote: On 08/29/2013 09:19 PM, Josh Durgin wrote: This prevents erroring out while adding a device when a snapshot unrelated to the current mapping is deleted between reading the snapshot context and reading

Re: [PATCH v3 3/5] rbd: fix use-after free of rbd_dev-disk

2013-09-09 Thread Josh Durgin
On 09/09/2013 06:37 AM, Alex Elder wrote: On 09/09/2013 02:17 AM, Josh Durgin wrote: Removing a device deallocates the disk, unschedules the watch, and finally cleans up the rbd_dev structure. rbd_dev_refresh(), called from the watch callback, updates the disk size and rbd_dev structure

Re: [PATCH v2] rbd: add ioctl for rbd

2013-09-17 Thread Josh Durgin
On 09/17/2013 07:42 AM, Alex Elder wrote: On 09/17/2013 02:04 AM, Guangliang Zhao wrote: When running the following commands: [root@ceph0 mnt]# blockdev --setro /dev/rbd2 [root@ceph0 mnt]# blockdev --getro /dev/rbd2 0 I think this is a good change to make, and I think what

Re: [PATCH v2] rbd: add ioctl for rbd

2013-09-17 Thread Josh Durgin
On 09/17/2013 09:11 AM, Alex Elder wrote: On 09/17/2013 10:24 AM, Josh Durgin wrote: block/ioctl.c will already call set_device_ro() for us after this driver-specific handling completes successfully, so we don't need to call it here. Also, it appears the block layer has a bug in that it does

Re: [ceph-users] Scaling RBD module

2013-09-18 Thread Josh Durgin
On 09/17/2013 03:30 PM, Somnath Roy wrote: Hi, I am running Ceph on a 3 node cluster and each of my server node is running 10 OSDs, one for each disk. I have one admin node and all the nodes are connected with 2 X 10G network. One network is for cluster and other one configured as public

Re: [ceph-users] Scaling RBD module

2013-09-19 Thread Josh Durgin
-home on /home type ext4 (rw) Any idea what went wrong here ? Thanks Regards Somnath -Original Message- From: Josh Durgin [mailto:josh.dur...@inktank.com] Sent: Wednesday, September 18, 2013 6:10 PM To: Somnath Roy Cc: Sage Weil; ceph-devel@vger.kernel.org; Anirban Ray; ceph-us

Re: [PATCH v3] rbd: add ioctl for rbd

2013-09-23 Thread Josh Durgin
() is not called with the bdev lock held. With the change to rbd_request_fn() Alex mentioned, could you verify it has no problems with lockdep enabled? If so, it looks good to me. Reviewed-by: Josh Durgin josh.dur...@inktank.com + + switch (cmd) { + case BLKROSET: + ret

Re: Hiding auth key string for the qemu process

2013-09-23 Thread Josh Durgin
On 09/22/2013 12:54 PM, Andrey Korolyov wrote: Hello, Since it was a long time from enabling cephx by default and we may think that everyone using it, is seems worthy to introduce bits of code hiding the key from cmdline. First applicable place for such improvement is most-likely OpenStack envs

Re: [PATCH] fix librados aio read buffer handling

2013-09-30 Thread Josh Durgin
On 09/30/2013 08:38 AM, Sage Weil wrote: On Mon, 30 Sep 2013, Rutger ter Borg wrote: Dear all, please find attached a patch that enables a user to pass user-owned buffers into librados' aio_read. The patch (against dumpling) removes the buf and pbl data members in AioCompletionImpl. * The

[PATCH 2/2] rbd: only set disk to read-only once

2013-10-01 Thread Josh Durgin
-off-by: Josh Durgin josh.dur...@inktank.com --- drivers/block/rbd.c |2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c index b3b1b57..fc3ebd9 100644 --- a/drivers/block/rbd.c +++ b/drivers/block/rbd.c @@ -490,7 +490,6 @@ static int

[PATCH 0/2] clean up read only mode

2013-10-01 Thread Josh Durgin
branch of ceph-client.git. Josh Durgin (2): rbd: move calls that may sleep out of spin lock range rbd: only set disk to read-only once drivers/block/rbd.c | 38 -- 1 files changed, 24 insertions(+), 14 deletions(-) -- 1.7.2.5 -- To unsubscribe from

[PATCH 1/2] rbd: move calls that may sleep out of spin lock range

2013-10-01 Thread Josh Durgin
get_user() and set_disk_ro() may allocate memory, leading to a potential deadlock if theye are called while a spin lock is held. Move the acquisition and release of rbd_dev-lock from rbd_ioctl() into rbd_ioctl_set_ro(), so it can occur between get_user() and set_disk_ro(). Signed-off-by: Josh

Re: Kernel panic putting zpool on iSCSI LUN using bs_rbd

2013-10-14 Thread Josh Durgin
On 10/07/2013 08:49 PM, Eric Eastman wrote: I am not sure if this is a bs_rbd, tgt or zfs issue, but I can reliably crash my Centos 6.4 system running tgt 1.0.40 using a bs_rbd backstore by creating a zpool. Using tgt with a file backed store does not panic the system when creating a zpool.

Re: [ceph-users] poor read performance on rbd+LVM, LVM overload

2013-10-20 Thread Josh Durgin
On 10/20/2013 08:18 AM, Ugis wrote: output follows: #pvs -o pe_start /dev/rbd1p1 1st PE 4.00m # cat /sys/block/rbd1/queue/minimum_io_size 4194304 # cat /sys/block/rbd1/queue/optimal_io_size 4194304 Well, the parameters are being set at least. Mike, is it possible that having

Re: feedback on supporting libc++

2013-10-30 Thread Josh Durgin
On 10/29/2013 03:51 PM, Noah Watkins wrote: Out of the box on OSX Mavericks libc++ [1] is being used as opposed to libstdc++. One of the issues is that stuff from tr1 isn't available (e.g. std::tr1::shared_ptr), as they have moved to std in c++11. I'm looking for any feedback on this patch set,

Ceph and OpenStack unconference on Friday

2013-11-04 Thread Josh Durgin
We're having an unconference session at the OpenStack design summit on Friday at 2:20pm. If you're interested in Ceph integration with OpenStack, please join us to discuss future development. Josh -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to

Re: [RBD][OpenStack]The way to solve problem when boot VM and root disk size is specified

2013-11-12 Thread Josh Durgin
On 11/11/2013 11:10 PM, Haomai Wang wrote: Hi all, Now OpenStack Nova master branch still exists a bug when you boot a VM which root disk size is specified. The storage backend of Nova also is rbd. For example, you boot a VM and specify 10G as root disk size. But the image is only 1G. Then

Re: /etc/init.d/ceph vs upstart

2013-11-25 Thread Josh Durgin
On 11/25/2013 11:01 AM, Tim Spriggs wrote: ... ping On Thu, Nov 7, 2013 at 3:31 PM, Tim Spriggs t...@uahirise.org wrote: Oops, I just realized I did the patch in the wrong direction :) On Thu, Nov 7, 2013 at 3:06 PM, Tim Spriggs t...@uahirise.org wrote: Hi All, I am battling extraneous

Re: Osd magic detection failure in master

2013-12-02 Thread Josh Durgin
On 12/02/2013 06:42 PM, Mark Kirkwood wrote: I just updated master (a5eda4fcc34461dbc0fcc47448f8456097de15eb), and am seeing OSDs failing to start: 2013-12-03 15:37:01.291200 7f488e1157c0 -1 OSD magic != my ceph osd volume v026 failed: 'ulimit -n 32768; /usr/bin/ceph-osd -i 0 --pid-file

[PATCH 0/3] block I/O when cluster is full

2013-12-03 Thread Josh Durgin
-full branch of ceph-client.git. Josh Durgin (3): libceph: block I/O when PAUSE or FULL osd map flags are set libceph: add an option to configure client behavior when osds are full rbd: document rbd-specific options Documentation/ABI/testing/sysfs-bus-rbd | 19

[PATCH 3/3] rbd: document rbd-specific options

2013-12-03 Thread Josh Durgin
osd_full_behavior only affects rbd, so document it along with read-only and read-write. Signed-off-by: Josh Durgin josh.dur...@inktank.com --- Documentation/ABI/testing/sysfs-bus-rbd | 19 +++ 1 file changed, 19 insertions(+) diff --git a/Documentation/ABI/testing/sysfs-bus

[PATCH 1/3] libceph: block I/O when PAUSE or FULL osd map flags are set

2013-12-03 Thread Josh Durgin
the monitor if any of these flags are set, so paused requests can be unblocked as soon as possible. Fixes: http://tracker.ceph.com/issues/6079 Signed-off-by: Josh Durgin josh.dur...@inktank.com --- include/linux/ceph/osd_client.h |1 + net/ceph/osd_client.c | 29

[PATCH 2/3] libceph: add an option to configure client behavior when osds are full

2013-12-03 Thread Josh Durgin
Default to blocking requests to be consistent with userspace. Some applications may prefer the previous behavior of returning an error instead, so make that an option. CephFS implements returning -ENOSPC at a higher level, so only rbd is really affected by this. Signed-off-by: Josh Durgin

Re: [PATCH 0/3] block I/O when cluster is full

2013-12-05 Thread Josh Durgin
On 12/03/2013 03:12 PM, Josh Durgin wrote: These patches allow rbd to block writes instead of returning errors when OSDs are full enough that the FULL flag is set in the osd map. This avoids filesystems on top of rbd getting confused by transient EIOs if the cluster oscillates between full

Re: [PATCH 0/3] block I/O when cluster is full

2013-12-06 Thread Josh Durgin
On 12/05/2013 08:58 PM, Gregory Farnum wrote: On Thu, Dec 5, 2013 at 5:47 PM, Josh Durgin josh.dur...@inktank.com wrote: On 12/03/2013 03:12 PM, Josh Durgin wrote: These patches allow rbd to block writes instead of returning errors when OSDs are full enough that the FULL flag is set

Re: [PATCH 1/3] libceph: block I/O when PAUSE or FULL osd map flags are set

2013-12-09 Thread Josh Durgin
devices behave this way as well, blocking when they run out of space until more space is available. Do you have an idea for avoiding this? On 2013/12/4 7:12, Josh Durgin wrote: The PAUSEWR and PAUSERD flags are meant to stop the cluster from processing writes and reads, respectively. The FULL flag

Re: [PATCH 0/3] block I/O when cluster is full

2013-12-09 Thread Josh Durgin
On 12/06/2013 06:24 PM, Gregory Farnum wrote: On Fri, Dec 6, 2013 at 6:16 PM, Josh Durgin josh.dur...@inktank.com wrote: On 12/05/2013 08:58 PM, Gregory Farnum wrote: On Thu, Dec 5, 2013 at 5:47 PM, Josh Durgin josh.dur...@inktank.com wrote: On 12/03/2013 03:12 PM, Josh Durgin wrote

Re: [PATCH 0/3] block I/O when cluster is full

2013-12-09 Thread Josh Durgin
On 12/09/2013 04:19 PM, Gregory Farnum wrote: On Mon, Dec 9, 2013 at 4:11 PM, Josh Durgin josh.dur...@inktank.com wrote: On 12/06/2013 06:24 PM, Gregory Farnum wrote: On Fri, Dec 6, 2013 at 6:16 PM, Josh Durgin josh.dur...@inktank.com wrote: Don't bother trying to stop ENOSPC on the client

[PATCH v2 1/2] libceph: block I/O when PAUSE or FULL osd map flags are set

2013-12-11 Thread Josh Durgin
the monitor if any of these flags are set, so paused requests can be unblocked as soon as possible. Fixes: http://tracker.ceph.com/issues/6079 Signed-off-by: Josh Durgin josh.dur...@inktank.com --- include/linux/ceph/osd_client.h |1 + net/ceph/osd_client.c | 29

[PATCH v2 0/2] block I/O when cluster is full

2013-12-11 Thread Josh Durgin
this at all, it is left for future work. These patches can also be found in the wip-full-6938 branch of ceph-client.git. Josh Durgin (2): libceph: block I/O when PAUSE or FULL osd map flags are set libceph: resend all writes after the osdmap loses the full flag include/linux/ceph

[PATCH v2 2/2] libceph: resend all writes after the osdmap loses the full flag

2013-12-11 Thread Josh Durgin
to be fixed to avoid the race. Old clients talking to osds with this fix may hang instead of returning EIO and potentially corrupting an fs. New clients talking to old osds have the same behavior as before if they encounter this race. Fixes: http://tracker.ceph.com/issues/6938 Signed-off-by: Josh

Re: [PATCH 0/2] rbd: tear down watch request if rbd_dev_device_setup() fails

2013-12-26 Thread Josh Durgin
changed, 28 insertions(+), 13 deletions(-) These look good to me. Reviewed-by: Josh Durgin josh.dur...@inktank.com -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo

Re: [PATCH] fix librados aio read buffer handling

2013-12-30 Thread Josh Durgin
On 11/27/2013 12:49 AM, Rutger ter Borg wrote: On 2013-10-01 00:52, Josh Durgin wrote: I'm fine applying this now (with one fix). It's a nice cleanup even if things change more soon. For the C interface, the return value stored in the AioCompletionImpl needs to be the length read, so

Re: ceph and scsi reservation-like locking

2013-12-30 Thread Josh Durgin
On 12/28/2013 06:34 PM, James Harper wrote: Is the rbd locking feature-compatible with scsi3 persistent reservations? It wasn't designed with scsi-3 exactly in mind, but using exclusive locks and fencing might suffice. To fence a client, you'd get its address from rbd_list_lockers() and

Re: Shared library symbol visibility

2013-12-30 Thread Josh Durgin
On 12/30/2013 09:56 AM, Noah Watkins wrote: It looks like we may be outgrowing the use of export-symbols-regex and friends to control symbol visibility for published shared libraries. On Linux, ld seems to be quite content linking against hidden symbols, but at least on OSX with Clang it seems

Re: feedback on supporting libc++

2013-12-30 Thread Josh Durgin
On 12/27/2013 03:34 PM, Noah Watkins wrote: On Wed, Oct 30, 2013 at 2:02 PM, Josh Durgin josh.dur...@inktank.com wrote: On 10/29/2013 03:51 PM, Noah Watkins wrote: unsafe to me. Could you check whether you can run 'rados ls' compiled against an old librados, but dynamically loading librados

Re: Patch for rados tool to allow listomapvals to list all k/v pairs

2013-12-30 Thread Josh Durgin
On 12/28/2013 05:59 AM, Bjørnar Ness wrote: Current code only lists first 512 k/v pairs, attached patch. Thanks for looking into this! I moved the last_read assignment out of the conditional so it applies to nonprintable values too, and added a simple test case:

Re: feedback on supporting libc++

2013-12-31 Thread Josh Durgin
On 12/31/2013 08:59 AM, Noah Watkins wrote: Thanks for testing that Josh. Before cleaning up this patch set, I have a few questions. I'm still not clear on how to handle the std::tr1::shared_ptr ObjListCtx ctx; in librados.hpp. If we change this to ceph::shared_ptr, then we'll also need to

Re: Master needs newer version of boost on centos?

2014-02-14 Thread Josh Durgin
ICL was added in 1.46 according to http://www.boost.org/doc/libs/. This also fails on debian wheezy with only 1.42 available: http://tracker.ceph.com/issues/7422 On 02/14/2014 04:26 PM, Matt W. Benjamin wrote: according to google, 1.41 ahould have it. - Sage Weil s...@inktank.com wrote:

Re: Assertion error in librados

2014-02-25 Thread Josh Durgin
That's a good idea. This particular assert in a Mutex is almost always a use-after-free of the Mutex or structure containing it though. On 02/25/2014 09:33 AM, Noah Watkins wrote: Perhaps using gtest-style asserts (ASSERT_EQ(r, 0)) in Ceph would be useful so we can see parameter values to the

Re: [PATCH 1/3 v2] rbd: skip the copyup when an entire object writing

2014-03-27 Thread Josh Durgin
, but discards done e.g. by mkfs are more likely to benefit from this optimization. Nice refactoring of that conditional too (thanks for suggesting that Alex)! Reviewed-by: Josh Durgin josh.dur...@inktank.com drivers/block/rbd.c | 49 ++--- 1 file

Re: [PATCH 2/3 v2] rbd: extend the operation type

2014-03-27 Thread Josh Durgin
On 03/12/2014 08:21 PM, Guangliang Zhao wrote: It could only handle the read and write operations now, extend it for the coming discard support. Signed-off-by: Guangliang Zhao lucienc...@gmail.com --- Looks good. Reviewed-by: Josh Durgin josh.dur...@inktank.com drivers/block/rbd.c | 96

Re: [PATCH 3/3 v2] rbd: add discard support for rbd

2014-03-27 Thread Josh Durgin
On 03/12/2014 08:21 PM, Guangliang Zhao wrote: This patch add the discard support for rbd driver. There are there types operation in the driver: 1. The objects would be removed if they completely contained within the discard range. 2. The objects would be truncated if they partly contained

Re: [PATCH 0/5] wip-tunables3

2014-03-27 Thread Josh Durgin
good. Reviewed-by: Josh Durgin josh.dur...@inktank.com -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: queue for linus

2014-04-07 Thread Josh Durgin
On 04/04/2014 09:17 PM, Sage Weil wrote: Hi everyone, This is what we have queued up for Linus for 3.15-rc1. Is there anything missing? I pulled David Howell's patch out of for-linus (since I think it should go in via Al's tree) and rebased. Is there anything missing? Josh, do you want to

Re: [PATCH 3/3 v3] rbd: add discard support for rbd

2014-04-08 Thread Josh Durgin
On 04/01/2014 07:22 AM, Guangliang Zhao wrote: This patch add the discard support for rbd driver. There are three types operation in the driver: 1. The objects would be removed if they completely contained within the discard range. 2. The objects would be truncated if they partly contained

[PATCH 00/11] rbd bug fix and discard support

2014-04-08 Thread Josh Durgin
rbd: add discard support for rbd Josh Durgin (8): rbd: access snapshot context and mapping size safely rbd: read image size for discard check safely rbd: fix snapshot context reference count for discards rbd: tolerate -ENOENT for discard operations rbd: make discard trigger copy-on-write

[PATCH 01/11] rbd: access snapshot context and mapping size safely

2014-04-08 Thread Josh Durgin
, rbd_parent_request_create() can just pass NULL for snapc, since the snapshot context is only relevant for writes. Signed-off-by: Josh Durgin josh.dur...@inktank.com --- drivers/block/rbd.c | 31 +++ 1 file changed, 19 insertions(+), 12 deletions(-) diff --git a/drivers/block/rbd.c

[PATCH 06/11] rbd: fix snapshot context reference count for discards

2014-04-08 Thread Josh Durgin
Discards take a reference to the snapshot context of an image when they are created. This reference needs to be cleaned up when the request is done just as it is for regular writes. Signed-off-by: Josh Durgin josh.dur...@inktank.com --- drivers/block/rbd.c |3 ++- 1 file changed, 2

[PATCH 02/11] rbd: skip the copyup when an entire object writing

2014-04-08 Thread Josh Durgin
From: Guangliang Zhao lucienc...@gmail.com It need to copyup the parent's content when layered writing, but an entire object write would overwrite it, so skip it. Signed-off-by: Guangliang Zhao lucienc...@gmail.com Reviewed-by: Josh Durgin josh.dur...@inktank.com Reviewed-by: Alex Elder el

[PATCH 05/11] rbd: read image size for discard check safely

2014-04-08 Thread Josh Durgin
the semaphore in this function. Signed-off-by: Josh Durgin josh.dur...@inktank.com --- drivers/block/rbd.c | 18 -- 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c index 9dc33d9..486e4b5 100644 --- a/drivers/block/rbd.c +++ b

[PATCH 03/11] rbd: extend the operation type

2014-04-08 Thread Josh Durgin
From: Guangliang Zhao lucienc...@gmail.com It could only handle the read and write operations now, extend it for the coming discard support. Signed-off-by: Guangliang Zhao lucienc...@gmail.com Reviewed-by: Josh Durgin josh.dur...@inktank.com Reviewed-by: Alex Elder el...@linaro.org --- drivers

[PATCH 10/11] rbd: use helpers to handle discard for layered images correctly

2014-04-08 Thread Josh Durgin
discard requests as well. Signed-off-by: Josh Durgin josh.dur...@inktank.com --- drivers/block/rbd.c | 47 +++ 1 file changed, 19 insertions(+), 28 deletions(-) diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c index 7bcdeda..c8fc8fc 100644

[PATCH 09/11] rbd: extract a method for adding object operations

2014-04-08 Thread Josh Durgin
the original operations should be the same as when first sending them, so move it to a helper function. op_type only needs to be checked once, so create a helper for that as well and call it outside the loop in rbd_img_request_fill(). Signed-off-by: Josh Durgin josh.dur...@inktank.com --- drivers

[PATCH 04/11] rbd: add discard support for rbd

2014-04-08 Thread Josh Durgin
://tracker.ceph.com/issues/190 Signed-off-by: Guangliang Zhao lucienc...@gmail.com Reviewed-by: Josh Durgin josh.dur...@inktank.com Reviewed-by: Alex Elder el...@linaro.org --- drivers/block/rbd.c | 109 +++ 1 file changed, 92 insertions(+), 17 deletions

[PATCH 08/11] rbd: make discard trigger copy-on-write

2014-04-08 Thread Josh Durgin
Discard requests are a form of write, so they should go through the same process as plain write requests and trigger copy-on-write for layered images. Signed-off-by: Josh Durgin josh.dur...@inktank.com --- drivers/block/rbd.c |3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git

Re: xio-rados-firefly branch update

2014-04-28 Thread Josh Durgin
On 04/28/2014 04:40 PM, Gregory Farnum wrote: The buffer changes appear to have some unnecessary Xio class declarations in buffer.h, and it’d be nice if all of that was guarded by HAVE_XIO blocks. buffer.h is exposed by the librados c++ api, so hopefully nothing xio-specific really needs to be

Re: [ceph-users] Data still in OSD directories after removing

2014-05-21 Thread Josh Durgin
On 05/21/2014 03:03 PM, Olivier Bonvalet wrote: Le mercredi 21 mai 2014 à 08:20 -0700, Sage Weil a écrit : You're certain that that is the correct prefix for the rbd image you removed? Do you see the objects lists when you do 'rados -p rbd ls - | grep prefix'? I'm pretty sure yes : since I

Re: A plea for more tooling usage

2014-05-28 Thread Josh Durgin
On 05/27/2014 03:19 PM, Thorsten Behrens wrote: I wrote: Sage Weil wrote: If anybody is interested in helping with that effort, pull requests are very welcome! :) [snip] As hinted at in the patch, something like boost::program_options would be nice, but that's a chunk of work I'd rather

Re: Librbd licensing

2014-06-02 Thread Josh Durgin
On 06/02/2014 10:22 AM, Sage Weil wrote: Ideally the change comes from Josh, who originally put the notice there, but I think it shouldn't matter. We relicensed rbd.cc as LGPL2 a while back (it was GPL due to a header we used?) and got confirmations from all authors. It might be worth doing a

Re: [Feature]Proposal for adding a new flag named shared to support performance and statistic purpose

2014-06-09 Thread Josh Durgin
On 06/05/2014 12:01 AM, Haomai Wang wrote: Hi, Previously I sent a mail about the difficult of rbd snapshot size statistic. The main solution is using object map to store the changes. The problem is we can't handle with multi client concurrent modify. Lack of object map(like pointer map in

Re: [Feature]Proposal for adding a new flag named shared to support performance and statistic purpose

2014-06-10 Thread Josh Durgin
add a blueprint for it on the wiki: https://wiki.ceph.com/Planning/Blueprints/Submissions Josh On Tue, Jun 10, 2014 at 9:16 AM, Josh Durgin josh.dur...@inktank.com wrote: On 06/05/2014 12:01 AM, Haomai Wang wrote: Hi, Previously I sent a mail about the difficult of rbd snapshot size

Re: [ceph-users] Fwd: CEPH Multitenancy and Data Isolation

2014-06-10 Thread Josh Durgin
On 06/10/2014 01:56 AM, Vilobh Meshram wrote: How does CEPH guarantee data isolation for volumes which are not meant to be shared in a Openstack tenant? When used with OpenStack the data isolation is provided by the Openstack level so that all users who are part of same tenant will be able to

Re: [PATCH] rbd: handle parent_overlap on writes correctly

2014-06-12 Thread Josh Durgin
for the purposes of the overlap check. Signed-off-by: Ilya Dryomov ilya.dryo...@inktank.com --- Good catch! This should be included in any stable kernels 3.10 or later too. Reviewed-by: Josh Durgin josh.dur...@inktank.com drivers/block/rbd.c | 10 +- 1 file changed, 9 insertions(+), 1

Re: [PATCH] rbd.cc: Check io-size avoid floating point exception.

2014-07-10 Thread Josh Durgin
On 06/12/2014 01:15 AM, Ma, Jianpeng wrote: In func do_bench_write if io_size is zero,it can cause floating point execption. Signed-off-by: Jianpeng Ma jianpeng...@intel.com Applied, thanks! Josh --- src/rbd.cc | 8 1 file changed, 8 insertions(+) diff --git a/src/rbd.cc

Re: [PATCH REPOST 0/2] libceph: embed r_trail struct in ceph_osd_request()

2013-01-16 Thread Josh Durgin
not there. -Alex [PATCH REPOST 1/2] libceph: always allow trail in osd request [PATCH REPOST 2/2] libceph: kill op_needs_trail() These look good. Reviewed-by: Josh Durgin josh.dur...@inktank.com -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message

Re: [PATCH REPOST] rbd: separate layout init

2013-01-16 Thread Josh Durgin
On 01/03/2013 02:55 PM, Alex Elder wrote: Pull a block of code that initializes the layout structure in an osd request into its own function so it can be reused. Signed-off-by: Alex Elder el...@inktank.com --- Reviewed-by: Josh Durgin josh.dur...@inktank.com drivers/block/rbd.c | 23

Re: [PATCH REPOST 0/6] libceph: parameter cleanup

2013-01-16 Thread Josh Durgin
in ceph_osdc_alloc_request() These all look good. Reviewed-by: Josh Durgin josh.dur...@inktank.com -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: [PATCH REPOST 0/4] rbd: explicitly support only one osd op

2013-01-16 Thread Josh Durgin
with ops [PATCH REPOST 3/4] rbd: there is really only one op [PATCH REPOST 4/4] rbd: assume single op in a request These look good. Reviewed-by: Josh Durgin josh.dur...@inktank.com -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord

Re: [PATCH REPOST 0/3] rbd: no need for file mapping calculation

2013-01-16 Thread Josh Durgin
() [PATCH REPOST 2/3] rbd: open code rbd_calc_raw_layout() [PATCH REPOST 3/3] rbd: don't bother calculating file mapping We'll want to use similar methods later for fancier rbd striping with format 2 images, but that'll take more restructuring later anyway. This is fine for now. Reviewed-by: Josh

Re: [PATCH REPOST] rbd: kill ceph_osd_req_op-flags

2013-01-16 Thread Josh Durgin
On 01/04/2013 06:46 AM, Alex Elder wrote: The flags field of struct ceph_osd_req_op is never used, so just get rid of it. Signed-off-by: Alex Elder el...@inktank.com --- Reviewed-by: Josh Durgin josh.dur...@inktank.com include/linux/ceph/osd_client.h |1 - 1 file changed, 1 deletion

Re: [PATCH REPOST] rbd: use a common layout for each device

2013-01-16 Thread Josh Durgin
an assertion that was done when initializing the layout structures with code that catches and handles anything that would trigger the assertion as soon as it is identified. This precludes that (bad) condition from ever occurring. Signed-off-by: Alex Elder el...@inktank.com --- Reviewed-by: Josh Durgin

Re: [PATCH REPOST] rbd: combine rbd sync watch/unwatch functions

2013-01-16 Thread Josh Durgin
-by: Josh Durgin josh.dur...@inktank.com drivers/block/rbd.c | 81 +-- 1 file changed, 27 insertions(+), 54 deletions(-) diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c index 7c35608..c1e5f24 100644 --- a/drivers/block/rbd.c +++ b/drivers

  1   2   3   4   5   6   7   8   >