[PATCH 21/26] block: add BIO_HOLD_PAGES flag

2018-12-07 Thread Jens Axboe
-by: Jens Axboe --- block/bio.c | 6 -- include/linux/blk_types.h | 1 + 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/block/bio.c b/block/bio.c index 06760543ec81..3e45e5650265 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1636,7 +1636,8 @@ static void bio_dirty_fn

[PATCH 20/26] aio: batch aio_kiocb allocation

2018-12-07 Thread Jens Axboe
Similarly to how we use the state->ios_left to know how many references to get to a file, we can use it to allocate the aio_kiocb's we need in bulk. Signed-off-by: Jens Axboe --- fs/aio.c | 47 +++ 1 file changed, 39 insertions(+), 8 deleti

[PATCH 26/26] aio: add support for submission/completion rings

2018-12-07 Thread Jens Axboe
to enter the kernel to do IO. Sample application: http://brick.kernel.dk/snaps/aio-ring.c Signed-off-by: Jens Axboe --- arch/x86/entry/syscalls/syscall_64.tbl | 1 + fs/aio.c | 435 +++-- include/linux/syscalls.h | 4 +- include/uapi

[PATCH 25/26] aio: split old ring complete out from aio_complete()

2018-12-07 Thread Jens Axboe
Signed-off-by: Jens Axboe --- fs/aio.c | 17 - 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index fd323b3ba499..de48faeab0fd 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -1218,12 +1218,9 @@ static void aio_fill_event(struct io_event *ev, struct

[PATCH 17/26] fs: add fget_many() and fput_many()

2018-12-07 Thread Jens Axboe
for how many references to get on the file. Ditto fput_many(), which can drop an arbitrary number of references to a file. Signed-off-by: Jens Axboe --- fs/file.c| 15 ++- fs/file_table.c | 10 -- include/linux/file.h | 2 ++ include/linux/fs.h | 3 ++- 4

[PATCH 18/26] aio: use fget/fput_many() for file references

2018-12-07 Thread Jens Axboe
, hopefuly they are at least somewhat ordered. Could trivially be extended to cover multiple fds, if needed. On the completion side we do the same thing, except this is trivially done just locally in aio_iopoll_reap(). Signed-off-by: Jens Axboe --- fs/aio.c | 106

[PATCH 24/26] aio: add support for pre-mapped user IO buffers

2018-12-07 Thread Jens Axboe
change for how to utilize this feature: http://git.kernel.dk/cgit/fio/commit/?id=2041bd343da1c1e955253f62374588718c64f0f3 Signed-off-by: Jens Axboe --- fs/aio.c | 193 --- include/uapi/linux/aio_abi.h | 1 + 2 files changed, 177 insertions

[PATCH 19/26] aio: split iocb init from allocation

2018-12-07 Thread Jens Axboe
In preparation from having pre-allocated requests, that we then just need to initialize before use. Signed-off-by: Jens Axboe --- fs/aio.c | 17 +++-- 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index 3c07cc9cb11a..51c7159f09bf 100644 --- a/fs

[PATCH 23/26] fs: add support for mapping an ITER_BVEC for O_DIRECT

2018-12-07 Thread Jens Axboe
This adds support for sync/async O_DIRECT to make a bvec type iter for bdev access, as well as iomap. Signed-off-by: Jens Axboe --- fs/block_dev.c | 16 fs/iomap.c | 10 +++--- 2 files changed, 19 insertions(+), 7 deletions(-) diff --git a/fs/block_dev.c b/fs

[PATCH 22/26] block: implement bio helper to add iter bvec pages to bio

2018-12-07 Thread Jens Axboe
For an ITER_BVEC, we can just iterate the iov and add the pages to the bio directly. Signed-off-by: Jens Axboe --- block/bio.c | 27 +++ include/linux/bio.h | 1 + 2 files changed, 28 insertions(+) diff --git a/block/bio.c b/block/bio.c index 3e45e5650265

[PATCH 16/26] aio: add submission side request cache

2018-12-07 Thread Jens Axboe
-by: Jens Axboe --- fs/aio.c | 136 +-- 1 file changed, 113 insertions(+), 23 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index cc8763b395c1..5e840396f6d8 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -240,6 +240,21 @@ struct aio_kiocb

[PATCH 12/26] aio: abstract out io_event filler helper

2018-12-07 Thread Jens Axboe
Signed-off-by: Jens Axboe --- fs/aio.c | 14 ++ 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index 06c8bcc72496..173f1f79dc8f 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -1063,6 +1063,15 @@ static inline void iocb_put(struct aio_kiocb *iocb

[PATCH 14/26] aio: add support for having user mapped iocbs

2018-12-07 Thread Jens Axboe
?id=3c3168e91329c83880c91e5abc28b9d6b940fd95 Signed-off-by: Jens Axboe --- fs/aio.c | 126 +++ include/uapi/linux/aio_abi.h | 2 + 2 files changed, 116 insertions(+), 12 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index 26631d6872d2..bb6f07ca6940 100644 --- a/fs/ai

[PATCH 11/26] aio: split out iocb copy from io_submit_one()

2018-12-07 Thread Jens Axboe
In preparation of handing in iocbs in a different fashion as well. Also make it clear that the iocb being passed in isn't modified, by marking it const throughout. Signed-off-by: Jens Axboe --- fs/aio.c | 68 +++- 1 file changed, 38 insertions

[PATCH 08/26] aio: don't zero entire aio_kiocb aio_get_req()

2018-12-07 Thread Jens Axboe
It's 192 bytes, fairly substantial. Most items don't need to be cleared, especially not upfront. Clear the ones we do need to clear, and leave the other ones for setup when the iocb is prepared and submitted. Reviewed-by: Christoph Hellwig Signed-off-by: Jens Axboe --- fs/aio.c | 9

[PATCH 13/26] aio: add io_setup2() system call

2018-12-07 Thread Jens Axboe
This is just like io_setup(), except add a flags argument to let the caller control/define some of the io_context behavior. Outside of the flags, we add an iocb array and two user pointers for future use. Signed-off-by: Jens Axboe --- arch/x86/entry/syscalls/syscall_64.tbl | 1 + fs/aio.c

[PATCH 15/26] aio: support for IO polling

2018-12-07 Thread Jens Axboe
to support completion reaping from userspace by just looking at the ring. The application itself is the one that pulls completion entries. Signed-off-by: Jens Axboe --- fs/aio.c | 396 +++ include/uapi/linux/aio_abi.h | 3 + 2 files changed, 363

[PATCH 07/26] aio: separate out ring reservation from req allocation

2018-12-07 Thread Jens Axboe
From: Christoph Hellwig This is in preparation for certain types of IO not needing a ring reserveration. Signed-off-by: Christoph Hellwig Signed-off-by: Jens Axboe --- fs/aio.c | 30 +- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/fs/aio.c b/fs

[PATCH 05/26] iomap: wire up the iopoll method

2018-12-07 Thread Jens Axboe
Modified to use REQ_HIPRI_ASYNC for async polled IO. Signed-off-by: Jens Axboe --- fs/gfs2/file.c| 2 ++ fs/iomap.c| 47 +-- fs/xfs/xfs_file.c | 1 + include/linux/iomap.h | 1 + 4 files changed, 36 insertions(+), 15 deletions(-) diff

[PATCH 09/26] aio: only use blk plugs for > 2 depth submissions

2018-12-07 Thread Jens Axboe
Plugging is meant to optimize submission of a string of IOs, if we don't have more than 2 being submitted, don't bother setting up a plug. Reviewed-by: Christoph Hellwig Signed-off-by: Jens Axboe --- fs/aio.c | 18 ++ 1 file changed, 14 insertions(+), 4 deletions(-) diff --git

[PATCH 10/26] aio: use iocb_put() instead of open coding it

2018-12-07 Thread Jens Axboe
Replace the percpu_ref_put() + kmem_cache_free() with a call to iocb_put() instead. Reviewed-by: Christoph Hellwig Signed-off-by: Jens Axboe --- fs/aio.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index ed6c3914477a..cf93b92bfb1e 100644 --- a/fs

[PATCH 04/26] block: use REQ_HIPRI_ASYNC for non-sync polled IO

2018-12-07 Thread Jens Axboe
Tell the block layer if it's a sync or async polled request, so it can do the right thing. Signed-off-by: Jens Axboe --- fs/block_dev.c | 8 ++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/fs/block_dev.c b/fs/block_dev.c index 6de8d35f6e41..b8f574615792 100644 --- a/fs

[PATCH 06/26] aio: use assigned completion handler

2018-12-07 Thread Jens Axboe
We know this is a read/write request, but in preparation for having different kinds of those, ensure that we call the assigned handler instead of assuming it's aio_complete_rq(). Reviewed-by: Christoph Hellwig Signed-off-by: Jens Axboe --- fs/aio.c | 2 +- 1 file changed, 1 insertion(+), 1

[PATCH 01/26] fs: add an iopoll method to struct file_operations

2018-12-07 Thread Jens Axboe
to store the polling cookie. TODO: we can probably union ki_cookie with the existing hint and I/O priority fields to avoid struct kiocb growth. Reviewed-by: Johannes Thumshirn Signed-off-by: Christoph Hellwig Signed-off-by: Jens Axboe --- Documentation/filesystems/vfs.txt | 3 +++ include/linux

[PATCHSET v6] Support for polled aio (and more)

2018-12-07 Thread Jens Axboe
files changed, 1552 insertions(+), 197 deletions(-) -- Jens Axboe

[PATCH 02/26] block: add REQ_HIPRI_ASYNC

2018-12-07 Thread Jens Axboe
For the upcoming async polled IO, we can't sleep allocating requests. If we do, then we introduce a deadlock where the submitter already has async polled IO in-flight, but can't wait for them to complete since polled requests must be active found and reaped. Signed-off-by: Jens Axboe

[PATCH 03/26] block: wire up block device iopoll method

2018-12-07 Thread Jens Axboe
From: Christoph Hellwig Just call blk_poll on the iocb cookie, we can derive the block device from the inode trivially. Reviewed-by: Johannes Thumshirn Signed-off-by: Christoph Hellwig Signed-off-by: Jens Axboe --- fs/block_dev.c | 10 ++ 1 file changed, 10 insertions(+) diff --git

Re: [PATCHSET v5] Support for polled aio

2018-12-07 Thread Jens Axboe
On 12/7/18 3:00 PM, Jens Axboe wrote: > On 12/7/18 2:59 PM, Jens Axboe wrote: >> On 12/7/18 2:58 PM, Jens Axboe wrote: >>> On 12/7/18 12:35 PM, Jens Axboe wrote: >>>> On 12/7/18 12:34 PM, Jeff Moyer wrote: >>>>> Jens Axboe writes: >>>>

Re: [PATCHSET v5] Support for polled aio

2018-12-07 Thread Jens Axboe
On 12/7/18 2:59 PM, Jens Axboe wrote: > On 12/7/18 2:58 PM, Jens Axboe wrote: >> On 12/7/18 12:35 PM, Jens Axboe wrote: >>> On 12/7/18 12:34 PM, Jeff Moyer wrote: >>>> Jens Axboe writes: >>>> >>>>> BTW, quick guess is that it doesn't wor

Re: [PATCHSET v5] Support for polled aio

2018-12-07 Thread Jens Axboe
On 12/7/18 2:58 PM, Jens Axboe wrote: > On 12/7/18 12:35 PM, Jens Axboe wrote: >> On 12/7/18 12:34 PM, Jeff Moyer wrote: >>> Jens Axboe writes: >>> >>>> BTW, quick guess is that it doesn't work so well with fixed buffers, as >>>> tha

Re: [PATCHSET v5] Support for polled aio

2018-12-07 Thread Jens Axboe
On 12/7/18 12:35 PM, Jens Axboe wrote: > On 12/7/18 12:34 PM, Jeff Moyer wrote: >> Jens Axboe writes: >> >>> BTW, quick guess is that it doesn't work so well with fixed buffers, as that >>> hasn't been tested. You could try and remove IOCTX_FLAG_FIXEDBUFS

Re: [GIT PULL] first batch of nvme updates for 4.21

2018-12-07 Thread Jens Axboe
t; drivers/nvme/target/nvmet.h | 53 +++++--- > drivers/nvme/target/rdma.c| 2 +- > include/linux/nvme-fc-driver.h| 16 - > include/linux/nvme.h | 51 +-- > 18 files changed, 514 insertions(+), 287 deletions(-) Pulled, thanks. -- Jens Axboe

Re: [PATCHSET v5] Support for polled aio

2018-12-07 Thread Jens Axboe
On 12/7/18 12:34 PM, Jeff Moyer wrote: > Jens Axboe writes: > >> BTW, quick guess is that it doesn't work so well with fixed buffers, as that >> hasn't been tested. You could try and remove IOCTX_FLAG_FIXEDBUFS from the >> test program and see if that works. > >

Re: [PATCHSET v5] Support for polled aio

2018-12-07 Thread Jens Axboe
On 12/7/18 11:52 AM, Jens Axboe wrote: > On 12/7/18 11:48 AM, Jeff Moyer wrote: >> Hi, Jens, >> >> Jens Axboe writes: >> >>> You can also find the patches in my aio-poll branch: >>> >>> http://git.kernel.dk/cgit/linux-block/log/?h=aio-poll &g

Re: [PATCHSET v5] Support for polled aio

2018-12-07 Thread Jens Axboe
On 12/7/18 11:48 AM, Jeff Moyer wrote: > Hi, Jens, > > Jens Axboe writes: > >> You can also find the patches in my aio-poll branch: >> >> http://git.kernel.dk/cgit/linux-block/log/?h=aio-poll >> >> or by cloning: >> >> git://git.kernel.

[GIT PULL] Block fixes for 4.20-rc6

2018-12-07 Thread Jens Axboe
use after free James Smart (1): nvme: validate controller state before rescheduling keep alive Jens Axboe (2): blk-mq: punt failed direct issue to dispatch list Merge branch 'nvme-4.20' of git://git.infradead.org/nvme into for-linus Paolo Valente (1): block, bfq: fix

Re: [PATCH v3] blk-mq: punt failed direct issue to dispatch list

2018-12-07 Thread Jens Axboe
On 12/7/18 9:41 AM, Bart Van Assche wrote: > On Fri, 2018-12-07 at 09:35 -0700, Jens Axboe wrote: >> diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c >> index 29bfe8017a2d..9e5bda8800f8 100644 >> --- a/block/blk-mq-sched.c >> +++ b/block/blk-mq-sched.c &

Re: [PATCH v3] blk-mq: punt failed direct issue to dispatch list

2018-12-07 Thread Jens Axboe
On 12/7/18 9:24 AM, Jens Axboe wrote: > On 12/7/18 9:19 AM, Bart Van Assche wrote: >> On Thu, 2018-12-06 at 22:17 -0700, Jens Axboe wrote: >>> Instead of making special cases for what we can direct issue, and now >>> having to deal with DM solving the livelock w

Re: [PATCH v3] blk-mq: punt failed direct issue to dispatch list

2018-12-07 Thread Jens Axboe
On 12/7/18 9:19 AM, Bart Van Assche wrote: > On Thu, 2018-12-06 at 22:17 -0700, Jens Axboe wrote: >> Instead of making special cases for what we can direct issue, and now >> having to deal with DM solving the livelock while still retaining a BUSY >> condition feedback

Re: [GIT PULL] nvme fixes for 4.20

2018-12-07 Thread Jens Axboe
nvmet-rdma: fix response use after free > > James Smart (1): > nvme: validate controller state before rescheduling keep alive > > drivers/nvme/host/core.c | 10 +- > drivers/nvme/target/rdma.c | 3 ++- > 2 files changed, 11 insertions(+), 2 deletions(-) Pulled, thanks. -- Jens Axboe

Re: [PATCH V2] blk-mq: re-build queue map in case of kdump kernel

2018-12-06 Thread Jens Axboe
virtio_scsi > ip_tables > [4.489428] Dumping ftrace buffer: > [4.489939](ftrace buffer empty) > [4.490492] CR2: 0098 > [4.491052] ---[ end trace 03cd268ad5a86ff7 ]--- Works for me, tested various configs and stubbed out the kdump check. Thanks for fixing this, applied to 4.21. -- Jens Axboe

[PATCH v3] blk-mq: punt failed direct issue to dispatch list

2018-12-06 Thread Jens Axboe
uot;) Cc: sta...@vger.kernel.org Suggested-by: Ming Lei Reported-by: Bart Van Assche Signed-off-by: Jens Axboe --- I've thrown the initial hang test reported by Bart at it, works fine. My reproducer for the corruption case is also happy, as expected. I'm running blktests and xfstests on

Re: [PATCH v2] block/dm: fix handling of busy off direct dispatch path

2018-12-06 Thread Jens Axboe
On 12/6/18 9:18 PM, Mike Snitzer wrote: > On Thu, Dec 06 2018 at 11:06pm -0500, > Jens Axboe wrote: > >> On 12/6/18 8:54 PM, Mike Snitzer wrote: >>> On Thu, Dec 06 2018 at 9:49pm -0500, >>> Jens Axboe wrote: >>> >>>> After the direct d

Re: [PATCH v2] block/dm: fix handling of busy off direct dispatch path

2018-12-06 Thread Jens Axboe
On 12/6/18 8:54 PM, Mike Snitzer wrote: > On Thu, Dec 06 2018 at 9:49pm -0500, > Jens Axboe wrote: > >> After the direct dispatch corruption fix, we permanently disallow direct >> dispatch of non read/write requests. This works fine off the normal IO >> path, as th

Re: [PATCH] blk-mq: fix corruption with direct issue

2018-12-06 Thread Jens Axboe
prep next time again. Otherwise I fear the direct dispatch isn't going to be super useful, if a failed direct dispatch prevents future merging. This would be a lot less error prone as well for other cases. -- Jens Axboe

Re: [PATCH] blk-mq: re-build queue map in case of kdump kernel

2018-12-06 Thread Jens Axboe
q.c > @@ -38,6 +38,11 @@ > #include "blk-mq-sched.h" > #include "blk-rq-qos.h" > > +static inline bool blk_mq_kdump_kernel(void) > +{ > + return !!is_kdump_kernel(); > +} Let's drop the redundant !! here, and the wrapper? I would imagine the wrapper is handy for testing outside of kdump, but I don't think we should include it in the final. Otherwise this looks fine, I can test it here too. -- Jens Axboe

[PATCH v2] block/dm: fix handling of busy off direct dispatch path

2018-12-06 Thread Jens Axboe
don't over-dispatch to the lower level queue and mess up opportunities for merging on the DM queue. Fixes: ffe81d45322c ("blk-mq: fix corruption with direct issue") Reported-by: Bart Van Assche Cc: sta...@vger.kernel.org Signed-off-by: Jens Axboe --- This passes my testing as

Re: block: fix direct dispatch issue failure for clones

2018-12-06 Thread Jens Axboe
On 12/6/18 7:16 PM, Jens Axboe wrote: > On 12/6/18 7:06 PM, Jens Axboe wrote: >> On 12/6/18 6:58 PM, Mike Snitzer wrote: >>>> There is another way to fix this - still do the direct dispatch, but have >>>> dm track if it failed and do bypass insert in that case. I

Re: block: fix direct dispatch issue failure for clones

2018-12-06 Thread Jens Axboe
On 12/6/18 7:06 PM, Jens Axboe wrote: > On 12/6/18 6:58 PM, Mike Snitzer wrote: >>> There is another way to fix this - still do the direct dispatch, but have >>> dm track if it failed and do bypass insert in that case. I didn't want do >>> to that since it's

Re: [GIT PULL] Follow up block fix

2018-12-06 Thread Jens Axboe
On 12/6/18 5:33 PM, Jens Axboe wrote: > On 12/6/18 5:31 PM, Bart Van Assche wrote: >> On Thu, 2018-12-06 at 17:26 -0700, Jens Axboe wrote: >>> On 12/6/18 5:20 PM, Bart Van Assche wrote: >>>> Which branch does that tag correspond to? Even after having run git fetch >

Re: block: fix direct dispatch issue failure for clones

2018-12-06 Thread Jens Axboe
struct request *rq, bool force_insert); extern int blk_rq_append_bio(struct request *rq, struct bio **bio); extern void blk_delay_queue(struct request_queue *, unsigned long); extern void blk_queue_split(struct request_queue *, struct bio **); -- Jens Axboe

Re: [PATCH] block: fix direct dispatch issue failure for clones

2018-12-06 Thread Jens Axboe
On 12/6/18 6:22 PM, jianchao.wang wrote: > > > On 12/7/18 9:13 AM, Jens Axboe wrote: >> On 12/6/18 6:04 PM, jianchao.wang wrote: >>> >>> >>> On 12/7/18 6:20 AM, Jens Axboe wrote: >>>> After the direct dispatch corruption fix, we perma

Re: [PATCH] block: fix direct dispatch issue failure for clones

2018-12-06 Thread Jens Axboe
On 12/6/18 6:04 PM, jianchao.wang wrote: > > > On 12/7/18 6:20 AM, Jens Axboe wrote: >> After the direct dispatch corruption fix, we permanently disallow direct >> dispatch of non read/write requests. This works fine off the normal IO >> path, as they will be retried l

Re: [GIT PULL] Follow up block fix

2018-12-06 Thread Jens Axboe
On 12/6/18 5:31 PM, Bart Van Assche wrote: > On Thu, 2018-12-06 at 17:26 -0700, Jens Axboe wrote: >> On 12/6/18 5:20 PM, Bart Van Assche wrote: >>> Which branch does that tag correspond to? Even after having run git fetch >>> --tags I can't find that tag in your

Re: [GIT PULL] Follow up block fix

2018-12-06 Thread Jens Axboe
On 12/6/18 5:20 PM, Bart Van Assche wrote: > On Thu, 2018-12-06 at 16:59 -0700, Jens Axboe wrote: >> Just a single followup fix to the corruption fix from yesterday. We have >> an exported interface that does direct dispatch, with DM being the sole >> user of it. Change tha

[GIT PULL] Follow up block fix

2018-12-06 Thread Jens Axboe
. Additionally, it gets rid of any exported user of direct dispatch, and enables DM to drop any BUSY handling from using the interface. Please pull! git://git.kernel.dk/linux-block.git tags/for-linus-20181206 Jens Axboe (1): block

Re: block: fix direct dispatch issue failure for clones

2018-12-06 Thread Jens Axboe
On 12/6/18 3:28 PM, Mike Snitzer wrote: > On Thu, Dec 06 2018 at 5:20pm -0500, > Jens Axboe wrote: > >> After the direct dispatch corruption fix, we permanently disallow direct >> dispatch of non read/write requests. This works fine off the normal IO >> path, as th

Re: [PATCH] block: fix direct dispatch issue failure for clones

2018-12-06 Thread Jens Axboe
On 12/6/18 3:20 PM, Jens Axboe wrote: > After the direct dispatch corruption fix, we permanently disallow direct > dispatch of non read/write requests. This works fine off the normal IO > path, as they will be retried like any other failed direct dispatch

[PATCH] block: fix direct dispatch issue failure for clones

2018-12-06 Thread Jens Axboe
scheduler, which is what DM wants. Fixes: ffe81d45322c ("blk-mq: fix corruption with direct issue") Signed-off-by: Jens Axboe --- diff --git a/block/blk-core.c b/block/blk-core.c index deb56932f8c4..4c44e6fa0d08 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -2637,7 +2637,8 @@ bl

Re: for-next branch and blktests/srp

2018-12-06 Thread Jens Axboe
On 12/6/18 2:04 PM, Jens Axboe wrote: > On 12/6/18 1:56 PM, Bart Van Assche wrote: >> On Thu, 2018-12-06 at 08:47 -0800, Bart Van Assche wrote: >>> If I merge Jens' for-next branch with Linus' master branch, boot the >>> resulting kernel in a VM and run blktests/t

Re: for-next branch and blktests/srp

2018-12-06 Thread Jens Axboe
spin_lock_irqsave(q->queue_lock, flags); it works fine. Well, at least this regression is less serious, I'll bang out a fix for it and ensure we make -rc6. I'm guessing it's the bypassin of non-read/write, which your top of dispatch also shows to be a non-read/write. But there should be no new failure case here that wasn't possible before, only it's easier to hit now. -- Jens Axboe

Re: [PATCH 08/26] aio: don't zero entire aio_kiocb aio_get_req()

2018-12-06 Thread Jens Axboe
On 12/6/18 12:38 PM, Jeff Moyer wrote: > Jens Axboe writes: > >> On 12/6/18 12:27 PM, Jeff Moyer wrote: >>> Jens Axboe writes: >>> >>>> It's 192 bytes, fairly substantial. Most items don't need to be cleared, >>>> especially not

Re: [PATCH 09/26] aio: only use blk plugs for > 2 depth submissions

2018-12-06 Thread Jens Axboe
On 12/6/18 12:29 PM, Jeff Moyer wrote: > Jens Axboe writes: > >> Plugging is meant to optimize submission of a string of IOs, if we don't >> have more than 2 being submitted, don't bother setting up a plug. > > Is there really that much overhead in blk_{start|fini

Re: [PATCH 08/26] aio: don't zero entire aio_kiocb aio_get_req()

2018-12-06 Thread Jens Axboe
On 12/6/18 12:27 PM, Jeff Moyer wrote: > Jens Axboe writes: > >> It's 192 bytes, fairly substantial. Most items don't need to be cleared, >> especially not upfront. Clear the ones we do need to clear, and leave >> the other ones for setup when the iocb is prepared

Re: [PATCH 03/26] block: wire up block device iopoll method

2018-12-06 Thread Jens Axboe
On 12/6/18 12:15 PM, Christoph Hellwig wrote: > On Thu, Dec 06, 2018 at 12:14:29PM -0700, Jens Axboe wrote: >> On 12/6/18 12:11 PM, Jeff Moyer wrote: >>> Jens Axboe writes: >>> >>>> From: Christoph Hellwig >>>> >>>> Just call blk_po

Re: [PATCH 03/26] block: wire up block device iopoll method

2018-12-06 Thread Jens Axboe
On 12/6/18 12:11 PM, Jeff Moyer wrote: > Jens Axboe writes: > >> From: Christoph Hellwig >> >> Just call blk_poll on the iocb cookie, we can derive the block device >> from the inode trivially. > > Does this work for multi-device file systems? It should

Re: for-next branch and blktests/srp

2018-12-06 Thread Jens Axboe
On 12/6/18 11:10 AM, Bart Van Assche wrote: > On Thu, 2018-12-06 at 10:02 -0800, Bart Van Assche wrote: >> On Thu, 2018-12-06 at 10:48 -0700, Jens Axboe wrote: >>> which would result in a non-zero exit, which should be expected for this >>> test? >> >> Test s

Re: for-next branch and blktests/srp

2018-12-06 Thread Jens Axboe
On 12/6/18 11:02 AM, Bart Van Assche wrote: > On Thu, 2018-12-06 at 10:48 -0700, Jens Axboe wrote: >> which would result in a non-zero exit, which should be expected for this >> test? > > Hi Jens, > > Test srp/002 simulates network failures while running fio

Re: for-next branch and blktests/srp

2018-12-06 Thread Jens Axboe
On 12/6/18 10:28 AM, Jens Axboe wrote: > On 12/6/18 10:19 AM, Bart Van Assche wrote: >> On Thu, 2018-12-06 at 10:00 -0700, Jens Axboe wrote: >>> On 12/6/18 9:47 AM, Bart Van Assche wrote: >>>> If I merge Jens' for-next branch with Linus' master branch, boot the

Re: for-next branch and blktests/srp

2018-12-06 Thread Jens Axboe
On 12/6/18 10:19 AM, Bart Van Assche wrote: > On Thu, 2018-12-06 at 10:00 -0700, Jens Axboe wrote: >> On 12/6/18 9:47 AM, Bart Van Assche wrote: >>> If I merge Jens' for-next branch with Linus' master branch, boot the >>> resulting kernel in a VM and run blktests/t

Re: for-next branch and blktests/srp

2018-12-06 Thread Jens Axboe
is is a regression. The following appears in the system log if > I run that test: You are running that test on a dm device? Can you shed some light on how that dm device is setup? -- Jens Axboe

Re: for-next branch and throttling

2018-12-06 Thread Jens Axboe
ainted 4.20.0-rc5-dbg+ #1 > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 > 04/01/2014 > RIP: 0010:blk_throtl_bio+0xc00/0x1120 It's some over-zealous rcu removal, simple fix is coming. CC Dennis. -- Jens Axboe

[GIT PULL] Block fixes for 4.20-rc6

2018-12-05 Thread Jens Axboe
Jens Axboe (1): blk-mq: fix corruption with direct issue Juha-Matti Tilli (1): libata: whitelist all SAMSUNG MZ7KM* solid-state disks block/blk-mq.c| 26 +- drivers/ata/libata-core.c

Re: [PATCH] blk-mq: fix corruption with direct issue

2018-12-05 Thread Jens Axboe
On 12/5/18 12:09 PM, Guenter Roeck wrote: > On Wed, Dec 05, 2018 at 10:59:21AM -0700, Jens Axboe wrote: > [ ... ] >> >>> Also, it seems to me that even with this problem fixed, blk-mq may not >>> be ready for primetime after all. With that in mind, maybe commit

Re: [PATCH] blk-mq: fix corruption with direct issue

2018-12-05 Thread Jens Axboe
On 12/5/18 10:55 AM, Guenter Roeck wrote: > On Tue, Dec 04, 2018 at 07:25:05PM -0700, Jens Axboe wrote: >> On 12/4/18 6:38 PM, Guenter Roeck wrote: >>> On Tue, Dec 04, 2018 at 03:47:46PM -0700, Jens Axboe wrote: >>>> If we attempt a direct issue to a SCSI device, and

Re: [PATCH 0/3] Unify the throttling code for wbt and io-latency

2018-12-05 Thread Jens Axboe
this with my blktests test. Thanks, Applies fine to for-4.21/block, which is what I care about. Looks good to me, applied, thanks. -- Jens Axboe

Re: [PATCH] blk-mq: fix corruption with direct issue

2018-12-05 Thread Jens Axboe
On 12/5/18 7:41 AM, Christoph Hellwig wrote: > On Tue, Dec 04, 2018 at 03:47:46PM -0700, Jens Axboe wrote: >> If we attempt a direct issue to a SCSI device, and it returns BUSY, then >> we queue the request up normally. However, the SCSI layer may have >> alread

Re: [PATCH 15/26] aio: support for IO polling

2018-12-05 Thread Jens Axboe
!*nr_events || !need_resched()) { >> +int tmin = 0; >> + >> +if (*nr_events < min_nr) >> +tmin = min_nr - *nr_events; > > Otherwise, shouldn't the function just return 0? > > Or perhaps you explicitly want to go through the tmin==0 case > in aio_iopoll_getevents if *nr_events == min_nr (or min_nr==0)? No, we need to go through poll, if min_nr == 0, but only if min_nr == 0 and !nr_events. But we only get here for nr_events != 0, so should be fine as-is. Thanks for your review, very useful! I'll make the above changes right now. -- Jens Axboe

Re: [PATCH] blk-mq: fix corruption with direct issue

2018-12-04 Thread Jens Axboe
On 12/4/18 8:03 PM, Ming Lei wrote: > On Wed, Dec 05, 2018 at 10:58:02AM +0800, Ming Lei wrote: >> On Tue, Dec 04, 2018 at 07:30:24PM -0700, Jens Axboe wrote: >>> On 12/4/18 7:27 PM, Ming Lei wrote: >>>> On Tue, Dec 04, 2018 at 07:16:11PM -0700, Jens Axboe wrote: >

Re: [PATCH] blk-mq: fix corruption with direct issue

2018-12-04 Thread Jens Axboe
On 12/4/18 7:58 PM, Ming Lei wrote: > On Tue, Dec 04, 2018 at 07:30:24PM -0700, Jens Axboe wrote: >> On 12/4/18 7:27 PM, Ming Lei wrote: >>> On Tue, Dec 04, 2018 at 07:16:11PM -0700, Jens Axboe wrote: >>>> On 12/4/18 6:37 PM, Ming Lei wrote: >>>>> O

Re: [PATCH] blk-mq: fix corruption with direct issue

2018-12-04 Thread Jens Axboe
On 12/4/18 7:27 PM, Ming Lei wrote: > On Tue, Dec 04, 2018 at 07:16:11PM -0700, Jens Axboe wrote: >> On 12/4/18 6:37 PM, Ming Lei wrote: >>> On Tue, Dec 04, 2018 at 03:47:46PM -0700, Jens Axboe wrote: >>>> If we attempt a direct issue to a SCSI device, and it re

Re: [PATCH] blk-mq: fix corruption with direct issue

2018-12-04 Thread Jens Axboe
On 12/4/18 6:38 PM, Guenter Roeck wrote: > On Tue, Dec 04, 2018 at 03:47:46PM -0700, Jens Axboe wrote: >> If we attempt a direct issue to a SCSI device, and it returns BUSY, then >> we queue the request up normally. However, the SCSI layer may have >> alread

Re: [PATCH] blk-mq: fix corruption with direct issue

2018-12-04 Thread Jens Axboe
On 12/4/18 7:16 PM, Jens Axboe wrote: > On 12/4/18 6:37 PM, Ming Lei wrote: >> On Tue, Dec 04, 2018 at 03:47:46PM -0700, Jens Axboe wrote: >>> If we attempt a direct issue to a SCSI device, and it returns BUSY, then >>> we queue the request up normally. However, the SCSI

Re: [PATCH] blk-mq: fix corruption with direct issue

2018-12-04 Thread Jens Axboe
On 12/4/18 6:37 PM, Ming Lei wrote: > On Tue, Dec 04, 2018 at 03:47:46PM -0700, Jens Axboe wrote: >> If we attempt a direct issue to a SCSI device, and it returns BUSY, then >> we queue the request up normally. However, the SCSI layer may have >> already setup SG tables et

[PATCH 26/26] aio: add support for submission/completion rings

2018-12-04 Thread Jens Axboe
to enter the kernel to do IO. Sample application: http://brick.kernel.dk/snaps/aio-ring.c Signed-off-by: Jens Axboe --- arch/x86/entry/syscalls/syscall_64.tbl | 1 + fs/aio.c | 312 +++-- include/linux/syscalls.h | 4 +- include/uapi

[PATCH 25/26] aio: split old ring complete out from aio_complete()

2018-12-04 Thread Jens Axboe
Signed-off-by: Jens Axboe --- fs/aio.c | 17 - 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index 1c8a8bb631a9..39aaffd6d436 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -1218,12 +1218,9 @@ static void aio_fill_event(struct io_event *ev, struct

[PATCH 23/26] fs: add support for mapping an ITER_BVEC for O_DIRECT

2018-12-04 Thread Jens Axboe
This adds support for sync/async O_DIRECT to make a bvec type iter for bdev access, as well as iomap. Signed-off-by: Jens Axboe --- fs/block_dev.c | 16 fs/iomap.c | 5 - 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/fs/block_dev.c b/fs/block_dev.c

[PATCH 24/26] aio: add support for pre-mapped user IO buffers

2018-12-04 Thread Jens Axboe
change for how to utilize this feature: http://git.kernel.dk/cgit/fio/commit/?id=2041bd343da1c1e955253f62374588718c64f0f3 Signed-off-by: Jens Axboe --- fs/aio.c | 193 --- include/uapi/linux/aio_abi.h | 1 + 2 files changed, 177 insertions

[PATCH 21/26] block: add BIO_HOLD_PAGES flag

2018-12-04 Thread Jens Axboe
-by: Jens Axboe --- block/bio.c | 6 -- include/linux/blk_types.h | 1 + 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/block/bio.c b/block/bio.c index 03895cc0d74a..ab174bce5436 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1635,7 +1635,8 @@ static void bio_dirty_fn

[PATCH 19/26] aio: split iocb init from allocation

2018-12-04 Thread Jens Axboe
In preparation from having pre-allocated requests, that we then just need to initialize before use. Signed-off-by: Jens Axboe --- fs/aio.c | 17 +++-- 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index 634b540b0c92..416bb2e365e0 100644 --- a/fs

[PATCH 04/26] block: use REQ_HIPRI_ASYNC for non-sync polled IO

2018-12-04 Thread Jens Axboe
Tell the block layer if it's a sync or async polled request, so it can do the right thing. Signed-off-by: Jens Axboe --- fs/block_dev.c | 8 ++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/fs/block_dev.c b/fs/block_dev.c index 6de8d35f6e41..b8f574615792 100644 --- a/fs

[PATCH 15/26] aio: support for IO polling

2018-12-04 Thread Jens Axboe
to support completion reaping from userspace by just looking at the ring. The application itself is the one that pulls completion entries. Signed-off-by: Jens Axboe --- fs/aio.c | 393 +++ include/uapi/linux/aio_abi.h | 3 + 2 files changed, 360

[PATCH 03/26] block: wire up block device iopoll method

2018-12-04 Thread Jens Axboe
From: Christoph Hellwig Just call blk_poll on the iocb cookie, we can derive the block device from the inode trivially. Reviewed-by: Johannes Thumshirn Signed-off-by: Christoph Hellwig Signed-off-by: Jens Axboe --- fs/block_dev.c | 10 ++ 1 file changed, 10 insertions(+) diff --git

[PATCH 02/26] block: add REQ_HIPRI_ASYNC

2018-12-04 Thread Jens Axboe
For the upcoming async polled IO, we can't sleep allocating requests. If we do, then we introduce a deadlock where the submitter already has async polled IO in-flight, but can't wait for them to complete since polled requests must be active found and reaped. Signed-off-by: Jens Axboe

[PATCH 20/26] aio: batch aio_kiocb allocation

2018-12-04 Thread Jens Axboe
Similarly to how we use the state->ios_left to know how many references to get to a file, we can use it to allocate the aio_kiocb's we need in bulk. Signed-off-by: Jens Axboe --- fs/aio.c | 47 +++ 1 file changed, 39 insertions(+), 8 deleti

[PATCH 13/26] aio: add io_setup2() system call

2018-12-04 Thread Jens Axboe
This is just like io_setup(), except add a flags argument to let the caller control/define some of the io_context behavior. Outside of the flags, we add an iocb array and two user pointers for future use. Signed-off-by: Jens Axboe --- arch/x86/entry/syscalls/syscall_64.tbl | 1 + fs/aio.c

[PATCH 11/26] aio: split out iocb copy from io_submit_one()

2018-12-04 Thread Jens Axboe
In preparation of handing in iocbs in a different fashion as well. Also make it clear that the iocb being passed in isn't modified, by marking it const throughout. Signed-off-by: Jens Axboe --- fs/aio.c | 68 +++- 1 file changed, 38 insertions

[PATCH 14/26] aio: add support for having user mapped iocbs

2018-12-04 Thread Jens Axboe
?id=3c3168e91329c83880c91e5abc28b9d6b940fd95 Signed-off-by: Jens Axboe --- fs/aio.c | 126 +++ include/uapi/linux/aio_abi.h | 2 + 2 files changed, 116 insertions(+), 12 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index 26631d6872d2..bb6f07ca6940 100644 --- a/fs/ai

[PATCH 16/26] aio: add submission side request cache

2018-12-04 Thread Jens Axboe
-by: Jens Axboe --- fs/aio.c | 136 +-- 1 file changed, 113 insertions(+), 23 deletions(-) diff --git a/fs/aio.c b/fs/aio.c index 5d34317c2929..ae0805dc814c 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -240,6 +240,21 @@ struct aio_kiocb

[PATCH 01/26] fs: add an iopoll method to struct file_operations

2018-12-04 Thread Jens Axboe
to store the polling cookie. TODO: we can probably union ki_cookie with the existing hint and I/O priority fields to avoid struct kiocb growth. Reviewed-by: Johannes Thumshirn Signed-off-by: Christoph Hellwig Signed-off-by: Jens Axboe --- Documentation/filesystems/vfs.txt | 3 +++ include/linux

  1   2   3   4   5   6   7   8   9   10   >