Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Sun, 2018-05-20 at 18:17 -0400, Kent Overstreet wrote: > On Fri, May 18, 2018 at 03:12:27PM +, Bart Van Assche wrote: > > On Fri, 2018-05-18 at 05:06 -0400, Kent Overstreet wrote: > > > On Thu, May 17, 2018 at 08:54:57PM +, Bart Van Assche wrote: > > > > With Jens' latest for-next branch I hit the kernel warning shown below. > > > > Can > > > > you have a look? > > > > > > Any hints on how to reproduce it? > > > > Sure. This is how I triggered it: > > * Clone https://github.com/bvanassche/srp-test. > > * Follow the instructions in README.md. > > * Run srp-test/run_tests -c -r 10 > > Can you bisect it? I don't have infiniband hardware handy... Hello Kent, The bisect I ran reported the following commit as the first faulty commit: block: Add warning for bi_next not NULL in bio_endio() Bart.
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Mon, 2018-05-21 at 11:37 -0700, Omar Sandoval wrote: > Have you made any progress in porting srp-test to blktests so we don't > have to have this conversation again? Hello Omar, Porting the srp-test software to the blktests framework is still high on my to-do list. I will start working on this as soon as the work on adding SMR support for fio has been completed. Bart.
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Mon, May 21, 2018 at 03:11:08PM +, Bart Van Assche wrote:
> On Sun, 2018-05-20 at 19:58 -0400, Kent Overstreet wrote:
> > On Sun, May 20, 2018 at 11:40:45PM +, Bart Van Assche wrote:
> > > On Sun, 2018-05-20 at 19:21 -0400, Kent Overstreet wrote:
> > > > I really have better things to do than debug someone else's tests...
> > > > [ ... ]
> > > > ../run_tests: line 65: cd: /lib/modules/4.16.0+/kernel/block: No such
> > > > file or directory
> > >
> > > Kernel v4.16 is too old to run these tests. The srp-test script needs the
> > > following commit that went upstream in kernel v4.17-rc1:
> > >
> > > 63cf1a902c9d ("IB/srpt: Add RDMA/CM support")
> >
> > Same output on Jens' for-next branch.
>
> Others have been able to run the srp-test software with the instructions
> provided earlier in this e-mail thread. Can you share the kernel messages from
> around the time the test was run (dmesg, /var/log/messages or
> /var/log/syslog)?
>
> Thanks,
>
> Bart.
Bart,
Have you made any progress in porting srp-test to blktests so we don't
have to have this conversation again?
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Sun, 2018-05-20 at 19:58 -0400, Kent Overstreet wrote:
> On Sun, May 20, 2018 at 11:40:45PM +, Bart Van Assche wrote:
> > On Sun, 2018-05-20 at 19:21 -0400, Kent Overstreet wrote:
> > > I really have better things to do than debug someone else's tests...
> > > [ ... ]
> > > ../run_tests: line 65: cd: /lib/modules/4.16.0+/kernel/block: No such
> > > file or directory
> >
> > Kernel v4.16 is too old to run these tests. The srp-test script needs the
> > following commit that went upstream in kernel v4.17-rc1:
> >
> > 63cf1a902c9d ("IB/srpt: Add RDMA/CM support")
>
> Same output on Jens' for-next branch.
Others have been able to run the srp-test software with the instructions
provided earlier in this e-mail thread. Can you share the kernel messages from
around the time the test was run (dmesg, /var/log/messages or /var/log/syslog)?
Thanks,
Bart.
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Sun, May 20, 2018 at 11:40:45PM +, Bart Van Assche wrote:
> On Sun, 2018-05-20 at 19:21 -0400, Kent Overstreet wrote:
> > I really have better things to do than debug someone else's tests...
> > [ ... ]
> > ../run_tests: line 65: cd: /lib/modules/4.16.0+/kernel/block: No such file
> > or directory
>
> Kernel v4.16 is too old to run these tests. The srp-test script needs the
> following commit that went upstream in kernel v4.17-rc1:
>
> 63cf1a902c9d ("IB/srpt: Add RDMA/CM support")
Same output on Jens' for-next branch.
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Sun, 2018-05-20 at 19:21 -0400, Kent Overstreet wrote:
> I really have better things to do than debug someone else's tests...
> [ ... ]
> ../run_tests: line 65: cd: /lib/modules/4.16.0+/kernel/block: No such file or
> directory
Kernel v4.16 is too old to run these tests. The srp-test script needs the
following commit that went upstream in kernel v4.17-rc1:
63cf1a902c9d ("IB/srpt: Add RDMA/CM support")
Bart.
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Sun, May 20, 2018 at 10:35:29PM +, Bart Van Assche wrote: > On Sun, 2018-05-20 at 18:31 -0400, Kent Overstreet wrote: > > On Sun, May 20, 2018 at 10:19:13PM +, Bart Van Assche wrote: > > > On Sun, 2018-05-20 at 18:17 -0400, Kent Overstreet wrote: > > > > On Fri, May 18, 2018 at 03:12:27PM +, Bart Van Assche wrote: > > > > > On Fri, 2018-05-18 at 05:06 -0400, Kent Overstreet wrote: > > > > > > On Thu, May 17, 2018 at 08:54:57PM +, Bart Van Assche wrote: > > > > > > > With Jens' latest for-next branch I hit the kernel warning shown > > > > > > > below. Can > > > > > > > you have a look? > > > > > > > > > > > > Any hints on how to reproduce it? > > > > > > > > > > Sure. This is how I triggered it: > > > > > * Clone https://github.com/bvanassche/srp-test. > > > > > * Follow the instructions in README.md. > > > > > * Run srp-test/run_tests -c -r 10 > > > > > > > > Can you bisect it? I don't have infiniband hardware handy... > > > > > > Hello Kent, > > > > > > Have you noticed that the test I described uses the rdma_rxe driver and > > > hence that > > > no InfiniBand hardware is needed to run that test? > > > > No, I'm not terribly familiar with infiniband stuff > > > > Do you have some sort of self contained test/qemu recipe? I would really > > rather > > not have to figure out how to configure multipath, and infiniband, and I'm > > not > > even sure what else is needed based on that readme... > > Hello Kent, > > Please have another look at the srp-test README. The instructions in that > document > are easy to follow. No multipath nor any InfiniBand knowledge is required. > The test > even can be run in a virtual machine in case you would be worried about > potential > impact of the test on the rest of the system. I really have better things to do than debug someone else's tests... Restarting multipath-tools (via systemctl): multipath-tools.service. multipathd> reconfigure ok multipathd> make -C discontiguous-io discontiguous-io make[1]: Entering directory '/host/home/kent/ktest/tests/srp-test/discontiguous-io' make[1]: 'discontiguous-io' is up to date. make[1]: Leaving directory '/host/home/kent/ktest/tests/srp-test/discontiguous-io' Unloaded the ib_srpt kernel module Unloaded the rdma_rxe kernel module ../run_tests: line 65: cd: /lib/modules/4.16.0+/kernel/block: No such file or directory Zero-initializing /dev/ram0 ... done Zero-initializing /dev/ram1 ... done Unable to load target_core_pscsi Unable to load target_core_user Configured SRP target driver Running test ../tests/01 ... Unloaded the ib_srp kernel module SRP login failed Test ../tests/01 failed Running test ../tests/02-mq ... Test file I/O on top of multipath concurrently with logout and login (10 min; mq) Unloaded the ib_srp kernel module SRP login failed Test ../tests/02-mq failed Running test ../tests/02-sq ... Test file I/O on top of multipath concurrently with logout and login (10 min; sq) Unloaded the ib_srp kernel module SRP login failed Test ../tests/02-sq failed Running test ../tests/02-sq-on-mq ... Test file I/O on top of multipath concurrently with logout and login (10 min; sq-on-mq) Unloaded the ib_srp kernel module SRP login failed Test ../tests/02-sq-on-mq failed Running test ../tests/03-4M ... Test direct I/O with large transfer sizes and cmd_sg_entries=255 Unloaded the ib_srp kernel module SRP login failed Test ../tests/03-4M failed Running test ../tests/03-8M ... Test direct I/O with large transfer sizes and cmd_sg_entries=255 Unloaded the ib_srp kernel module SRP login failed Test ../tests/03-8M failed Running test ../tests/04-4M ... Test direct I/O with large transfer sizes and cmd_sg_entries=1 Unloaded the ib_srp kernel module SRP login failed Test ../tests/04-4M failed Running test ../tests/04-8M ... Test direct I/O with large transfer sizes and cmd_sg_entries=1 Unloaded the ib_srp kernel module SRP login failed Test ../tests/04-8M failed Running test ../tests/05-4M ... Test buffered I/O with large transfer sizes and cmd_sg_entries=255 Unloaded the ib_srp kernel module SRP login failed Test ../tests/05-4M failed Running test ../tests/05-8M ... Test buffered I/O with large transfer sizes and cmd_sg_entries=255 Unloaded the ib_srp kernel module SRP login failed Test ../tests/05-8M failed Running test ../tests/06 ... Test block I/O on top of multipath concurrently with logout and login (10 min) Unloaded the ib_srp kernel module SRP login failed Test ../tests/06 failed 0 tests succeeded and 11 tests failed Unloaded the ib_srpt kernel module Unloaded the rdma_rxe kernel module = PASSED srp in 18s
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Sun, 2018-05-20 at 19:00 -0400, Kent Overstreet wrote:
> On Sun, May 20, 2018 at 10:35:29PM +, Bart Van Assche wrote:
> > On Sun, 2018-05-20 at 18:31 -0400, Kent Overstreet wrote:
> > > On Sun, May 20, 2018 at 10:19:13PM +, Bart Van Assche wrote:
> > > > On Sun, 2018-05-20 at 18:17 -0400, Kent Overstreet wrote:
> > > > > On Fri, May 18, 2018 at 03:12:27PM +, Bart Van Assche wrote:
> > > > > > On Fri, 2018-05-18 at 05:06 -0400, Kent Overstreet wrote:
> > > > > > > On Thu, May 17, 2018 at 08:54:57PM +, Bart Van Assche wrote:
> > > > > > > > With Jens' latest for-next branch I hit the kernel warning
> > > > > > > > shown below. Can
> > > > > > > > you have a look?
> > > > > > >
> > > > > > > Any hints on how to reproduce it?
> > > > > >
> > > > > > Sure. This is how I triggered it:
> > > > > > * Clone https://github.com/bvanassche/srp-test.
> > > > > > * Follow the instructions in README.md.
> > > > > > * Run srp-test/run_tests -c -r 10
> > > > >
> > > > > Can you bisect it? I don't have infiniband hardware handy...
> > > >
> > > > Hello Kent,
> > > >
> > > > Have you noticed that the test I described uses the rdma_rxe driver and
> > > > hence that
> > > > no InfiniBand hardware is needed to run that test?
> > >
> > > No, I'm not terribly familiar with infiniband stuff
> > >
> > > Do you have some sort of self contained test/qemu recipe? I would really
> > > rather
> > > not have to figure out how to configure multipath, and infiniband, and
> > > I'm not
> > > even sure what else is needed based on that readme...
> >
> > Please have another look at the srp-test README. The instructions in that
> > document
> > are easy to follow. No multipath nor any InfiniBand knowledge is required.
> > The test
> > even can be run in a virtual machine in case you would be worried about
> > potential
> > impact of the test on the rest of the system.
>
> Your readme refers to kernel config options that don't even exist in the
> current
> kernel
That's probably a misunderstanding of your side. From a quick look it seems
like all
config symbols mentioned in the srp-test README.md exist in kernel v4.17-rc6.
The
following command does not report any non-existing Kconfig symbols:
$ cd linux-kernel
$ sed -n 's/^\* CONFIG_//p' ~/software/infiniband/srp-test/README.md |
while read f; do { git grep -lw "$f" | grep -q Kconfig; } || echo $f; done
Bart.
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Sun, May 20, 2018 at 10:35:29PM +, Bart Van Assche wrote: > On Sun, 2018-05-20 at 18:31 -0400, Kent Overstreet wrote: > > On Sun, May 20, 2018 at 10:19:13PM +, Bart Van Assche wrote: > > > On Sun, 2018-05-20 at 18:17 -0400, Kent Overstreet wrote: > > > > On Fri, May 18, 2018 at 03:12:27PM +, Bart Van Assche wrote: > > > > > On Fri, 2018-05-18 at 05:06 -0400, Kent Overstreet wrote: > > > > > > On Thu, May 17, 2018 at 08:54:57PM +, Bart Van Assche wrote: > > > > > > > With Jens' latest for-next branch I hit the kernel warning shown > > > > > > > below. Can > > > > > > > you have a look? > > > > > > > > > > > > Any hints on how to reproduce it? > > > > > > > > > > Sure. This is how I triggered it: > > > > > * Clone https://github.com/bvanassche/srp-test. > > > > > * Follow the instructions in README.md. > > > > > * Run srp-test/run_tests -c -r 10 > > > > > > > > Can you bisect it? I don't have infiniband hardware handy... > > > > > > Hello Kent, > > > > > > Have you noticed that the test I described uses the rdma_rxe driver and > > > hence that > > > no InfiniBand hardware is needed to run that test? > > > > No, I'm not terribly familiar with infiniband stuff > > > > Do you have some sort of self contained test/qemu recipe? I would really > > rather > > not have to figure out how to configure multipath, and infiniband, and I'm > > not > > even sure what else is needed based on that readme... > > Hello Kent, > > Please have another look at the srp-test README. The instructions in that > document > are easy to follow. No multipath nor any InfiniBand knowledge is required. > The test > even can be run in a virtual machine in case you would be worried about > potential > impact of the test on the rest of the system. Your readme refers to kernel config options that don't even exist in the current kernel
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Sun, 2018-05-20 at 18:31 -0400, Kent Overstreet wrote: > On Sun, May 20, 2018 at 10:19:13PM +, Bart Van Assche wrote: > > On Sun, 2018-05-20 at 18:17 -0400, Kent Overstreet wrote: > > > On Fri, May 18, 2018 at 03:12:27PM +, Bart Van Assche wrote: > > > > On Fri, 2018-05-18 at 05:06 -0400, Kent Overstreet wrote: > > > > > On Thu, May 17, 2018 at 08:54:57PM +, Bart Van Assche wrote: > > > > > > With Jens' latest for-next branch I hit the kernel warning shown > > > > > > below. Can > > > > > > you have a look? > > > > > > > > > > Any hints on how to reproduce it? > > > > > > > > Sure. This is how I triggered it: > > > > * Clone https://github.com/bvanassche/srp-test. > > > > * Follow the instructions in README.md. > > > > * Run srp-test/run_tests -c -r 10 > > > > > > Can you bisect it? I don't have infiniband hardware handy... > > > > Hello Kent, > > > > Have you noticed that the test I described uses the rdma_rxe driver and > > hence that > > no InfiniBand hardware is needed to run that test? > > No, I'm not terribly familiar with infiniband stuff > > Do you have some sort of self contained test/qemu recipe? I would really > rather > not have to figure out how to configure multipath, and infiniband, and I'm not > even sure what else is needed based on that readme... Hello Kent, Please have another look at the srp-test README. The instructions in that document are easy to follow. No multipath nor any InfiniBand knowledge is required. The test even can be run in a virtual machine in case you would be worried about potential impact of the test on the rest of the system. Bart.
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Sun, May 20, 2018 at 10:19:13PM +, Bart Van Assche wrote: > On Sun, 2018-05-20 at 18:17 -0400, Kent Overstreet wrote: > > On Fri, May 18, 2018 at 03:12:27PM +, Bart Van Assche wrote: > > > On Fri, 2018-05-18 at 05:06 -0400, Kent Overstreet wrote: > > > > On Thu, May 17, 2018 at 08:54:57PM +, Bart Van Assche wrote: > > > > > With Jens' latest for-next branch I hit the kernel warning shown > > > > > below. Can > > > > > you have a look? > > > > > > > > Any hints on how to reproduce it? > > > > > > Sure. This is how I triggered it: > > > * Clone https://github.com/bvanassche/srp-test. > > > * Follow the instructions in README.md. > > > * Run srp-test/run_tests -c -r 10 > > > > Can you bisect it? I don't have infiniband hardware handy... > > Hello Kent, > > Have you noticed that the test I described uses the rdma_rxe driver and hence > that > no InfiniBand hardware is needed to run that test? No, I'm not terribly familiar with infiniband stuff Do you have some sort of self contained test/qemu recipe? I would really rather not have to figure out how to configure multipath, and infiniband, and I'm not even sure what else is needed based on that readme...
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Sun, 2018-05-20 at 18:17 -0400, Kent Overstreet wrote: > On Fri, May 18, 2018 at 03:12:27PM +, Bart Van Assche wrote: > > On Fri, 2018-05-18 at 05:06 -0400, Kent Overstreet wrote: > > > On Thu, May 17, 2018 at 08:54:57PM +, Bart Van Assche wrote: > > > > With Jens' latest for-next branch I hit the kernel warning shown below. > > > > Can > > > > you have a look? > > > > > > Any hints on how to reproduce it? > > > > Sure. This is how I triggered it: > > * Clone https://github.com/bvanassche/srp-test. > > * Follow the instructions in README.md. > > * Run srp-test/run_tests -c -r 10 > > Can you bisect it? I don't have infiniband hardware handy... Hello Kent, Have you noticed that the test I described uses the rdma_rxe driver and hence that no InfiniBand hardware is needed to run that test? Thanks, Bart.
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Fri, May 18, 2018 at 03:12:27PM +, Bart Van Assche wrote: > On Fri, 2018-05-18 at 05:06 -0400, Kent Overstreet wrote: > > On Thu, May 17, 2018 at 08:54:57PM +, Bart Van Assche wrote: > > > With Jens' latest for-next branch I hit the kernel warning shown below. > > > Can > > > you have a look? > > > > Any hints on how to reproduce it? > > Sure. This is how I triggered it: > * Clone https://github.com/bvanassche/srp-test. > * Follow the instructions in README.md. > * Run srp-test/run_tests -c -r 10 Can you bisect it? I don't have infiniband hardware handy...
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On 5/18/18 10:23 AM, Christoph Hellwig wrote: > On Fri, May 11, 2018 at 03:13:38PM -0600, Jens Axboe wrote: >> Looked over the series, and looks like both good cleanups and optimizations. >> If we can get the mempool patch sorted, I can apply this for 4.18. > > FYI, I agree on the actual cleanups and optimization, but we really > shouldn't add new functions or even just exports without the code > using them. I think it is enough if we can collect ACKs on them, but > there is no point in using them. Especially as I'd really like to see > the users for some of them first. I certainly agree on that in general, but at the same time it makes the expected submission of bcachefs not having to carry a number of (essentially) unrelated patches. I'm assuming the likelihood of bcachefs being submitted soonish is high, hence we won't have exports that don't have in-kernel users in the longer term. -- Jens Axboe
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Fri, May 11, 2018 at 03:13:38PM -0600, Jens Axboe wrote: > Looked over the series, and looks like both good cleanups and optimizations. > If we can get the mempool patch sorted, I can apply this for 4.18. FYI, I agree on the actual cleanups and optimization, but we really shouldn't add new functions or even just exports without the code using them. I think it is enough if we can collect ACKs on them, but there is no point in using them. Especially as I'd really like to see the users for some of them first.
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Fri, 2018-05-18 at 05:06 -0400, Kent Overstreet wrote: > On Thu, May 17, 2018 at 08:54:57PM +, Bart Van Assche wrote: > > With Jens' latest for-next branch I hit the kernel warning shown below. Can > > you have a look? > > Any hints on how to reproduce it? Sure. This is how I triggered it: * Clone https://github.com/bvanassche/srp-test. * Follow the instructions in README.md. * Run srp-test/run_tests -c -r 10 Thanks, Bart.
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Thu, May 17, 2018 at 08:54:57PM +, Bart Van Assche wrote:
> On Tue, 2018-05-08 at 21:33 -0400, Kent Overstreet wrote:
> > [ ... ]
>
> Hello Kent,
>
> With Jens' latest for-next branch I hit the kernel warning shown below. Can
> you have a look?
Any hints on how to reproduce it?
> Thanks,
>
> Bart.
>
>
> ==
> BUG: KASAN: use-after-free in bio_advance+0x110/0x1b0
> Read of size 4 at addr 880156c5e6d0 by task ksoftirqd/10/72
>
> CPU: 10 PID: 72 Comm: ksoftirqd/10 Tainted: GW
> 4.17.0-rc4-dbg+ #5
> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
> 1.0.0-prebuilt.qemu-project.org 04/01/2014
> Call Trace:
> dump_stack+0x9a/0xeb
> print_address_description+0x65/0x270
> kasan_report+0x232/0x350
> bio_advance+0x110/0x1b0
> blk_update_request+0x9d/0x5a0
> scsi_end_request+0x4c/0x300 [scsi_mod]
> scsi_io_completion+0x71e/0xa40 [scsi_mod]
> __blk_mq_complete_request+0x143/0x220
> srp_recv_done+0x454/0x1100 [ib_srp]
> __ib_process_cq+0x9a/0xf0 [ib_core]
> ib_poll_handler+0x2d/0x90 [ib_core]
> irq_poll_softirq+0xe5/0x1e0
> __do_softirq+0x112/0x5f0
> run_ksoftirqd+0x29/0x50
> smpboot_thread_fn+0x30f/0x410
> kthread+0x1b2/0x1d0
> ret_from_fork+0x24/0x30
>
> Allocated by task 1356:
> kasan_kmalloc+0xa0/0xd0
> kmem_cache_alloc+0xed/0x320
> mempool_alloc+0xc6/0x210
> bio_alloc_bioset+0x128/0x2d0
> submit_bh_wbc+0x95/0x2d0
> __block_write_full_page+0x2a6/0x5c0
> __writepage+0x37/0x80
> write_cache_pages+0x305/0x7c0
> generic_writepages+0xb9/0x110
> do_writepages+0x96/0x180
> __filemap_fdatawrite_range+0x162/0x1b0
> file_write_and_wait_range+0x4d/0xb0
> blkdev_fsync+0x3c/0x70
> do_fsync+0x33/0x60
> __x64_sys_fsync+0x18/0x20
> do_syscall_64+0x6d/0x220
> entry_SYSCALL_64_after_hwframe+0x49/0xbe
>
> Freed by task 72:
> __kasan_slab_free+0x130/0x180
> kmem_cache_free+0xcd/0x380
> blk_update_request+0xc4/0x5a0
> blk_update_request+0xc4/0x5a0
> scsi_end_request+0x4c/0x300 [scsi_mod]
> scsi_io_completion+0x71e/0xa40 [scsi_mod]
> __blk_mq_complete_request+0x143/0x220
> srp_recv_done+0x454/0x1100 [ib_srp]
> __ib_process_cq+0x9a/0xf0 [ib_core]
> ib_poll_handler+0x2d/0x90 [ib_core]
> irq_poll_softirq+0xe5/0x1e0
> __do_softirq+0x112/0x5f0
>
> The buggy address belongs to the object at 880156c5e640
> which belongs to the cache bio-0 of size 200
> The buggy address is located 144 bytes inside of
> 200-byte region [880156c5e640, 880156c5e708)
> The buggy address belongs to the page:
> page:ea00055b1780 count:1 mapcount:0 mapping: index:0x0
> compound_mapcount: 0
> ib_srpt:srpt_zerolength_write: ib_srpt 10.196.159.179-24: queued zerolength
> write
> flags: 0x80008100(slab|head)
> raw: 80008100 000100190019
> raw: ea000543a800 00020002 88015a8f3a00
> ib_srpt:srpt_zerolength_write: ib_srpt 10.196.159.179-22: queued zerolength
> write
> page dumped because: kasan: bad access detected
> ib_srpt:srpt_zerolength_write: ib_srpt 10.196.159.179-20: queued zerolength
> write
>
> Memory state around the buggy address:
> ib_srpt:srpt_zerolength_write: ib_srpt 10.196.159.179-18: queued zerolength
> write
> 880156c5e580: 00 00 00 00 00 00 00 00 00 fc fc fc fc fc fc fc
> ib_srpt:srpt_zerolength_write_done: ib_srpt 10.196.159.179-24 wc->status 5
> 880156c5e600: fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb fb
> ib_srpt:srpt_zerolength_write_done: ib_srpt 10.196.159.179-22 wc->status 5
> >880156c5e680: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> ib_srpt:srpt_zerolength_write_done: ib_srpt 10.196.159.179-20 wc->status 5
> ^
> 880156c5e700: fb fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> ib_srpt:srpt_zerolength_write_done: ib_srpt 10.196.159.179-18 wc->status 5
> 880156c5e780: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> ib_srpt:srpt_release_channel_work: ib_srpt 10.196.159.179-24
> ==
>
> (gdb) list *(bio_advance+0x110)
> 0x81450090 is in bio_advance (./include/linux/bvec.h:82).
> 77 iter->bi_size = 0;
> 78 return false;
> 79 }
> 80
> 81 while (bytes) {
> 82 unsigned iter_len = bvec_iter_len(bv, *iter);
> 83 unsigned len = min(bytes, iter_len);
> 84
> 85 bytes -= len;
> 86 iter->bi_size -= len;
>
>
>
>
>
>
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Tue, 2018-05-08 at 21:33 -0400, Kent Overstreet wrote:
> [ ... ]
Hello Kent,
With Jens' latest for-next branch I hit the kernel warning shown below. Can
you have a look?
Thanks,
Bart.
==
BUG: KASAN: use-after-free in bio_advance+0x110/0x1b0
Read of size 4 at addr 880156c5e6d0 by task ksoftirqd/10/72
CPU: 10 PID: 72 Comm: ksoftirqd/10 Tainted: GW 4.17.0-rc4-dbg+
#5
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS
1.0.0-prebuilt.qemu-project.org 04/01/2014
Call Trace:
dump_stack+0x9a/0xeb
print_address_description+0x65/0x270
kasan_report+0x232/0x350
bio_advance+0x110/0x1b0
blk_update_request+0x9d/0x5a0
scsi_end_request+0x4c/0x300 [scsi_mod]
scsi_io_completion+0x71e/0xa40 [scsi_mod]
__blk_mq_complete_request+0x143/0x220
srp_recv_done+0x454/0x1100 [ib_srp]
__ib_process_cq+0x9a/0xf0 [ib_core]
ib_poll_handler+0x2d/0x90 [ib_core]
irq_poll_softirq+0xe5/0x1e0
__do_softirq+0x112/0x5f0
run_ksoftirqd+0x29/0x50
smpboot_thread_fn+0x30f/0x410
kthread+0x1b2/0x1d0
ret_from_fork+0x24/0x30
Allocated by task 1356:
kasan_kmalloc+0xa0/0xd0
kmem_cache_alloc+0xed/0x320
mempool_alloc+0xc6/0x210
bio_alloc_bioset+0x128/0x2d0
submit_bh_wbc+0x95/0x2d0
__block_write_full_page+0x2a6/0x5c0
__writepage+0x37/0x80
write_cache_pages+0x305/0x7c0
generic_writepages+0xb9/0x110
do_writepages+0x96/0x180
__filemap_fdatawrite_range+0x162/0x1b0
file_write_and_wait_range+0x4d/0xb0
blkdev_fsync+0x3c/0x70
do_fsync+0x33/0x60
__x64_sys_fsync+0x18/0x20
do_syscall_64+0x6d/0x220
entry_SYSCALL_64_after_hwframe+0x49/0xbe
Freed by task 72:
__kasan_slab_free+0x130/0x180
kmem_cache_free+0xcd/0x380
blk_update_request+0xc4/0x5a0
blk_update_request+0xc4/0x5a0
scsi_end_request+0x4c/0x300 [scsi_mod]
scsi_io_completion+0x71e/0xa40 [scsi_mod]
__blk_mq_complete_request+0x143/0x220
srp_recv_done+0x454/0x1100 [ib_srp]
__ib_process_cq+0x9a/0xf0 [ib_core]
ib_poll_handler+0x2d/0x90 [ib_core]
irq_poll_softirq+0xe5/0x1e0
__do_softirq+0x112/0x5f0
The buggy address belongs to the object at 880156c5e640
which belongs to the cache bio-0 of size 200
The buggy address is located 144 bytes inside of
200-byte region [880156c5e640, 880156c5e708)
The buggy address belongs to the page:
page:ea00055b1780 count:1 mapcount:0 mapping: index:0x0
compound_mapcount: 0
ib_srpt:srpt_zerolength_write: ib_srpt 10.196.159.179-24: queued zerolength
write
flags: 0x80008100(slab|head)
raw: 80008100 000100190019
raw: ea000543a800 00020002 88015a8f3a00
ib_srpt:srpt_zerolength_write: ib_srpt 10.196.159.179-22: queued zerolength
write
page dumped because: kasan: bad access detected
ib_srpt:srpt_zerolength_write: ib_srpt 10.196.159.179-20: queued zerolength
write
Memory state around the buggy address:
ib_srpt:srpt_zerolength_write: ib_srpt 10.196.159.179-18: queued zerolength
write
880156c5e580: 00 00 00 00 00 00 00 00 00 fc fc fc fc fc fc fc
ib_srpt:srpt_zerolength_write_done: ib_srpt 10.196.159.179-24 wc->status 5
880156c5e600: fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb fb
ib_srpt:srpt_zerolength_write_done: ib_srpt 10.196.159.179-22 wc->status 5
>880156c5e680: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ib_srpt:srpt_zerolength_write_done: ib_srpt 10.196.159.179-20 wc->status 5
^
880156c5e700: fb fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ib_srpt:srpt_zerolength_write_done: ib_srpt 10.196.159.179-18 wc->status 5
880156c5e780: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ib_srpt:srpt_release_channel_work: ib_srpt 10.196.159.179-24
==
(gdb) list *(bio_advance+0x110)
0x81450090 is in bio_advance (./include/linux/bvec.h:82).
77 iter->bi_size = 0;
78 return false;
79 }
80
81 while (bytes) {
82 unsigned iter_len = bvec_iter_len(bv, *iter);
83 unsigned len = min(bytes, iter_len);
84
85 bytes -= len;
86 iter->bi_size -= len;
Re: [PATCH 00/10] Misc block layer patches for bcachefs
Thanks! On Mon, May 14, 2018 at 3:24 PM, Jens Axboe wrote: > On 5/8/18 7:33 PM, Kent Overstreet wrote: >> - Add separately allowed mempools, biosets: bcachefs uses both all over the >>place >> >> - Bit of utility code - bio_copy_data_iter(), zero_fill_bio_iter() >> >> - bio_list_copy_data(), the bi_next check - defensiveness because of a bug I >>had fun chasing down at one point >> >> - add some exports, because bcachefs does dio its own way >> - show whether fua is supported in sysfs, because I don't know of anything >> that >>exports whether the _block layer_ specifically thinks fua is supported. > > Thanks Kent, applied for 4.18 with the update patch 1. > > -- > Jens Axboe >
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On 5/8/18 7:33 PM, Kent Overstreet wrote: > - Add separately allowed mempools, biosets: bcachefs uses both all over the >place > > - Bit of utility code - bio_copy_data_iter(), zero_fill_bio_iter() > > - bio_list_copy_data(), the bi_next check - defensiveness because of a bug I >had fun chasing down at one point > > - add some exports, because bcachefs does dio its own way > - show whether fua is supported in sysfs, because I don't know of anything > that >exports whether the _block layer_ specifically thinks fua is supported. Thanks Kent, applied for 4.18 with the update patch 1. -- Jens Axboe
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On 5/8/18 7:33 PM, Kent Overstreet wrote: > - Add separately allowed mempools, biosets: bcachefs uses both all over the >place > > - Bit of utility code - bio_copy_data_iter(), zero_fill_bio_iter() > > - bio_list_copy_data(), the bi_next check - defensiveness because of a bug I >had fun chasing down at one point > > - add some exports, because bcachefs does dio its own way > - show whether fua is supported in sysfs, because I don't know of anything > that >exports whether the _block layer_ specifically thinks fua is supported. Looked over the series, and looks like both good cleanups and optimizations. If we can get the mempool patch sorted, I can apply this for 4.18. -- Jens Axboe
