Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Sun, 2018-05-20 at 18:17 -0400, Kent Overstreet wrote: > On Fri, May 18, 2018 at 03:12:27PM +, Bart Van Assche wrote: > > On Fri, 2018-05-18 at 05:06 -0400, Kent Overstreet wrote: > > > On Thu, May 17, 2018 at 08:54:57PM +, Bart Van Assche wrote: > > > > With Jens' latest for-next branch I hit the kernel warning shown below. > > > > Can > > > > you have a look? > > > > > > Any hints on how to reproduce it? > > > > Sure. This is how I triggered it: > > * Clone https://github.com/bvanassche/srp-test. > > * Follow the instructions in README.md. > > * Run srp-test/run_tests -c -r 10 > > Can you bisect it? I don't have infiniband hardware handy... Hello Kent, The bisect I ran reported the following commit as the first faulty commit: block: Add warning for bi_next not NULL in bio_endio() Bart.
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Mon, 2018-05-21 at 11:37 -0700, Omar Sandoval wrote: > Have you made any progress in porting srp-test to blktests so we don't > have to have this conversation again? Hello Omar, Porting the srp-test software to the blktests framework is still high on my to-do list. I will start working on this as soon as the work on adding SMR support for fio has been completed. Bart.
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Mon, May 21, 2018 at 03:11:08PM +, Bart Van Assche wrote: > On Sun, 2018-05-20 at 19:58 -0400, Kent Overstreet wrote: > > On Sun, May 20, 2018 at 11:40:45PM +, Bart Van Assche wrote: > > > On Sun, 2018-05-20 at 19:21 -0400, Kent Overstreet wrote: > > > > I really have better things to do than debug someone else's tests... > > > > [ ... ] > > > > ../run_tests: line 65: cd: /lib/modules/4.16.0+/kernel/block: No such > > > > file or directory > > > > > > Kernel v4.16 is too old to run these tests. The srp-test script needs the > > > following commit that went upstream in kernel v4.17-rc1: > > > > > > 63cf1a902c9d ("IB/srpt: Add RDMA/CM support") > > > > Same output on Jens' for-next branch. > > Others have been able to run the srp-test software with the instructions > provided earlier in this e-mail thread. Can you share the kernel messages from > around the time the test was run (dmesg, /var/log/messages or > /var/log/syslog)? > > Thanks, > > Bart. Bart, Have you made any progress in porting srp-test to blktests so we don't have to have this conversation again?
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Sun, 2018-05-20 at 19:58 -0400, Kent Overstreet wrote: > On Sun, May 20, 2018 at 11:40:45PM +, Bart Van Assche wrote: > > On Sun, 2018-05-20 at 19:21 -0400, Kent Overstreet wrote: > > > I really have better things to do than debug someone else's tests... > > > [ ... ] > > > ../run_tests: line 65: cd: /lib/modules/4.16.0+/kernel/block: No such > > > file or directory > > > > Kernel v4.16 is too old to run these tests. The srp-test script needs the > > following commit that went upstream in kernel v4.17-rc1: > > > > 63cf1a902c9d ("IB/srpt: Add RDMA/CM support") > > Same output on Jens' for-next branch. Others have been able to run the srp-test software with the instructions provided earlier in this e-mail thread. Can you share the kernel messages from around the time the test was run (dmesg, /var/log/messages or /var/log/syslog)? Thanks, Bart.
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Sun, May 20, 2018 at 11:40:45PM +, Bart Van Assche wrote: > On Sun, 2018-05-20 at 19:21 -0400, Kent Overstreet wrote: > > I really have better things to do than debug someone else's tests... > > [ ... ] > > ../run_tests: line 65: cd: /lib/modules/4.16.0+/kernel/block: No such file > > or directory > > Kernel v4.16 is too old to run these tests. The srp-test script needs the > following commit that went upstream in kernel v4.17-rc1: > > 63cf1a902c9d ("IB/srpt: Add RDMA/CM support") Same output on Jens' for-next branch.
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Sun, 2018-05-20 at 19:21 -0400, Kent Overstreet wrote: > I really have better things to do than debug someone else's tests... > [ ... ] > ../run_tests: line 65: cd: /lib/modules/4.16.0+/kernel/block: No such file or > directory Kernel v4.16 is too old to run these tests. The srp-test script needs the following commit that went upstream in kernel v4.17-rc1: 63cf1a902c9d ("IB/srpt: Add RDMA/CM support") Bart.
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Sun, May 20, 2018 at 10:35:29PM +, Bart Van Assche wrote: > On Sun, 2018-05-20 at 18:31 -0400, Kent Overstreet wrote: > > On Sun, May 20, 2018 at 10:19:13PM +, Bart Van Assche wrote: > > > On Sun, 2018-05-20 at 18:17 -0400, Kent Overstreet wrote: > > > > On Fri, May 18, 2018 at 03:12:27PM +, Bart Van Assche wrote: > > > > > On Fri, 2018-05-18 at 05:06 -0400, Kent Overstreet wrote: > > > > > > On Thu, May 17, 2018 at 08:54:57PM +, Bart Van Assche wrote: > > > > > > > With Jens' latest for-next branch I hit the kernel warning shown > > > > > > > below. Can > > > > > > > you have a look? > > > > > > > > > > > > Any hints on how to reproduce it? > > > > > > > > > > Sure. This is how I triggered it: > > > > > * Clone https://github.com/bvanassche/srp-test. > > > > > * Follow the instructions in README.md. > > > > > * Run srp-test/run_tests -c -r 10 > > > > > > > > Can you bisect it? I don't have infiniband hardware handy... > > > > > > Hello Kent, > > > > > > Have you noticed that the test I described uses the rdma_rxe driver and > > > hence that > > > no InfiniBand hardware is needed to run that test? > > > > No, I'm not terribly familiar with infiniband stuff > > > > Do you have some sort of self contained test/qemu recipe? I would really > > rather > > not have to figure out how to configure multipath, and infiniband, and I'm > > not > > even sure what else is needed based on that readme... > > Hello Kent, > > Please have another look at the srp-test README. The instructions in that > document > are easy to follow. No multipath nor any InfiniBand knowledge is required. > The test > even can be run in a virtual machine in case you would be worried about > potential > impact of the test on the rest of the system. I really have better things to do than debug someone else's tests... Restarting multipath-tools (via systemctl): multipath-tools.service. multipathd> reconfigure ok multipathd> make -C discontiguous-io discontiguous-io make[1]: Entering directory '/host/home/kent/ktest/tests/srp-test/discontiguous-io' make[1]: 'discontiguous-io' is up to date. make[1]: Leaving directory '/host/home/kent/ktest/tests/srp-test/discontiguous-io' Unloaded the ib_srpt kernel module Unloaded the rdma_rxe kernel module ../run_tests: line 65: cd: /lib/modules/4.16.0+/kernel/block: No such file or directory Zero-initializing /dev/ram0 ... done Zero-initializing /dev/ram1 ... done Unable to load target_core_pscsi Unable to load target_core_user Configured SRP target driver Running test ../tests/01 ... Unloaded the ib_srp kernel module SRP login failed Test ../tests/01 failed Running test ../tests/02-mq ... Test file I/O on top of multipath concurrently with logout and login (10 min; mq) Unloaded the ib_srp kernel module SRP login failed Test ../tests/02-mq failed Running test ../tests/02-sq ... Test file I/O on top of multipath concurrently with logout and login (10 min; sq) Unloaded the ib_srp kernel module SRP login failed Test ../tests/02-sq failed Running test ../tests/02-sq-on-mq ... Test file I/O on top of multipath concurrently with logout and login (10 min; sq-on-mq) Unloaded the ib_srp kernel module SRP login failed Test ../tests/02-sq-on-mq failed Running test ../tests/03-4M ... Test direct I/O with large transfer sizes and cmd_sg_entries=255 Unloaded the ib_srp kernel module SRP login failed Test ../tests/03-4M failed Running test ../tests/03-8M ... Test direct I/O with large transfer sizes and cmd_sg_entries=255 Unloaded the ib_srp kernel module SRP login failed Test ../tests/03-8M failed Running test ../tests/04-4M ... Test direct I/O with large transfer sizes and cmd_sg_entries=1 Unloaded the ib_srp kernel module SRP login failed Test ../tests/04-4M failed Running test ../tests/04-8M ... Test direct I/O with large transfer sizes and cmd_sg_entries=1 Unloaded the ib_srp kernel module SRP login failed Test ../tests/04-8M failed Running test ../tests/05-4M ... Test buffered I/O with large transfer sizes and cmd_sg_entries=255 Unloaded the ib_srp kernel module SRP login failed Test ../tests/05-4M failed Running test ../tests/05-8M ... Test buffered I/O with large transfer sizes and cmd_sg_entries=255 Unloaded the ib_srp kernel module SRP login failed Test ../tests/05-8M failed Running test ../tests/06 ... Test block I/O on top of multipath concurrently with logout and login (10 min) Unloaded the ib_srp kernel module SRP login failed Test ../tests/06 failed 0 tests succeeded and 11 tests failed Unloaded the ib_srpt kernel module Unloaded the rdma_rxe kernel module = PASSED srp in 18s
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Sun, 2018-05-20 at 19:00 -0400, Kent Overstreet wrote: > On Sun, May 20, 2018 at 10:35:29PM +, Bart Van Assche wrote: > > On Sun, 2018-05-20 at 18:31 -0400, Kent Overstreet wrote: > > > On Sun, May 20, 2018 at 10:19:13PM +, Bart Van Assche wrote: > > > > On Sun, 2018-05-20 at 18:17 -0400, Kent Overstreet wrote: > > > > > On Fri, May 18, 2018 at 03:12:27PM +, Bart Van Assche wrote: > > > > > > On Fri, 2018-05-18 at 05:06 -0400, Kent Overstreet wrote: > > > > > > > On Thu, May 17, 2018 at 08:54:57PM +, Bart Van Assche wrote: > > > > > > > > With Jens' latest for-next branch I hit the kernel warning > > > > > > > > shown below. Can > > > > > > > > you have a look? > > > > > > > > > > > > > > Any hints on how to reproduce it? > > > > > > > > > > > > Sure. This is how I triggered it: > > > > > > * Clone https://github.com/bvanassche/srp-test. > > > > > > * Follow the instructions in README.md. > > > > > > * Run srp-test/run_tests -c -r 10 > > > > > > > > > > Can you bisect it? I don't have infiniband hardware handy... > > > > > > > > Hello Kent, > > > > > > > > Have you noticed that the test I described uses the rdma_rxe driver and > > > > hence that > > > > no InfiniBand hardware is needed to run that test? > > > > > > No, I'm not terribly familiar with infiniband stuff > > > > > > Do you have some sort of self contained test/qemu recipe? I would really > > > rather > > > not have to figure out how to configure multipath, and infiniband, and > > > I'm not > > > even sure what else is needed based on that readme... > > > > Please have another look at the srp-test README. The instructions in that > > document > > are easy to follow. No multipath nor any InfiniBand knowledge is required. > > The test > > even can be run in a virtual machine in case you would be worried about > > potential > > impact of the test on the rest of the system. > > Your readme refers to kernel config options that don't even exist in the > current > kernel That's probably a misunderstanding of your side. From a quick look it seems like all config symbols mentioned in the srp-test README.md exist in kernel v4.17-rc6. The following command does not report any non-existing Kconfig symbols: $ cd linux-kernel $ sed -n 's/^\* CONFIG_//p' ~/software/infiniband/srp-test/README.md | while read f; do { git grep -lw "$f" | grep -q Kconfig; } || echo $f; done Bart.
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Sun, May 20, 2018 at 10:35:29PM +, Bart Van Assche wrote: > On Sun, 2018-05-20 at 18:31 -0400, Kent Overstreet wrote: > > On Sun, May 20, 2018 at 10:19:13PM +, Bart Van Assche wrote: > > > On Sun, 2018-05-20 at 18:17 -0400, Kent Overstreet wrote: > > > > On Fri, May 18, 2018 at 03:12:27PM +, Bart Van Assche wrote: > > > > > On Fri, 2018-05-18 at 05:06 -0400, Kent Overstreet wrote: > > > > > > On Thu, May 17, 2018 at 08:54:57PM +, Bart Van Assche wrote: > > > > > > > With Jens' latest for-next branch I hit the kernel warning shown > > > > > > > below. Can > > > > > > > you have a look? > > > > > > > > > > > > Any hints on how to reproduce it? > > > > > > > > > > Sure. This is how I triggered it: > > > > > * Clone https://github.com/bvanassche/srp-test. > > > > > * Follow the instructions in README.md. > > > > > * Run srp-test/run_tests -c -r 10 > > > > > > > > Can you bisect it? I don't have infiniband hardware handy... > > > > > > Hello Kent, > > > > > > Have you noticed that the test I described uses the rdma_rxe driver and > > > hence that > > > no InfiniBand hardware is needed to run that test? > > > > No, I'm not terribly familiar with infiniband stuff > > > > Do you have some sort of self contained test/qemu recipe? I would really > > rather > > not have to figure out how to configure multipath, and infiniband, and I'm > > not > > even sure what else is needed based on that readme... > > Hello Kent, > > Please have another look at the srp-test README. The instructions in that > document > are easy to follow. No multipath nor any InfiniBand knowledge is required. > The test > even can be run in a virtual machine in case you would be worried about > potential > impact of the test on the rest of the system. Your readme refers to kernel config options that don't even exist in the current kernel
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Sun, 2018-05-20 at 18:31 -0400, Kent Overstreet wrote: > On Sun, May 20, 2018 at 10:19:13PM +, Bart Van Assche wrote: > > On Sun, 2018-05-20 at 18:17 -0400, Kent Overstreet wrote: > > > On Fri, May 18, 2018 at 03:12:27PM +, Bart Van Assche wrote: > > > > On Fri, 2018-05-18 at 05:06 -0400, Kent Overstreet wrote: > > > > > On Thu, May 17, 2018 at 08:54:57PM +, Bart Van Assche wrote: > > > > > > With Jens' latest for-next branch I hit the kernel warning shown > > > > > > below. Can > > > > > > you have a look? > > > > > > > > > > Any hints on how to reproduce it? > > > > > > > > Sure. This is how I triggered it: > > > > * Clone https://github.com/bvanassche/srp-test. > > > > * Follow the instructions in README.md. > > > > * Run srp-test/run_tests -c -r 10 > > > > > > Can you bisect it? I don't have infiniband hardware handy... > > > > Hello Kent, > > > > Have you noticed that the test I described uses the rdma_rxe driver and > > hence that > > no InfiniBand hardware is needed to run that test? > > No, I'm not terribly familiar with infiniband stuff > > Do you have some sort of self contained test/qemu recipe? I would really > rather > not have to figure out how to configure multipath, and infiniband, and I'm not > even sure what else is needed based on that readme... Hello Kent, Please have another look at the srp-test README. The instructions in that document are easy to follow. No multipath nor any InfiniBand knowledge is required. The test even can be run in a virtual machine in case you would be worried about potential impact of the test on the rest of the system. Bart.
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Sun, May 20, 2018 at 10:19:13PM +, Bart Van Assche wrote: > On Sun, 2018-05-20 at 18:17 -0400, Kent Overstreet wrote: > > On Fri, May 18, 2018 at 03:12:27PM +, Bart Van Assche wrote: > > > On Fri, 2018-05-18 at 05:06 -0400, Kent Overstreet wrote: > > > > On Thu, May 17, 2018 at 08:54:57PM +, Bart Van Assche wrote: > > > > > With Jens' latest for-next branch I hit the kernel warning shown > > > > > below. Can > > > > > you have a look? > > > > > > > > Any hints on how to reproduce it? > > > > > > Sure. This is how I triggered it: > > > * Clone https://github.com/bvanassche/srp-test. > > > * Follow the instructions in README.md. > > > * Run srp-test/run_tests -c -r 10 > > > > Can you bisect it? I don't have infiniband hardware handy... > > Hello Kent, > > Have you noticed that the test I described uses the rdma_rxe driver and hence > that > no InfiniBand hardware is needed to run that test? No, I'm not terribly familiar with infiniband stuff Do you have some sort of self contained test/qemu recipe? I would really rather not have to figure out how to configure multipath, and infiniband, and I'm not even sure what else is needed based on that readme...
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Sun, 2018-05-20 at 18:17 -0400, Kent Overstreet wrote: > On Fri, May 18, 2018 at 03:12:27PM +, Bart Van Assche wrote: > > On Fri, 2018-05-18 at 05:06 -0400, Kent Overstreet wrote: > > > On Thu, May 17, 2018 at 08:54:57PM +, Bart Van Assche wrote: > > > > With Jens' latest for-next branch I hit the kernel warning shown below. > > > > Can > > > > you have a look? > > > > > > Any hints on how to reproduce it? > > > > Sure. This is how I triggered it: > > * Clone https://github.com/bvanassche/srp-test. > > * Follow the instructions in README.md. > > * Run srp-test/run_tests -c -r 10 > > Can you bisect it? I don't have infiniband hardware handy... Hello Kent, Have you noticed that the test I described uses the rdma_rxe driver and hence that no InfiniBand hardware is needed to run that test? Thanks, Bart.
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Fri, May 18, 2018 at 03:12:27PM +, Bart Van Assche wrote: > On Fri, 2018-05-18 at 05:06 -0400, Kent Overstreet wrote: > > On Thu, May 17, 2018 at 08:54:57PM +, Bart Van Assche wrote: > > > With Jens' latest for-next branch I hit the kernel warning shown below. > > > Can > > > you have a look? > > > > Any hints on how to reproduce it? > > Sure. This is how I triggered it: > * Clone https://github.com/bvanassche/srp-test. > * Follow the instructions in README.md. > * Run srp-test/run_tests -c -r 10 Can you bisect it? I don't have infiniband hardware handy...
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On 5/18/18 10:23 AM, Christoph Hellwig wrote: > On Fri, May 11, 2018 at 03:13:38PM -0600, Jens Axboe wrote: >> Looked over the series, and looks like both good cleanups and optimizations. >> If we can get the mempool patch sorted, I can apply this for 4.18. > > FYI, I agree on the actual cleanups and optimization, but we really > shouldn't add new functions or even just exports without the code > using them. I think it is enough if we can collect ACKs on them, but > there is no point in using them. Especially as I'd really like to see > the users for some of them first. I certainly agree on that in general, but at the same time it makes the expected submission of bcachefs not having to carry a number of (essentially) unrelated patches. I'm assuming the likelihood of bcachefs being submitted soonish is high, hence we won't have exports that don't have in-kernel users in the longer term. -- Jens Axboe
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Fri, May 11, 2018 at 03:13:38PM -0600, Jens Axboe wrote: > Looked over the series, and looks like both good cleanups and optimizations. > If we can get the mempool patch sorted, I can apply this for 4.18. FYI, I agree on the actual cleanups and optimization, but we really shouldn't add new functions or even just exports without the code using them. I think it is enough if we can collect ACKs on them, but there is no point in using them. Especially as I'd really like to see the users for some of them first.
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Fri, 2018-05-18 at 05:06 -0400, Kent Overstreet wrote: > On Thu, May 17, 2018 at 08:54:57PM +, Bart Van Assche wrote: > > With Jens' latest for-next branch I hit the kernel warning shown below. Can > > you have a look? > > Any hints on how to reproduce it? Sure. This is how I triggered it: * Clone https://github.com/bvanassche/srp-test. * Follow the instructions in README.md. * Run srp-test/run_tests -c -r 10 Thanks, Bart.
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Thu, May 17, 2018 at 08:54:57PM +, Bart Van Assche wrote: > On Tue, 2018-05-08 at 21:33 -0400, Kent Overstreet wrote: > > [ ... ] > > Hello Kent, > > With Jens' latest for-next branch I hit the kernel warning shown below. Can > you have a look? Any hints on how to reproduce it? > Thanks, > > Bart. > > > == > BUG: KASAN: use-after-free in bio_advance+0x110/0x1b0 > Read of size 4 at addr 880156c5e6d0 by task ksoftirqd/10/72 > > CPU: 10 PID: 72 Comm: ksoftirqd/10 Tainted: GW > 4.17.0-rc4-dbg+ #5 > Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS > 1.0.0-prebuilt.qemu-project.org 04/01/2014 > Call Trace: > dump_stack+0x9a/0xeb > print_address_description+0x65/0x270 > kasan_report+0x232/0x350 > bio_advance+0x110/0x1b0 > blk_update_request+0x9d/0x5a0 > scsi_end_request+0x4c/0x300 [scsi_mod] > scsi_io_completion+0x71e/0xa40 [scsi_mod] > __blk_mq_complete_request+0x143/0x220 > srp_recv_done+0x454/0x1100 [ib_srp] > __ib_process_cq+0x9a/0xf0 [ib_core] > ib_poll_handler+0x2d/0x90 [ib_core] > irq_poll_softirq+0xe5/0x1e0 > __do_softirq+0x112/0x5f0 > run_ksoftirqd+0x29/0x50 > smpboot_thread_fn+0x30f/0x410 > kthread+0x1b2/0x1d0 > ret_from_fork+0x24/0x30 > > Allocated by task 1356: > kasan_kmalloc+0xa0/0xd0 > kmem_cache_alloc+0xed/0x320 > mempool_alloc+0xc6/0x210 > bio_alloc_bioset+0x128/0x2d0 > submit_bh_wbc+0x95/0x2d0 > __block_write_full_page+0x2a6/0x5c0 > __writepage+0x37/0x80 > write_cache_pages+0x305/0x7c0 > generic_writepages+0xb9/0x110 > do_writepages+0x96/0x180 > __filemap_fdatawrite_range+0x162/0x1b0 > file_write_and_wait_range+0x4d/0xb0 > blkdev_fsync+0x3c/0x70 > do_fsync+0x33/0x60 > __x64_sys_fsync+0x18/0x20 > do_syscall_64+0x6d/0x220 > entry_SYSCALL_64_after_hwframe+0x49/0xbe > > Freed by task 72: > __kasan_slab_free+0x130/0x180 > kmem_cache_free+0xcd/0x380 > blk_update_request+0xc4/0x5a0 > blk_update_request+0xc4/0x5a0 > scsi_end_request+0x4c/0x300 [scsi_mod] > scsi_io_completion+0x71e/0xa40 [scsi_mod] > __blk_mq_complete_request+0x143/0x220 > srp_recv_done+0x454/0x1100 [ib_srp] > __ib_process_cq+0x9a/0xf0 [ib_core] > ib_poll_handler+0x2d/0x90 [ib_core] > irq_poll_softirq+0xe5/0x1e0 > __do_softirq+0x112/0x5f0 > > The buggy address belongs to the object at 880156c5e640 > which belongs to the cache bio-0 of size 200 > The buggy address is located 144 bytes inside of > 200-byte region [880156c5e640, 880156c5e708) > The buggy address belongs to the page: > page:ea00055b1780 count:1 mapcount:0 mapping: index:0x0 > compound_mapcount: 0 > ib_srpt:srpt_zerolength_write: ib_srpt 10.196.159.179-24: queued zerolength > write > flags: 0x80008100(slab|head) > raw: 80008100 000100190019 > raw: ea000543a800 00020002 88015a8f3a00 > ib_srpt:srpt_zerolength_write: ib_srpt 10.196.159.179-22: queued zerolength > write > page dumped because: kasan: bad access detected > ib_srpt:srpt_zerolength_write: ib_srpt 10.196.159.179-20: queued zerolength > write > > Memory state around the buggy address: > ib_srpt:srpt_zerolength_write: ib_srpt 10.196.159.179-18: queued zerolength > write > 880156c5e580: 00 00 00 00 00 00 00 00 00 fc fc fc fc fc fc fc > ib_srpt:srpt_zerolength_write_done: ib_srpt 10.196.159.179-24 wc->status 5 > 880156c5e600: fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb fb > ib_srpt:srpt_zerolength_write_done: ib_srpt 10.196.159.179-22 wc->status 5 > >880156c5e680: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > ib_srpt:srpt_zerolength_write_done: ib_srpt 10.196.159.179-20 wc->status 5 > ^ > 880156c5e700: fb fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc > ib_srpt:srpt_zerolength_write_done: ib_srpt 10.196.159.179-18 wc->status 5 > 880156c5e780: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > ib_srpt:srpt_release_channel_work: ib_srpt 10.196.159.179-24 > == > > (gdb) list *(bio_advance+0x110) > 0x81450090 is in bio_advance (./include/linux/bvec.h:82). > 77 iter->bi_size = 0; > 78 return false; > 79 } > 80 > 81 while (bytes) { > 82 unsigned iter_len = bvec_iter_len(bv, *iter); > 83 unsigned len = min(bytes, iter_len); > 84 > 85 bytes -= len; > 86 iter->bi_size -= len; > > > > > >
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On Tue, 2018-05-08 at 21:33 -0400, Kent Overstreet wrote: > [ ... ] Hello Kent, With Jens' latest for-next branch I hit the kernel warning shown below. Can you have a look? Thanks, Bart. == BUG: KASAN: use-after-free in bio_advance+0x110/0x1b0 Read of size 4 at addr 880156c5e6d0 by task ksoftirqd/10/72 CPU: 10 PID: 72 Comm: ksoftirqd/10 Tainted: GW 4.17.0-rc4-dbg+ #5 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.0.0-prebuilt.qemu-project.org 04/01/2014 Call Trace: dump_stack+0x9a/0xeb print_address_description+0x65/0x270 kasan_report+0x232/0x350 bio_advance+0x110/0x1b0 blk_update_request+0x9d/0x5a0 scsi_end_request+0x4c/0x300 [scsi_mod] scsi_io_completion+0x71e/0xa40 [scsi_mod] __blk_mq_complete_request+0x143/0x220 srp_recv_done+0x454/0x1100 [ib_srp] __ib_process_cq+0x9a/0xf0 [ib_core] ib_poll_handler+0x2d/0x90 [ib_core] irq_poll_softirq+0xe5/0x1e0 __do_softirq+0x112/0x5f0 run_ksoftirqd+0x29/0x50 smpboot_thread_fn+0x30f/0x410 kthread+0x1b2/0x1d0 ret_from_fork+0x24/0x30 Allocated by task 1356: kasan_kmalloc+0xa0/0xd0 kmem_cache_alloc+0xed/0x320 mempool_alloc+0xc6/0x210 bio_alloc_bioset+0x128/0x2d0 submit_bh_wbc+0x95/0x2d0 __block_write_full_page+0x2a6/0x5c0 __writepage+0x37/0x80 write_cache_pages+0x305/0x7c0 generic_writepages+0xb9/0x110 do_writepages+0x96/0x180 __filemap_fdatawrite_range+0x162/0x1b0 file_write_and_wait_range+0x4d/0xb0 blkdev_fsync+0x3c/0x70 do_fsync+0x33/0x60 __x64_sys_fsync+0x18/0x20 do_syscall_64+0x6d/0x220 entry_SYSCALL_64_after_hwframe+0x49/0xbe Freed by task 72: __kasan_slab_free+0x130/0x180 kmem_cache_free+0xcd/0x380 blk_update_request+0xc4/0x5a0 blk_update_request+0xc4/0x5a0 scsi_end_request+0x4c/0x300 [scsi_mod] scsi_io_completion+0x71e/0xa40 [scsi_mod] __blk_mq_complete_request+0x143/0x220 srp_recv_done+0x454/0x1100 [ib_srp] __ib_process_cq+0x9a/0xf0 [ib_core] ib_poll_handler+0x2d/0x90 [ib_core] irq_poll_softirq+0xe5/0x1e0 __do_softirq+0x112/0x5f0 The buggy address belongs to the object at 880156c5e640 which belongs to the cache bio-0 of size 200 The buggy address is located 144 bytes inside of 200-byte region [880156c5e640, 880156c5e708) The buggy address belongs to the page: page:ea00055b1780 count:1 mapcount:0 mapping: index:0x0 compound_mapcount: 0 ib_srpt:srpt_zerolength_write: ib_srpt 10.196.159.179-24: queued zerolength write flags: 0x80008100(slab|head) raw: 80008100 000100190019 raw: ea000543a800 00020002 88015a8f3a00 ib_srpt:srpt_zerolength_write: ib_srpt 10.196.159.179-22: queued zerolength write page dumped because: kasan: bad access detected ib_srpt:srpt_zerolength_write: ib_srpt 10.196.159.179-20: queued zerolength write Memory state around the buggy address: ib_srpt:srpt_zerolength_write: ib_srpt 10.196.159.179-18: queued zerolength write 880156c5e580: 00 00 00 00 00 00 00 00 00 fc fc fc fc fc fc fc ib_srpt:srpt_zerolength_write_done: ib_srpt 10.196.159.179-24 wc->status 5 880156c5e600: fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb fb ib_srpt:srpt_zerolength_write_done: ib_srpt 10.196.159.179-22 wc->status 5 >880156c5e680: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb ib_srpt:srpt_zerolength_write_done: ib_srpt 10.196.159.179-20 wc->status 5 ^ 880156c5e700: fb fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc ib_srpt:srpt_zerolength_write_done: ib_srpt 10.196.159.179-18 wc->status 5 880156c5e780: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb ib_srpt:srpt_release_channel_work: ib_srpt 10.196.159.179-24 == (gdb) list *(bio_advance+0x110) 0x81450090 is in bio_advance (./include/linux/bvec.h:82). 77 iter->bi_size = 0; 78 return false; 79 } 80 81 while (bytes) { 82 unsigned iter_len = bvec_iter_len(bv, *iter); 83 unsigned len = min(bytes, iter_len); 84 85 bytes -= len; 86 iter->bi_size -= len;
Re: [PATCH 00/10] Misc block layer patches for bcachefs
Thanks! On Mon, May 14, 2018 at 3:24 PM, Jens Axboe wrote: > On 5/8/18 7:33 PM, Kent Overstreet wrote: >> - Add separately allowed mempools, biosets: bcachefs uses both all over the >>place >> >> - Bit of utility code - bio_copy_data_iter(), zero_fill_bio_iter() >> >> - bio_list_copy_data(), the bi_next check - defensiveness because of a bug I >>had fun chasing down at one point >> >> - add some exports, because bcachefs does dio its own way >> - show whether fua is supported in sysfs, because I don't know of anything >> that >>exports whether the _block layer_ specifically thinks fua is supported. > > Thanks Kent, applied for 4.18 with the update patch 1. > > -- > Jens Axboe >
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On 5/8/18 7:33 PM, Kent Overstreet wrote: > - Add separately allowed mempools, biosets: bcachefs uses both all over the >place > > - Bit of utility code - bio_copy_data_iter(), zero_fill_bio_iter() > > - bio_list_copy_data(), the bi_next check - defensiveness because of a bug I >had fun chasing down at one point > > - add some exports, because bcachefs does dio its own way > - show whether fua is supported in sysfs, because I don't know of anything > that >exports whether the _block layer_ specifically thinks fua is supported. Thanks Kent, applied for 4.18 with the update patch 1. -- Jens Axboe
Re: [PATCH 00/10] Misc block layer patches for bcachefs
On 5/8/18 7:33 PM, Kent Overstreet wrote: > - Add separately allowed mempools, biosets: bcachefs uses both all over the >place > > - Bit of utility code - bio_copy_data_iter(), zero_fill_bio_iter() > > - bio_list_copy_data(), the bi_next check - defensiveness because of a bug I >had fun chasing down at one point > > - add some exports, because bcachefs does dio its own way > - show whether fua is supported in sysfs, because I don't know of anything > that >exports whether the _block layer_ specifically thinks fua is supported. Looked over the series, and looks like both good cleanups and optimizations. If we can get the mempool patch sorted, I can apply this for 4.18. -- Jens Axboe