Hi,
I'm working on an application that makes heavy use of iSCSI and I'm seeing
crashes in blk_rq_map_sg() every two or three days. I'm not necessarily
expecting it to be anything to do with open-iscsi, but since the
application is making heavy use of iSCSI I am curious whether anyone
knowledgeable about open-iscsi thinks it could be.
I have a fairly simple application which is writing large amounts of data
to several partitions on one iSCSI volume (/dev/sda1 up to /dev/sda40).
There is no filesystem involved, I'm simply opening 40 file descriptors on
"/dev/sdX" and then using pwrite() calls to write raw data to the device
(and some occasional pread()s). The app is writing constantly at about
100mbps, and randomly accessing all of the partitions all of the time.
Writes are not always sequential, it might write 512 bytes to /dev/sda5 at
offset X, then write 2048 bytes to /dev/sda5 at offset Y, then write 1024
bytes to a different partition /dev/sda30, then read 512b from /dev/sda12
etc. The point being that there will be a fair amount of both unrelated and
related 512 byte blocks in the VM's dirty cache, some of which can be
coalesced and some not, it's quite random.
The crash in question appears to be a bad address in the bio_vec list in
the request queue passed to blk_rq_map_sg(). It crashes derefencing a
bio_vec "bvec->bv_len". I'm guessing that it's either a memory stomp /
overrun, a locking issue, or that the stressful mix of partitions, offsets,
reads and writes is triggering a latent bug.
One thing that is probably quite unusual in terms of open-iscsi is that I
have set the block size for each partition to 512 bytes using ioctl
BLKBSZSET. This is because I need to write and read random offsets in
blocks of 512 bytes - if I leave the partition's block size at the default
4KB, then writing 512 bytes incurs a readback of 4KB from the target iSCSI
device to fill Linux's block cache. This is a write-heavy application, and
that typically led to 100mbps writes to the iSCSI target incurring 100mbps
readbacks from the iSCSI target. Using a block size of 512b means that
writes of 512b into the /dev/ node do not incur readbacks from the target,
so network bandwidth is 100mbps write and negligible read.
The kernel dump is below and this is from an ARM device running a Freescale
3.0.35 kernel.
Any thoughts would be much appreciated. Thanks,
Kelvin.
Unable to handle kernel paging request at virtual address 02601109
pgd = 80004000
[02601109] *pgd=00000000
Internal error: Oops: 5 [#1] PREEMPT SMP
Modules linked in:
CPU: 1 Not tainted (3.0.35-ge6e1b5a-dirty #8)
PC is at blk_rq_map_sg+0xd0/0x2cc
LR is at scsi_init_sgtable+0x50/0xb0
pc : [<801fbe44>] lr : [<8028dc20>] psr: a00f0093
sp : 93c63cf8 ip : 00000000 fp : 00000000
r10: 00000000 r9 : 00000200 r8 : 02601105
r7 : 98837a80 r6 : 00000001 r5 : baae7800 r4 : ba00c3c0
r3 : 98837a48 r2 : 00000000 r1 : 02601105 r0 : ba00c3c0
Flags: NzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment kernel
Control: 10c53c7d Table: 2f94404a DAC: 00000015
Process flush-8:16 (pid: 13070, stack limit = 0x93c622f0)
Stack: (0x93c63cf8 to 0x93c64000)
3ce0: 2b400599
baae7800
3d00: 00000000 ba15f438 00000020 ba15f438 bac54cb4 00000000 00000000
ba00c3c0
3d20: 8053ac8c 00000000 bfd9b400 8028dc20 8028dcdc 00010000 bac54c80
ba15f438
3d40: 00010000 8028de1c ba15f438 00000001 00010000 802a0e54 ba15f438
bfced260
3d60: bfced260 bfd9a800 6b73f7a4 00000000 bfd9ac00 ba00c3c0 00000017
8053ac8c
3d80: 00000000 bfd9ac00 bfced260 ba15f438 00003776 80204434 ba15f870
bfbfea00
3da0: 4460e224 ba15f438 ba00c3c0 00000000 00000000 bfd9b42c 8053ac8c
00000000
3dc0: bac54640 801f8818 bfd9b400 ba00c3c0 bfc14c00 8028eb78 80203b74
bfd9b4c0
3de0: ba00c3c0 ba00c3c0 93c63e10 00000000 600f0013 93c63e10 ba00c548
bc023c88
3e00: 00000000 801f6ff8 ba00c3c0 801f8ee8 93c63e10 93c63e10 93c63e18
93c63e18
3e20: 2bdbf361 93c63e48 93c63ee8 93c63ee8 00000000 bc023d64 bc023c88
801f8f40
3e40: 00000000 800c5c24 91827364 93c63e4c 93c63e4c 93c63e54 93c63e54
00000000
3e60: 00000000 bc023c88 bc023c9c 800c68f0 00000000 80114754 bc023cf4
ba00c508
3e80: bfff0800 93c63ee8 00000000 ba00c540 bc023c88 80114b40 8c011000
bc023c9c
3ea0: bc5406d0 bfff0800 ba00c508 bc560cd0 bfff0840 ba00c540 ba00c558
93c63ee8
3ec0: 00000000 80115004 00000048 93c63f4c ba00c508 00000400 00000000
8073e2f0
3ee0: 00000000 801152f8 00000000 00000000 008ccb0e 00000248 00000000
00000000
3f00: 00000000 00000000 00000000 00000000 00000048 00000000 800371b4
93c63f64
3f20: 000002fa 000005f5 008cca6e ba00c548 ba00c550 00000000 ba00c468
ba00c558
3f40: ba00c508 80115490 93c63f58 7fffffff 00000000 00000000 0000000c
00000000
3f60: 00000000 00000000 000002fa 000005f5 ffffffff 93c62000 ba00c508
ba00c468
3f80: ba00c558 00000001 806e0080 ba00c51c 80705188 801155c4 00000000
00000000
3fa0: 93c63fc4 bfde9f10 ba00c508 80115544 00000013 00000000 00000000
00000000
3fc0: 00000000 8008bb70 8003faa4 00000000 ba00c508 00000000 00000000
00000000
3fe0: 93c63fe0 93c63fe0 bfde9f10 8008baf0 8003faa4 8003faa4 e13c8138
59101cc7
[<801fbe44>] (blk_rq_map_sg+0xd0/0x2cc) from [<8028dc20>]
(scsi_init_sgtable+0x50/0xb0)
[<8028dc20>] (scsi_init_sgtable+0x50/0xb0) from [<8028de1c>]
(scsi_init_io+0x1c/0xb4)
[<8028de1c>] (scsi_init_io+0x1c/0xb4) from [<802a0e54>]
(sd_prep_fn+0x104/0xb70)
[<802a0e54>] (sd_prep_fn+0x104/0xb70) from [<801f8818>]
(blk_peek_request+0xb0/0x1e0)
[<801f8818>] (blk_peek_request+0xb0/0x1e0) from [<8028eb78>]
(scsi_request_fn+0x48/0x3f0)
[<8028eb78>] (scsi_request_fn+0x48/0x3f0) from [<801f6ff8>]
(queue_unplugged.isra.26+0x24/0x44)
[<801f6ff8>] (queue_unplugged.isra.26+0x24/0x44) from [<801f8ee8>]
(blk_flush_plug_list+0x194/0x1dc)
[<801f8ee8>] (blk_flush_plug_list+0x194/0x1dc) from [<801f8f40>]
(blk_finish_plug+0x10/0x34)
[<801f8f40>] (blk_finish_plug+0x10/0x34) from [<800c5c24>]
(generic_writepages+0x4c/0x5c)
[<800c5c24>] (generic_writepages+0x4c/0x5c) from [<800c68f0>]
(do_writepages+0x24/0x38)
[<800c68f0>] (do_writepages+0x24/0x38) from [<80114754>]
(writeback_single_inode+0xc4/0x284)
[<80114754>] (writeback_single_inode+0xc4/0x284) from [<80114b40>]
(writeback_sb_inodes+0xa8/0x160)
[<80114b40>] (writeback_sb_inodes+0xa8/0x160) from [<80115004>]
(writeback_inodes_wb+0xc0/0x138)
[<80115004>] (writeback_inodes_wb+0xc0/0x138) from [<801152f8>]
(wb_writeback+0x27c/0x2d8)
[<801152f8>] (wb_writeback+0x27c/0x2d8) from [<80115490>]
(wb_do_writeback+0x13c/0x1f0)
[<80115490>] (wb_do_writeback+0x13c/0x1f0) from [<801155c4>]
(bdi_writeback_thread+0x80/0x168)
[<801155c4>] (bdi_writeback_thread+0x80/0x168) from [<8008bb70>]
(kthread+0x80/0x88)
[<8008bb70>] (kthread+0x80/0x88) from [<8003faa4>]
(kernel_thread_exit+0x0/0x8)
Code: e1a08002 e3530000 03a02000 120b2001 (e5989004)
INFO: rcu_preempt_state detected stalls on CPUs/tasks: { 1} (detected by 2,
t=6002 jiffies)
INFO: rcu_bh_state detected stalls on CPUs/tasks: { 1} (detected by 3,
t=6002 jiffies)
--
You received this message because you are subscribed to the Google Groups
"open-iscsi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/open-iscsi.
For more options, visit https://groups.google.com/groups/opt_out.
