Consistent reproducer:
inaddy@iscsixfs:~$ uname -a
Linux iscsixfs 4.4.0-87-generic #110~14.04.1-Ubuntu SMP Tue Jul 18 14:51:32 UTC
2017 x86_64 x86_64 x86_64 GNU/Linux
inaddy@iscsixfs:~$ dmesg | grep xfs
[ 381.648984] XFS (sda1): xfs_do_force_shutdown(0x2) called from line 1197 of
file /build/linux-lts-xenial-N0BD4p/linux-lts-xenial-4.4.0/fs/xfs/xfs_log.c.
Return address = 0xffffffffc02a16a8
[ 381.700064] XFS (sda1): xfs_log_force: error -5 returned.
inaddy@iscsixfs:~$ lsmod | grep xfs
xfs 954368 1
libcrc32c 16384 1 xfs
inaddy@iscsixfs:~$ sudo umount /xfs01
[ 449.785939] sd 2:0:0:1: rejecting I/O to offline device
[ 449.789923] blk_update_request: I/O error, dev sda, sector 0
inaddy@iscsixfs:~$ sudo rmmod xfs
[ 452.928198]
=============================================================================
[ 452.929647] BUG xfs_log_ticket (Not tainted): Objects remaining in
xfs_log_ticket on kmem_cache_close()
[ 452.931259]
-----------------------------------------------------------------------------
[ 452.931259]
[ 452.933002] INFO: Slab 0xffffea0002e67a80 objects=22 used=3
fp=0xffff8800b99eaf18 flags=0xffffc000000080
[ 452.934773] INFO: Object 0xffff8800b99ea000 @offset=0
[ 452.935693] INFO: Object 0xffff8800b99ea228 @offset=552
[ 452.936675] INFO: Object 0xffff8800b99ea2e0 @offset=736
[ 452.937642] kmem_cache_destroy xfs_log_ticket: Slab cache still has objects
** Description changed:
[Impact]
- * xfs kernel memory leak in case of xfs shutdown due to i/o errors
- * if xfs on iscsi, iscsi disconnection and module unload will case mem leak
+ * xfs kernel memory leak in case of xfs shutdown due to i/o errors
+ * if xfs on iscsi, iscsi disconnection and module unload will case mem leak
[Test Case]
- * configure tgtd with 1 lun and make it available through tcp/ip
- * configure open-iscsi to map this lun with a timeout >= 1
- * in tgtd server, drop iscsi packets and watch client to have i/o errors
- * after sometime (depending on timeout) xfs will call shutdown
- * after shutdown you try to remove xfs module and it will leak
+ * configure tgtd with 1 lun and make it available through tcp/ip
+ * configure open-iscsi to map this lun
+ * make sure node.session.timeo.replacement_timeout = 0 in iscsid.conf
+ * mount a xfs volume using the lun from tgtd host, run bonnie -d /xfsdir
+ * in tgtd server, drop iscsi packets and watch client to have i/o errors
+ * after sometime (depending on timeout) xfs will call shutdown
+ * make sure the i/o errors led to xfs shutdown (comment #3)
+ * after shutdown you try to remove xfs module and it will leak
[Regression Potential]
- * based on upstream fix
- * tested in the same environment
- * potential damage to xfs
+ * based on upstream fix
+ * tested in the same environment
+ * potential damage to xfs
[Other Info]
Original Description:
#### This scenario is testing [iscsi <-> scsi <-> disk <-> xfs]
[ 551.125604] sd 2:0:0:1: rejecting I/O to offline device
[ 551.125615] sd 2:0:0:1: rejecting I/O to offline device
[ 551.125627] sd 2:0:0:1: rejecting I/O to offline device
[ 551.125639] sd 2:0:0:1: rejecting I/O to offline device
[ 551.135216] XFS (sda1): metadata I/O error: block 0xeffe01
("xfs_trans_read_buf_map") error 5 numblks 1
[ 551.135274] XFS (sda1): page discard on page ffffea0002a89cc0, inode 0x83,
offset 6442385408.
# when XFS shuts down because of an error (or offline disk, example):
[ 551.850498] XFS (sda1): xfs_do_force_shutdown(0x2) called from line 1197
of file /build/linux-lts-xenial-roXrYH/linux-lts-xenial-4.4.0/fs/xfs/xfs_log.c.
Return address = 0xffffffffc0300388
[ 551.850568] XFS (sda1): Log I/O Error Detected. Shutting down filesystem
[ 551.850618] XFS (sda1): xfs_log_force: error -5 returned.
[ 551.850630] XFS (sda1): Failing async write on buffer block 0x77ff08.
Retrying async write.
[ 551.850634] XFS (sda1): Failing async write on buffer block 0x77ff10.
Retrying async write.
[ 551.850638] XFS (sda1): Failing async write on buffer block 0x77ff01.
Retrying async write.
[ 551.853171] XFS (sda1): Please umount the filesystem and rectify the
problem(s)
[ 551.874131] XFS (sda1): metadata I/O error: block 0x1dffc49
("xlog_iodone") error 5 numblks 64
[ 551.877993] XFS (sda1): xfs_do_force_shutdown(0x2) called from line 1197
of file /build/linux-lts-xenial-roXrYH/linux-lts-xenial-4.4.0/fs/xfs/xfs_log.c.
Return address = 0xffffffffc0300388
[ 551.899036] XFS (sda1): xfs_log_force: error -5 returned.
[ 569.323074] XFS (sda1): xfs_log_force: error -5 returned.
[ 599.403085] XFS (sda1): xfs_log_force: error -5 returned.
[ 629.483111] XFS (sda1): xfs_log_force: error -5 returned.
[ 659.563115] XFS (sda1): xfs_log_force: error -5 returned.
[ 689.643014] XFS (sda1): xfs_log_force: error -5 returned.
# when I execute:
# sudo umount /dev/sda1:
[81634.923043] XFS (sda1): xfs_log_force: error -5 returned.
[81640.739097] XFS (sda1): xfs_log_force: error -5 returned.
[81640.739137] XFS (sda1): Unmounting Filesystem
[81640.739463] XFS (sda1): xfs_log_force: error -5 returned.
[81640.739508] XFS (sda1): xfs_log_force: error -5 returned.
[81640.742741] sd 2:0:0:1: rejecting I/O to offline device
[81640.745576] blk_update_request: 25 callbacks suppressed
[81640.745601] blk_update_request: I/O error, dev sda, sector 0
# i was able to umount and then to remove iscsi disk.
# but if i try to unload the xfs module:
inaddy@(trustyiscsicli):~$ sudo rmmod xfs
[82211.059301]
=============================================================================
[82211.063764] BUG xfs_log_ticket (Tainted: G OE ): Objects
remaining in xfs_log_ticket on kmem_cache_close()
[82211.067450]
-----------------------------------------------------------------------------
[82211.067450]
[82211.070580] INFO: Slab 0xffffea0002eb7640 objects=22 used=1
fp=0xffff8800badd9f18 flags=0xffffc000000080
[82211.074430] INFO: Object 0xffff8800badd9228 @offset=552
[82211.076133] kmem_cache_destroy xfs_log_ticket: Slab cache still has objects
AND
[82211.059301]
=============================================================================
[82211.063764] BUG xfs_log_ticket (Tainted: G OE ): Objects
remaining in xfs_log_ticket on kmem_cache_close()
[82211.067450]
-----------------------------------------------------------------------------
[82211.067450]
[82211.070570] Disabling lock debugging due to kernel taint
[82211.070580] INFO: Slab 0xffffea0002eb7640 objects=22 used=1
fp=0xffff8800badd9f18 flags=0xffffc000000080
[82211.073964] CPU: 3 PID: 32230 Comm: rmmod Tainted: G B OE
4.4.0-74-generic #95~14.04.1-Ubuntu
[82211.073970] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
Ubuntu-1.8.2-1ubuntu1 04/01/2014
[82211.073975] 0000000000000000 ffff8800baf53d28 ffffffff813dce3c
ffffea0002eb7640
[82211.073984] ffff880036614300 ffff8800baf53e00 ffffffff811dc4b4
ffff880000000020
[82211.073991] ffff8800baf53e10 ffff8800baf53dc0 656a624f02e294c0
616d657220737463
[82211.074013] Call Trace:
[82211.074039] [<ffffffff813dce3c>] dump_stack+0x63/0x87
[82211.074066] [<ffffffff811dc4b4>] slab_err+0xb4/0xe0
[82211.074081] [<ffffffff8118cfd2>] ? __free_kmem_pages+0x22/0x40
[82211.074089] [<ffffffff811dcc6c>] ? __free_slab+0xcc/0x1a0
[82211.074097] [<ffffffff811df799>] ? __kmalloc+0x229/0x270
[82211.074106] [<ffffffff811e14b1>] ? __kmem_cache_shutdown+0x111/0x2b0
[82211.074127] [<ffffffff811e14d1>] __kmem_cache_shutdown+0x131/0x2b0
[82211.074157] [<ffffffff811a85ef>] kmem_cache_destroy+0x1bf/0x220
[82211.074289] [<ffffffffc031a368>] xfs_destroy_zones+0xeb/0x10a [xfs]
[82211.074382] [<ffffffffc031a3fc>] exit_xfs_fs+0x6f/0xc73 [xfs]
[82211.074395] [<ffffffff81101f3d>] SyS_delete_module+0x18d/0x220
[82211.074404] [<ffffffff8107986b>] ? exit_to_usermode_loop+0x78/0xa2
[82211.074416] [<ffffffff818066b6>] entry_SYSCALL_64_fastpath+0x16/0x75
[82211.074430] INFO: Object 0xffff8800badd9228 @offset=552
[82211.076133] kmem_cache_destroy xfs_log_ticket: Slab cache still has objects
[82211.078221] CPU: 3 PID: 32230 Comm: rmmod Tainted: G B OE
4.4.0-74-generic #95~14.04.1-Ubuntu
[82211.078226] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
Ubuntu-1.8.2-1ubuntu1 04/01/2014
[82211.078230] 0000000000000000 ffff8800baf53e60 ffffffff813dce3c
ffff880036614300
[82211.078238] ffff880036614300 ffff8800baf53ec8 ffffffff811a8648
0000000000000000
[82211.078245] 00ff8800baf53eb8 ffff8800baf53e80 ffff8800baf53e80
ffff8800baf53e90
[82211.078253] Call Trace:
[82211.078262] [<ffffffff813dce3c>] dump_stack+0x63/0x87
[82211.078271] [<ffffffff811a8648>] kmem_cache_destroy+0x218/0x220
[82211.078334] [<ffffffffc031a368>] xfs_destroy_zones+0xeb/0x10a [xfs]
[82211.078382] [<ffffffffc031a3fc>] exit_xfs_fs+0x6f/0xc73 [xfs]
[82211.078389] [<ffffffff81101f3d>] SyS_delete_module+0x18d/0x220
[82211.078395] [<ffffffff8107986b>] ? exit_to_usermode_loop+0x78/0xa2
[82211.078401] [<ffffffff818066b6>] entry_SYSCALL_64_fastpath+0x16/0x75
[82269.173849] SGI XFS with ACLs, security attributes, realtime, no debug
enabled
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1706132
Title:
xfs slab objects (memory) leak when xfs shutdown is called
Status in linux package in Ubuntu:
Confirmed
Bug description:
[Impact]
* xfs kernel memory leak in case of xfs shutdown due to i/o errors
* if xfs on iscsi, iscsi disconnection and module unload will case mem leak
[Test Case]
* configure tgtd with 1 lun and make it available through tcp/ip
* configure open-iscsi to map this lun
* make sure node.session.timeo.replacement_timeout = 0 in iscsid.conf
* mount a xfs volume using the lun from tgtd host, run bonnie -d /xfsdir
* in tgtd server, drop iscsi packets and watch client to have i/o errors
* after sometime (depending on timeout) xfs will call shutdown
* make sure the i/o errors led to xfs shutdown (comment #3)
* after shutdown you try to remove xfs module and it will leak
[Regression Potential]
* based on upstream fix
* tested in the same environment
* potential damage to xfs
[Other Info]
Original Description:
#### This scenario is testing [iscsi <-> scsi <-> disk <-> xfs]
[ 551.125604] sd 2:0:0:1: rejecting I/O to offline device
[ 551.125615] sd 2:0:0:1: rejecting I/O to offline device
[ 551.125627] sd 2:0:0:1: rejecting I/O to offline device
[ 551.125639] sd 2:0:0:1: rejecting I/O to offline device
[ 551.135216] XFS (sda1): metadata I/O error: block 0xeffe01
("xfs_trans_read_buf_map") error 5 numblks 1
[ 551.135274] XFS (sda1): page discard on page ffffea0002a89cc0, inode 0x83,
offset 6442385408.
# when XFS shuts down because of an error (or offline disk, example):
[ 551.850498] XFS (sda1): xfs_do_force_shutdown(0x2) called from line 1197
of file /build/linux-lts-xenial-roXrYH/linux-lts-xenial-4.4.0/fs/xfs/xfs_log.c.
Return address = 0xffffffffc0300388
[ 551.850568] XFS (sda1): Log I/O Error Detected. Shutting down filesystem
[ 551.850618] XFS (sda1): xfs_log_force: error -5 returned.
[ 551.850630] XFS (sda1): Failing async write on buffer block 0x77ff08.
Retrying async write.
[ 551.850634] XFS (sda1): Failing async write on buffer block 0x77ff10.
Retrying async write.
[ 551.850638] XFS (sda1): Failing async write on buffer block 0x77ff01.
Retrying async write.
[ 551.853171] XFS (sda1): Please umount the filesystem and rectify the
problem(s)
[ 551.874131] XFS (sda1): metadata I/O error: block 0x1dffc49
("xlog_iodone") error 5 numblks 64
[ 551.877993] XFS (sda1): xfs_do_force_shutdown(0x2) called from line 1197
of file /build/linux-lts-xenial-roXrYH/linux-lts-xenial-4.4.0/fs/xfs/xfs_log.c.
Return address = 0xffffffffc0300388
[ 551.899036] XFS (sda1): xfs_log_force: error -5 returned.
[ 569.323074] XFS (sda1): xfs_log_force: error -5 returned.
[ 599.403085] XFS (sda1): xfs_log_force: error -5 returned.
[ 629.483111] XFS (sda1): xfs_log_force: error -5 returned.
[ 659.563115] XFS (sda1): xfs_log_force: error -5 returned.
[ 689.643014] XFS (sda1): xfs_log_force: error -5 returned.
# when I execute:
# sudo umount /dev/sda1:
[81634.923043] XFS (sda1): xfs_log_force: error -5 returned.
[81640.739097] XFS (sda1): xfs_log_force: error -5 returned.
[81640.739137] XFS (sda1): Unmounting Filesystem
[81640.739463] XFS (sda1): xfs_log_force: error -5 returned.
[81640.739508] XFS (sda1): xfs_log_force: error -5 returned.
[81640.742741] sd 2:0:0:1: rejecting I/O to offline device
[81640.745576] blk_update_request: 25 callbacks suppressed
[81640.745601] blk_update_request: I/O error, dev sda, sector 0
# i was able to umount and then to remove iscsi disk.
# but if i try to unload the xfs module:
inaddy@(trustyiscsicli):~$ sudo rmmod xfs
[82211.059301]
=============================================================================
[82211.063764] BUG xfs_log_ticket (Tainted: G OE ): Objects
remaining in xfs_log_ticket on kmem_cache_close()
[82211.067450]
-----------------------------------------------------------------------------
[82211.067450]
[82211.070580] INFO: Slab 0xffffea0002eb7640 objects=22 used=1
fp=0xffff8800badd9f18 flags=0xffffc000000080
[82211.074430] INFO: Object 0xffff8800badd9228 @offset=552
[82211.076133] kmem_cache_destroy xfs_log_ticket: Slab cache still has objects
AND
[82211.059301]
=============================================================================
[82211.063764] BUG xfs_log_ticket (Tainted: G OE ): Objects
remaining in xfs_log_ticket on kmem_cache_close()
[82211.067450]
-----------------------------------------------------------------------------
[82211.067450]
[82211.070570] Disabling lock debugging due to kernel taint
[82211.070580] INFO: Slab 0xffffea0002eb7640 objects=22 used=1
fp=0xffff8800badd9f18 flags=0xffffc000000080
[82211.073964] CPU: 3 PID: 32230 Comm: rmmod Tainted: G B OE
4.4.0-74-generic #95~14.04.1-Ubuntu
[82211.073970] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
Ubuntu-1.8.2-1ubuntu1 04/01/2014
[82211.073975] 0000000000000000 ffff8800baf53d28 ffffffff813dce3c
ffffea0002eb7640
[82211.073984] ffff880036614300 ffff8800baf53e00 ffffffff811dc4b4
ffff880000000020
[82211.073991] ffff8800baf53e10 ffff8800baf53dc0 656a624f02e294c0
616d657220737463
[82211.074013] Call Trace:
[82211.074039] [<ffffffff813dce3c>] dump_stack+0x63/0x87
[82211.074066] [<ffffffff811dc4b4>] slab_err+0xb4/0xe0
[82211.074081] [<ffffffff8118cfd2>] ? __free_kmem_pages+0x22/0x40
[82211.074089] [<ffffffff811dcc6c>] ? __free_slab+0xcc/0x1a0
[82211.074097] [<ffffffff811df799>] ? __kmalloc+0x229/0x270
[82211.074106] [<ffffffff811e14b1>] ? __kmem_cache_shutdown+0x111/0x2b0
[82211.074127] [<ffffffff811e14d1>] __kmem_cache_shutdown+0x131/0x2b0
[82211.074157] [<ffffffff811a85ef>] kmem_cache_destroy+0x1bf/0x220
[82211.074289] [<ffffffffc031a368>] xfs_destroy_zones+0xeb/0x10a [xfs]
[82211.074382] [<ffffffffc031a3fc>] exit_xfs_fs+0x6f/0xc73 [xfs]
[82211.074395] [<ffffffff81101f3d>] SyS_delete_module+0x18d/0x220
[82211.074404] [<ffffffff8107986b>] ? exit_to_usermode_loop+0x78/0xa2
[82211.074416] [<ffffffff818066b6>] entry_SYSCALL_64_fastpath+0x16/0x75
[82211.074430] INFO: Object 0xffff8800badd9228 @offset=552
[82211.076133] kmem_cache_destroy xfs_log_ticket: Slab cache still has objects
[82211.078221] CPU: 3 PID: 32230 Comm: rmmod Tainted: G B OE
4.4.0-74-generic #95~14.04.1-Ubuntu
[82211.078226] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
Ubuntu-1.8.2-1ubuntu1 04/01/2014
[82211.078230] 0000000000000000 ffff8800baf53e60 ffffffff813dce3c
ffff880036614300
[82211.078238] ffff880036614300 ffff8800baf53ec8 ffffffff811a8648
0000000000000000
[82211.078245] 00ff8800baf53eb8 ffff8800baf53e80 ffff8800baf53e80
ffff8800baf53e90
[82211.078253] Call Trace:
[82211.078262] [<ffffffff813dce3c>] dump_stack+0x63/0x87
[82211.078271] [<ffffffff811a8648>] kmem_cache_destroy+0x218/0x220
[82211.078334] [<ffffffffc031a368>] xfs_destroy_zones+0xeb/0x10a [xfs]
[82211.078382] [<ffffffffc031a3fc>] exit_xfs_fs+0x6f/0xc73 [xfs]
[82211.078389] [<ffffffff81101f3d>] SyS_delete_module+0x18d/0x220
[82211.078395] [<ffffffff8107986b>] ? exit_to_usermode_loop+0x78/0xa2
[82211.078401] [<ffffffff818066b6>] entry_SYSCALL_64_fastpath+0x16/0x75
[82269.173849] SGI XFS with ACLs, security attributes, realtime, no debug
enabled
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1706132/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
