Hi,
During some testing, I hit an issue during unmount that was seems quite
similar to the problem I reported a few weeks ago (BUG during umount() after
withdrawal). I am using 5.7.1 which includes the patches for the earlier issue.
Here is the log:
[ 304.212698] connection1:0: ping timeout of 5 secs expired, recv timeout 5,
last rx 4294965568, last ping 4294966848, now 4294968128
[ 304.212834] connection1:0: detected conn error (1022)
[ 332.856419] sd 8:0:0:0: rejecting I/O to offline device
[ 332.856473] blk_update_request: I/O error, dev sdb, sector 2168544 op
0x0:(READ) flags 0x3000 phys_seg 1 prio class 0
[ 332.856490] blk_update_request: I/O error, dev sdb, sector 47608 op
0x1:(WRITE) flags 0x0 phys_seg 2 prio class 0
[ 332.856500] gfs2: fsid=xapi-clusterd:16d3f769-afce-7d.0: Error 10 writing to
journal, jid=0
[ 332.856525] gfs2: fsid=xapi-clusterd:16d3f769-afce-7d.0: fatal: I/O error(s)
[ 332.856537] gfs2: fsid=xapi-clusterd:16d3f769-afce-7d.0: about to withdraw
this file system
[ 332.856539] gfs2: fsid=xapi-clusterd:16d3f769-afce-7d.0: dirty_inode:
gfs2_trans_begin -30
[ 332.856571] gfs2: fsid=xapi-clusterd:16d3f769-afce-7d.0: The logd daemon is
withdrawing.
[ 332.856598] gfs2: fsid=xapi-clusterd:16d3f769-afce-7d.0: Requesting recovery
of jid 0.
[ 332.857124] sd 8:0:0:0: [sdb] Synchronizing SCSI cache
[ 332.857157] sd 8:0:0:0: [sdb] Synchronize Cache(10) failed: Result:
hostbyte=DID_TRANSPORT_FAILFAST driverbyte=DRIVER_OK
[ 332.920535] gfs2: fsid=xapi-clusterd:16d3f769-afce-7d.0: No other mounters
found. Trying to recover our own journal jid 0.
[ 332.920540] gfs2: fsid=xapi-clusterd:16d3f769-afce-7d.0: jid=0: Recovery not
attempted due to withdraw.
[ 332.920641] gfs2: fsid=xapi-clusterd:16d3f769-afce-7d.0: recovery_result jid
0 ignored due to withdraw.
[ 332.920683] gfs2: fsid=xapi-clusterd:16d3f769-afce-7d.0: Reprocessing of jid
0 failed with -5.
[ 332.920684] gfs2: fsid=xapi-clusterd:16d3f769-afce-7d.0: Journal recovery
complete for jid 0.
[ 332.920684] gfs2: fsid=xapi-clusterd:16d3f769-afce-7d.0: Glock dequeues
delayed: 1
[ 332.920706] ------------[ cut here ]------------
[ 332.920708] bdi-block not registered
[ 332.920719] WARNING: CPU: 5 PID: 6236 at
/local/scratch/rossla/src/linux/fs/fs-writeback.c:2323
__mark_inode_dirty+0x269/0x360
[ 332.920719] Modules linked in: gfs2 iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi rpcsec_gss_krb5 auth_rpcgss oid_registry nfsv4 nfs lockd
grace md4 cmac nls_utf8 cifs libarc4 fscache libdes dlm bnx2fc cnic uio fcoe
libfcoe libfc scsi_transport_fc 8021q garp mrp stp llc openvswitch nsh
nf_conncount nf_nat ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_multiport
xt_conntrack nf_conntrack nf_defrag_ipv4 libcrc32c dm_multipath iptable_filter
sunrpc dm_mod intel_powerclamp crct10dif_pclmul crc32_pclmul
ghash_clmulni_intel aesni_intel crypto_simd cryptd sg glue_helper igb mei_me
video mei intel_rapl_perf ie31200_edac i2c_i801 acpi_power_meter
intel_pch_thermal ipmi_si backlight ipmi_devintf ipmi_msghandler ip_tables
x_tables hid_generic usbhid hid sd_mod t10_pi ahci libahci xhci_pci
intel_ish_ipc libata xhci_hcd intel_ishtp scsi_dh_rdac scsi_dh_hp_sw
scsi_dh_emc scsi_dh_alua scsi_mod xen_wdt ipv6 crc_ccitt nf_defrag_ipv6
[ 332.920763] CPU: 5 PID: 6236 Comm: python2 Not tainted 5.7.1.uup2 #2
[ 332.920764] Hardware name: Supermicro MBI-6119G-T4/B2SS1-F, BIOS 2.0a
06/10/2017
[ 332.920765] RIP: e030:__mark_inode_dirty+0x269/0x360
[ 332.920766] Code: f6 ff ff 48 89 c5 48 8b 00 f6 40 5c 02 75 1d 48 8b 55 08
83 e2 01 75 14 48 8b 70 50 48 c7 c7 1c 83 e8 81 31 c0 e8 17 26 df ff <0f> 0b 48
8b 05 5e bf d7 00 45 85 f6 49 89 84 24 c8 00 00 00 74 0f
[ 332.920767] RSP: e02b:ffffc90041653d28 EFLAGS: 00010282
[ 332.920768] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000001
[ 332.920768] RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff88811a159840
[ 332.920769] RBP: ffff888109674878 R08: 0000000000000422 R09: 0000000000000422
[ 332.920769] R10: 0000000000000028 R11: ffffffff82602544 R12: ffff888071c99ec0
[ 332.920769] R13: ffff888071c99f48 R14: 0000000000000000 R15: 0000000000000000
[ 332.920777] FS: 00007f77e53eb740(0000) GS:ffff88811a140000(0000)
knlGS:0000000000000000
[ 332.920778] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 332.920778] CR2: 00007fab6c9885b0 CR3: 000000007690a000 CR4: 0000000000040660
[ 332.920782] Call Trace:
[ 332.920786] generic_update_time+0x6b/0xd0
[ 332.920787] file_update_time+0xaa/0xf0
[ 332.920792] gfs2_set_aops+0x1ed7/0x2130 [gfs2]
[ 332.920794] ? get_signal+0x479/0x8f0
[ 332.920796] new_sync_write+0x11e/0x1a0
[ 332.920798] vfs_write+0xad/0x1a0
[ 332.920799] ? kern_select+0xc9/0x100
[ 332.920800] ksys_write+0x55/0xd0
[ 332.920802] do_syscall_64+0x4e/0x140
[ 332.920804] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 332.920805] RIP: 0033:0x7f77e4209cd0
[ 332.920806] Code: 73 01 c3 48 8b 0d c0 61 2d 00 f7 d8 64 89 01 48 83 c8 ff
c3 66 0f 1f 44 00 00 83 3d cd c2 2d 00 00 75 10 b8 01 00 00 00 0f 05 <48> 3d 01
f0 ff ff 73 31 c3 48 83 ec 08 e8 ee cb 01 00 48 89 04 24
[ 332.920807] RSP: 002b:00007ffe316f5108 EFLAGS: 00000246 ORIG_RAX:
0000000000000001
[ 332.920808] RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 00007f77e4209cd0
[ 332.920808] RDX: 0000000000000005 RSI: 00007f77e53f7000 RDI: 0000000000000003
[ 332.920808] RBP: 00007f77e53f7000 R08: 00007f77e53eb740 R09: 000000000073c0a0
[ 332.920822] R10: 000000000000006a R11: 0000000000000246 R12: 0000000000775250
[ 332.920823] R13: 0000000000000005 R14: 00007f77e5393050 R15: 00007f77e5295830
[ 332.920824] ---[ end trace 3cb072c1f481ae5c ]---
[ 332.921511] Buffer I/O error on dev sdb, logical block 262833, lost async
page write
[ 332.921537] ------------[ cut here ]------------
[ 332.921544] WARNING: CPU: 5 PID: 421 at
/local/scratch/rossla/src/linux/fs/gfs2/glops.c:189
gfs2_unregister_debugfs+0xafc/0xb10 [gfs2]
[ 332.921544] Modules linked in: gfs2 iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi rpcsec_gss_krb5 auth_rpcgss oid_registry nfsv4 nfs lockd
grace md4 cmac nls_utf8 cifs libarc4 fscache libdes dlm bnx2fc cnic uio fcoe
libfcoe libfc scsi_transport_fc 8021q garp mrp stp llc openvswitch nsh
nf_conncount nf_nat ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_multiport
xt_conntrack nf_conntrack nf_defrag_ipv4 libcrc32c dm_multipath iptable_filter
sunrpc dm_mod intel_powerclamp crct10dif_pclmul crc32_pclmul
ghash_clmulni_intel aesni_intel crypto_simd cryptd sg glue_helper igb mei_me
video mei intel_rapl_perf ie31200_edac i2c_i801 acpi_power_meter
intel_pch_thermal ipmi_si backlight ipmi_devintf ipmi_msghandler ip_tables
x_tables hid_generic usbhid hid sd_mod t10_pi ahci libahci xhci_pci
intel_ish_ipc libata xhci_hcd intel_ishtp scsi_dh_rdac scsi_dh_hp_sw
scsi_dh_emc scsi_dh_alua scsi_mod xen_wdt ipv6 crc_ccitt nf_defrag_ipv6
[ 332.921565] CPU: 5 PID: 421 Comm: kworker/5:1H Tainted: G W
5.7.1.uup2 #2
[ 332.921566] Hardware name: Supermicro MBI-6119G-T4/B2SS1-F, BIOS 2.0a
06/10/2017
[ 332.921571] Workqueue: glock_workqueue gfs2_glock_dq_uninit [gfs2]
[ 332.921573] RIP: e030:gfs2_unregister_debugfs+0xafc/0xb10 [gfs2]
[ 332.921574] Code: e8 59 66 01 00 48 89 ef e8 31 75 86 c0 66 90 5b 5d 44 89
e0 41 5c 41 5d c3 ba 01 00 00 00 48 89 de 31 ff e8 96 d0 ff ff 0f 0b <0f> 0b 89
c6 4c 89 ef e8 31 00 02 00 eb b4 66 0f 1f 44 00 00 0f 1f
[ 332.921575] RSP: e02b:ffffc9004050bde0 EFLAGS: 00010282
[ 332.921575] RAX: 00000000fffffffb RBX: ffff888108c6d570 RCX: 0000000000ffffff
[ 332.921576] RDX: 0000000000000001 RSI: 00000000000402b5 RDI: ffff888111574748
[ 332.921578] RBP: ffff888108c6d590 R08: ffff88807e5e6fe8 R09: 0000000001000000
[ 332.921579] R10: 0000000001000000 R11: 0000000000000228 R12: 00000000fffffffb
[ 332.921580] R13: ffff888111574748 R14: ffffffffc088c3a0 R15: ffff888108c6d590
[ 332.921590] FS: 00007f77e53eb740(0000) GS:ffff88811a140000(0000)
knlGS:0000000000000000
[ 332.921590] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 332.921591] CR2: 00007fab6c9885b0 CR3: 0000000002008000 CR4: 0000000000040660
[ 332.921598] Call Trace:
[ 332.921601] gfs2_glock_dq_uninit+0x4c6/0x9c0 [gfs2]
[ 332.921603] gfs2_glock_dq_uninit+0x908/0x9c0 [gfs2]
[ 332.921607] process_one_work+0x166/0x370
[ 332.921609] worker_thread+0x49/0x3e0
[ 332.921610] kthread+0xf8/0x130
[ 332.921611] ? rescuer_thread+0x330/0x330
[ 332.921612] ? kthread_bind+0x10/0x10
[ 332.921614] ret_from_fork+0x35/0x40
[ 332.921617] ---[ end trace 3cb072c1f481ae5d ]---
[ 332.921620] gfs2: fsid=xapi-clusterd:16d3f769-afce-7d.0: Error -5 syncing
glock
[ 332.921631] gfs2: fsid=xapi-clusterd:16d3f769-afce-7d.0: G: s:EX n:3/402b1
f:lDpiIqo t:UN d:UN/0 a:0 v:0 r:3 m:50
[ 332.921646] gfs2: fsid=xapi-clusterd:16d3f769-afce-7d.0: R: n:262833
f:b0000000 b:57279/57279 i:107 r:0 e:57311
[ 332.921733] ------------[ cut here ]------------
[ 332.921734] kernel BUG at /local/scratch/rossla/src/linux/fs/inode.c:1588!
[ 332.921746] invalid opcode: 0000 [#1] SMP NOPTI
[ 332.921751] CPU: 5 PID: 6236 Comm: python2 Tainted: G W
5.7.1.uup2 #2
[ 332.921758] Hardware name: Supermicro MBI-6119G-T4/B2SS1-F, BIOS 2.0a
06/10/2017
[ 332.921767] RIP: e030:iput+0x1a5/0x1b0
[ 332.921771] Code: 8b 85 98 00 00 00 eb a5 48 89 ef e8 15 fe ff ff 4c 89 e7
e8 dd d8 e5 ff 66 90 e9 d9 fe ff ff 0f 0b eb 86 0f 0b e9 26 ff ff ff <0f> 0b 66
0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 41 57 41 56 49 89
[ 332.921786] RSP: e02b:ffffc90041653c30 EFLAGS: 00010202
[ 332.921791] RAX: dead000000000122 RBX: ffff888004190240 RCX: 0000000080400001
[ 332.921798] RDX: ffff888004190e40 RSI: 0000000000000001 RDI: ffff888071c75c40
[ 332.921805] RBP: ffff888071c75c40 R08: 0000000000000001 R09: 0000000000000000
[ 332.921811] R10: ffff888004548140 R11: 0000000000000000 R12: ffff888111574558
[ 332.921818] R13: ffff888111574558 R14: ffff88810fe92000 R15: ffff88806fb9d7c0
[ 332.921830] FS: 00007f77e53eb740(0000) GS:ffff88811a140000(0000)
knlGS:0000000000000000
[ 332.921837] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 332.921843] CR2: 00007fab6c9885b0 CR3: 0000000002008000 CR4: 0000000000040660
[ 332.921853] Call Trace:
[ 332.921859] gfs2_jindex_free+0xd5/0x110 [gfs2]
[ 332.921866] gfs2_make_fs_ro+0x57e/0x5e0 [gfs2]
[ 332.921872] generic_shutdown_super+0x6c/0x120
[ 332.921878] kill_block_super+0x21/0x50
[ 332.921882] deactivate_locked_super+0x3a/0x70
[ 332.921891] cleanup_mnt+0xc7/0x100
[ 332.921896] task_work_run+0x6f/0xa0
[ 332.921901] do_exit+0x2a8/0xb60
[ 332.921906] do_group_exit+0x39/0xa0
[ 332.921910] get_signal+0x17f/0x8f0
[ 332.921915] do_signal+0x30/0x690
[ 332.921921] exit_to_usermode_loop+0x5e/0xba
[ 332.921927] do_syscall_64+0x105/0x140
[ 332.921931] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 332.921936] RIP: 0033:0x7f77e4209cd0
[ 332.921945] Code: Bad RIP value.
[ 332.921948] RSP: 002b:00007ffe316f5108 EFLAGS: 00000246 ORIG_RAX:
0000000000000001
[ 332.921955] RAX: fffffffffffffffb RBX: 0000000000000005 RCX: 00007f77e4209cd0
[ 332.921962] RDX: 0000000000000005 RSI: 00007f77e53f7000 RDI: 0000000000000003
[ 332.921969] RBP: 00007f77e53f7000 R08: 00007f77e53eb740 R09: 000000000073c0a0
[ 332.921975] R10: 000000000000006a R11: 0000000000000246 R12: 0000000000775250
[ 332.921982] R13: 0000000000000005 R14: 00007f77e5393050 R15: 00007f77e5295830
[ 332.921989] Modules linked in: gfs2 iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi rpcsec_gss_krb5 auth_rpcgss oid_registry nfsv4 nfs lockd
grace md4 cmac nls_utf8 cifs libarc4 fscache libdes dlm bnx2fc cnic uio fcoe
libfcoe libfc scsi_transport_fc 8021q garp mrp stp llc openvswitch nsh
nf_conncount nf_nat ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_multiport
xt_conntrack nf_conntrack nf_defrag_ipv4 libcrc32c dm_multipath iptable_filter
sunrpc dm_mod intel_powerclamp crct10dif_pclmul crc32_pclmul
ghash_clmulni_intel aesni_intel crypto_simd cryptd sg glue_helper igb mei_me
video mei intel_rapl_perf ie31200_edac i2c_i801 acpi_power_meter
intel_pch_thermal ipmi_si backlight ipmi_devintf ipmi_msghandler ip_tables
x_tables hid_generic usbhid hid sd_mod t10_pi ahci libahci xhci_pci
intel_ish_ipc libata xhci_hcd intel_ishtp scsi_dh_rdac scsi_dh_hp_sw
scsi_dh_emc scsi_dh_alua scsi_mod xen_wdt ipv6 crc_ccitt nf_defrag_ipv6
[ 332.922067] ---[ end trace 3cb072c1f481ae5e ]---
[ 334.650814] RIP: e030:iput+0x1a5/0x1b0
[ 334.650865] Code: 8b 85 98 00 00 00 eb a5 48 89 ef e8 15 fe ff ff 4c 89 e7
e8 dd d8 e5 ff 66 90 e9 d9 fe ff ff 0f 0b eb 86 0f 0b e9 26 ff ff ff <0f> 0b 66
0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 41 57 41 56 49 89
[ 334.650880] RSP: e02b:ffffc90041653c30 EFLAGS: 00010202
[ 334.650886] RAX: dead000000000122 RBX: ffff888004190240 RCX: 0000000080400001
[ 334.650892] RDX: ffff888004190e40 RSI: 0000000000000001 RDI: ffff888071c75c40
[ 334.650899] RBP: ffff888071c75c40 R08: 0000000000000001 R09: 0000000000000000
[ 334.650909] R10: ffff888004548140 R11: 0000000000000000 R12: ffff888111574558
[ 334.650916] R13: ffff888111574558 R14: ffff88810fe92000 R15: ffff88806fb9d7c0
[ 334.650928] FS: 00007f77e53eb740(0000) GS:ffff88811a140000(0000)
knlGS:0000000000000000
[ 334.650935] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 334.650941] CR2: 00007f77e4209ca6 CR3: 0000000002008000 CR4: 0000000000040660
[ 334.650952] Fixing recursive fault but reboot is needed!
[ 334.679677] gfs2: fsid=xapi-clusterd:16d3f769-afce-7d.0: telling LM to
unmount
[ 334.679701] dlm: 16d3f769-afce-7d: leaving the lockspace group...
[ 334.680051] gfs2: fsid=xapi-clusterd:16d3f769-afce-7d.0: recover_prep
ignored due to withdraw.
[ 334.680152] dlm: 16d3f769-afce-7d: group event done 0 0
[ 334.680280] dlm: 16d3f769-afce-7d: release_lockspace final free
[ 334.680300] gfs2: fsid=xapi-clusterd:16d3f769-afce-7d.0: File system
withdrawn
[ 334.680308] CPU: 6 PID: 6098 Comm: gfs2_logd Tainted: G D W
5.7.1.uup2 #2
[ 334.680315] Hardware name: Supermicro MBI-6119G-T4/B2SS1-F, BIOS 2.0a
06/10/2017
[ 334.680322] Call Trace:
[ 334.680332] dump_stack+0x64/0x83
[ 334.680340] gfs2_withdraw+0x147/0x450 [gfs2]
[ 334.680350] gfs2_add_revoke+0x304/0x320 [gfs2]
[ 334.680361] gfs2_logd+0x334/0x14f0 [gfs2]
[ 334.680368] ? wait_woken+0x80/0x80
[ 334.680374] kthread+0xf8/0x130
[ 334.680380] ? gfs2_log_commit+0x350/0x350 [gfs2]
[ 334.680386] ? kthread_bind+0x10/0x10
[ 334.680395] ret_from_fork+0x35/0x40
[ 334.680416] BUG: kernel NULL pointer dereference, address: 000000000000005c
[ 334.680423] #PF: supervisor read access in kernel mode
[ 334.680428] #PF: error_code(0x0000) - not-present page
[ 334.680438] PGD 0 P4D 0
[ 334.680445] Oops: 0000 [#2] SMP NOPTI
[ 334.680452] CPU: 6 PID: 6098 Comm: gfs2_logd Tainted: G D W
5.7.1.uup2 #2
[ 334.680460] Hardware name: Supermicro MBI-6119G-T4/B2SS1-F, BIOS 2.0a
06/10/2017
[ 334.680469] RIP: e030:gfs2_logd+0x132/0x14f0 [gfs2]
[ 334.680475] Code: 00 00 41 8b be 38 08 00 00 41 8b b6 30 08 00 00 01 fa 29
ca 39 f2 0f 83 f6 01 00 00 49 8b 96 88 05 00 00 41 8b 8e 34 08 00 00 <8b> 52 5c
f0 49 0f ba 76 50 09 0f 82 a5 01 00 00 41 8b be 38 08 00
[ 334.680493] RSP: e02b:ffffc90041013ea8 EFLAGS: 00010282
[ 334.680499] RAX: 0000000000000001 RBX: ffff888111574050 RCX: 0000000000001ff1
[ 334.680509] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 334.680518] RBP: 0000000000000000 R08: ffffffff82263880 R09: ffffffff8226387c
[ 334.680525] R10: ffffffff818001fa R11: 0000000000000000 R12: ffff888111574858
[ 334.680532] R13: ffff888111574280 R14: ffff888111574000 R15: ffff888108cf1d40
[ 334.680546] FS: 00007f838bf868c0(0000) GS:ffff88811a180000(0000)
knlGS:0000000000000000
[ 334.680553] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 334.680559] CR2: 000000000000005c CR3: 0000000113c78000 CR4: 0000000000040660
[ 334.680570] Call Trace:
[ 334.680575] ? wait_woken+0x80/0x80
[ 334.680580] kthread+0xf8/0x130
[ 334.680585] ? gfs2_log_commit+0x350/0x350 [gfs2]
[ 334.680591] ? kthread_bind+0x10/0x10
[ 334.680596] ret_from_fork+0x35/0x40
[ 334.680601] Modules linked in: gfs2 iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi rpcsec_gss_krb5 auth_rpcgss oid_registry nfsv4 nfs lockd
grace md4 cmac nls_utf8 cifs libarc4 fscache libdes dlm bnx2fc cnic uio fcoe
libfcoe libfc scsi_transport_fc 8021q garp mrp stp llc openvswitch nsh
nf_conncount nf_nat ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_multiport
xt_conntrack nf_conntrack nf_defrag_ipv4 libcrc32c dm_multipath iptable_filter
sunrpc dm_mod intel_powerclamp crct10dif_pclmul crc32_pclmul
ghash_clmulni_intel aesni_intel crypto_simd cryptd sg glue_helper igb mei_me
video mei intel_rapl_perf ie31200_edac i2c_i801 acpi_power_meter
intel_pch_thermal ipmi_si backlight ipmi_devintf ipmi_msghandler ip_tables
x_tables hid_generic usbhid hid sd_mod t10_pi ahci libahci xhci_pci
intel_ish_ipc libata xhci_hcd intel_ishtp scsi_dh_rdac scsi_dh_hp_sw
scsi_dh_emc scsi_dh_alua scsi_mod xen_wdt ipv6 crc_ccitt nf_defrag_ipv6
[ 334.680681] CR2: 000000000000005c
[ 334.680685] ---[ end trace 3cb072c1f481ae5f ]---
[ 334.684212] RIP: e030:iput+0x1a5/0x1b0
[ 334.684261] Code: 8b 85 98 00 00 00 eb a5 48 89 ef e8 15 fe ff ff 4c 89 e7
e8 dd d8 e5 ff 66 90 e9 d9 fe ff ff 0f 0b eb 86 0f 0b e9 26 ff ff ff <0f> 0b 66
0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 41 57 41 56 49 89
[ 334.684278] RSP: e02b:ffffc90041653c30 EFLAGS: 00010202
[ 334.684283] RAX: dead000000000122 RBX: ffff888004190240 RCX: 0000000080400001
[ 334.684290] RDX: ffff888004190e40 RSI: 0000000000000001 RDI: ffff888071c75c40
[ 334.684296] RBP: ffff888071c75c40 R08: 0000000000000001 R09: 0000000000000000
[ 334.684303] R10: ffff888004548140 R11: 0000000000000000 R12: ffff888111574558
[ 334.684310] R13: ffff888111574558 R14: ffff88810fe92000 R15: ffff88806fb9d7c0
[ 334.684322] FS: 00007f838bf868c0(0000) GS:ffff88811a180000(0000)
knlGS:0000000000000000
[ 334.684329] CS: e030 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 334.684335] CR2: 000000000000005c CR3: 0000000113c78000 CR4: 0000000000040660
Steps to reproduce this:
Log into an iSCSI target:
$ iscsiadm -m discovery -t sendtargets -p 10.71.57.19
$ iscsiadm -m node -T iqn.2014-08.com.example:t1 -p 10.71.57.19 -l
Mount the GFS2 fs somewhere:
$ mount /dev/sdb /mnt
Run a program to write to disk slowly:
$ cat wedge.py
import time
import sys
f = open(sys.argv[1], 'w')
while True:
f.write('blah\n')
f.flush()
time.sleep(1)
$ python2 wedge.py /mnt/test.txt &
[1] 6872
Check that it is working:
$ cat /mnt/test2.txt
blah
$ cat /mnt/test2.txt
blah
blah
$ cat /mnt/test2.txt
blah
blah
blah
Kill the iSCSI connection:
$ iptables -I RH-Firewall-1-INPUT -s 10.71.57.19 -j DROP
Wait for the Python process to become blocked:
$ ~]# cat /proc/6872/stack
[<0>] gfs2_log_reserve+0x13d/0x2e0 [gfs2]
[<0>] gfs2_trans_begin+0xc5/0x140 [gfs2]
[<0>] gfs2_dinode_out+0x324/0x3c0 [gfs2]
[<0>] __mark_inode_dirty+0x17c/0x360
[<0>] generic_update_time+0x6b/0xd0
[<0>] file_update_time+0xaa/0xf0
[<0>] gfs2_set_aops+0x1ed7/0x2130 [gfs2]
[<0>] new_sync_write+0x11e/0x1a0
[<0>] vfs_write+0xad/0x1a0
[<0>] ksys_write+0x55/0xd0
[<0>] do_syscall_64+0x4e/0x140
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
Do a force, lazy unmount and logout of the iSCSI target:
$ umount -f -l /mnt; iscsiadm -m node -T iqn.2014-08.com.example:t1 -p
10.71.57.19 --logout
Logging out of session [sid: 1, target: iqn.2014-08.com.example:t1, portal:
10.71.57.19,3260]
Logout of [sid: 1, target: iqn.2014-08.com.example:t1, portal:
10.71.57.19,3260] successful.
Most of the time this results in the above errors (BUG() in iput() and NULL
pointer dereference in gfs2_logd()).
I think this may be happening because gfs2_logd does a withdraw which does
iput()
on the journal inode which evicts it. It then tries to reload it and fails
(because of IO errors?). When gfs2_put_super() is called, it does iput() on the
same inode again and hits the BUG().
Any ideas on this?
Thanks,
Ross
