[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
** Changed in: zfs-linux (Ubuntu) Assignee: Colin Ian King (colin-king) => Dimitri John Ledkov (xnox) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: zfs-linux (Ubuntu Bionic) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
This time I don't see any error or panic in `journalctl -b` and apparently everything is fine according to smartctl, so it looks like we are not getting any obvious I/O error at the moment. IIUC you have done already a `zpool scrub` and that also didn't report any error, so apparently the zfs data integrigy checks passed locally. If that's the case, yes, I think the next step would be to try to `zfs send | zfs recv` commands and see if we get more information with zfs debugging enabled. Alternatively we could try to stress the local zpool more, maybe running some I/O stress test (fio for example) on some volumes / filesystems created from the "potentially faulty" zpool. Apart than that I don't see any other useful test that we can do... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
My guess is the next step is trying the zfs send | ssh zfs recv commands that were causing the VERIFY(size != 0) failed message with traces. I've always been uneasy with the 'main' error coming only with a recv operation, relying upon two systems to recreate a bug is iffy at best. And since it wedges the one, I don't like doing it just for fun, but I'd also like to get to the bottom of this. Is there anything else to try first? Thanks -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
The scrub has finished, no new errors in zpool status -v, dmesg is filled with debugging messages, I hope something is useful. :) Thanks $ uname -a Linux wopr 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux $ modinfo zfs | grep -i version version:0.7.5-1ubuntu16.10+lp1861235 srcversion: 62D1F1B1A1EE81EB3AA8ABA $ zpool status -v pool: fst state: ONLINE scan: scrub repaired 0B in 2h37m with 0 errors on Sun Nov 8 03:01:53 2020 config: NAMESTATE READ WRITE CKSUM fst ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 wwn-0x5002538e4095da39 ONLINE 0 0 0 wwn-0x5002538e4095bdd6 ONLINE 0 0 0 wwn-0x5002538e4093c6fd ONLINE 0 0 0 wwn-0x5002538e4095da30 ONLINE 0 0 0 errors: No known data errors pool: srv state: ONLINE scan: scrub repaired 0B in 4h12m with 0 errors on Wed Dec 2 18:53:44 2020 config: NAME STATE READ WRITE CKSUM srv ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ata-HGST_HUS724030ALA640_PN2234P8KTWJYY ONLINE 0 0 0 ata-HGST_HUS724030ALA640_PN2234P9G620TW ONLINE 0 0 0 ata-HGST_HUS724030ALA640_PN2234P9G66E2U ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 ata-HGST_HUS724030ALA640_PN2234P9G69TKU ONLINE 0 0 0 ata-HGST_HUS724030ALA640_PN2234P9G69TXU ONLINE 0 0 0 ata-HGST_HUS724030ALA640_PN2234P9G69U2U ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 ata-HGST_HUS724030ALA640_PN2234P9G6EBUU ONLINE 0 0 0 ata-HGST_HUS724030ALA640_PN2234P9G6ESAU ONLINE 0 0 0 ata-HGST_HUS724030ALA640_PN2234P9G6G70U ONLINE 0 0 0 logs nvme0n1p1 ONLINE 0 0 0 cache nvme0n1p2 ONLINE 0 0 0 errors: No known data errors ** Attachment added: "journalctl -b output" https://bugs.launchpad.net/linux/+bug/1861235/+attachment/5440572/+files/journalctl-b -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
Here's smartctl output on the disks in the pool. It's a bit hard to summarize nine drives of smartctl output but nothing stood out as interesting to me. My journals only go back a few months so they wouldn't show any IO errors at the time of this problem being introduced, but this currently gives no results: journalctl | grep -i "i/o error" so there's nothing recent. I started a new scrub, it'll finish in a few hours, if I forget to post results please do remind me. :) Thanks ** Attachment added: "smartctl" https://bugs.launchpad.net/linux/+bug/1861235/+attachment/5440418/+files/smartctl -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
Picking the previous bad object (which I'm pretty sure should still be intact): $ modinfo zfs | grep ^version version:0.7.5-1ubuntu16.10+lp1861235 $ uname -a Linux wopr 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux $ sudo zdb - srv/backups/millbarge/rpool/var/log 529 Dataset srv/backups/millbarge/rpool/var/log [ZPL], ID 39694, cr_txg 23197757, 554M, 274 objects, rootbp DVA[0]=<1:1d000d42000:1000> DVA[1]=<2:24416f4b000:1000> [L0 DMU objset] sha256 uncompressed LE contiguous unique double size=800L/800P birth=23437543L/23437543P fill=274 cksum=cf5fcb2f986c768c:7cc8638aeb1a5162:9de9f236f072e16:40c6a08d90b54951 Object lvl iblk dblk dsize dnsize lsize %full type dn->dn_nblkptr >= 1 (0x0 >= 0x1) ASSERT at ../../module/zfs/dnode.c:278:dnode_verify()Aborted $ dmesg | tail [ 43.017302] Ebtables v2.0 registered [ 44.977437] L1TF CPU bug present and SMT on, data leak possible. See CVE-2018-3646 and https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/l1tf.html for details. [ 47.047114] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this. [ 47.047966] virbr0: port 1(virbr0-nic) entered blocking state [ 47.047968] virbr0: port 1(virbr0-nic) entered disabled state [ 47.048063] device virbr0-nic entered promiscuous mode [ 47.433533] nf_conntrack version 0.5.0 (65536 buckets, 262144 max) [ 47.971009] virbr0: port 1(virbr0-nic) entered blocking state [ 47.971013] virbr0: port 1(virbr0-nic) entered listening state [ 48.321853] virbr0: port 1(virbr0-nic) entered disabled state $ ** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2018-3646 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
Hello Andrea, thanks for looking into my problem. # zpool status -v pool: fst state: ONLINE scan: scrub repaired 0B in 2h37m with 0 errors on Sun Nov 8 03:01:53 2020 config: NAMESTATE READ WRITE CKSUM fst ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 wwn-0x5002538e4095da39 ONLINE 0 0 0 wwn-0x5002538e4095bdd6 ONLINE 0 0 0 wwn-0x5002538e4093c6fd ONLINE 0 0 0 wwn-0x5002538e4095da30 ONLINE 0 0 0 errors: No known data errors pool: srv state: ONLINE scan: scrub repaired 0B in 3h44m with 0 errors on Sun Nov 8 04:08:16 2020 config: NAME STATE READ WRITE CKSUM srv ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ata-HGST_HUS724030ALA640_PN2234P8KTWJYY ONLINE 0 0 0 ata-HGST_HUS724030ALA640_PN2234P9G620TW ONLINE 0 0 0 ata-HGST_HUS724030ALA640_PN2234P9G66E2U ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 ata-HGST_HUS724030ALA640_PN2234P9G69TKU ONLINE 0 0 0 ata-HGST_HUS724030ALA640_PN2234P9G69TXU ONLINE 0 0 0 ata-HGST_HUS724030ALA640_PN2234P9G69U2U ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 ata-HGST_HUS724030ALA640_PN2234P9G6EBUU ONLINE 0 0 0 ata-HGST_HUS724030ALA640_PN2234P9G6ESAU ONLINE 0 0 0 ata-HGST_HUS724030ALA640_PN2234P9G6G70U ONLINE 0 0 0 logs nvme0n1p1 ONLINE 0 0 0 cache nvme0n1p2 ONLINE 0 0 0 errors: No known data errors I'll swap over to your dkms build and report back. Thanks -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
I've uploaded the latest version of zfs with debugging enabled (--enable-debug) in this ppa: https://launchpad.net/~arighi/+archive/ubuntu/zfs-linux sudo add-apt-repository ppa:arighi/zfs-linux sudo apt-get update sudo apt-get install zfs-dkms It'd be interesting to repeat the test using this version. Make sure the loaded zfs module is matching the right version: $ modinfo zfs | grep ^version version:0.7.5-1ubuntu16.10+lp1861235 Moreover, if the data got corrupted, it'd be interesting to know why... can you provide some smartctl information? Did you notice any I/O error in your logs? Thanks! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
I just noticed the bug on github that also mentions about potential data corruption. It looks like we need to recompile zfs with --enable-debug. I'll try to produce a debug package with that. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
Hi @seth-arnold, adding some details to this bug: dn->dn_type = 19 = DMU_OT_PLAIN_FILE_CONTENTS dn->dn_type = 20 = DMU_OT_DIRECTORY_CONTENTS So they look correct, they seem to be just plain files and directories. However, the specific object that is causing the panic seems to have dn_type=32 that is DMU_OT_DSL_PERMS (that's a bit odd, I don't see any other object types like this in your dmesg), moreover the output of "zdb -" shows a lot of UNKNONW stuff (checksum and compression type for example, they are UNKNOWN). So, it seems that we may have some corrupted data here. Do you see any error if you try to run a simple `zpool status -v`? Do you know exactly which pool / volume / filesystem is causing this problem? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
Is it interesting that I'm seeing both 19 and 20 in my dmesg? [1229822.406130] dmu_object_free: object = 0x266d2904, dn->dn_type = 20 [1229823.980888] dmu_object_free: object = 0x266d0d5a, dn->dn_type = 20 [1229823.994690] dmu_object_free: object = 0x266d0d5b, dn->dn_type = 20 [1229823.998123] dmu_object_free: object = 0x266d00c1, dn->dn_type = 19 [1229823.998191] dmu_object_free: object = 0x266d00c3, dn->dn_type = 19 [1229823.998324] dmu_object_free: object = 0x266d00c5, dn->dn_type = 19 [1229823.998377] dmu_object_free: object = 0x266d00c4, dn->dn_type = 20 Thanks -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
I picked the last dataset given in the command output from an earlier, but not the most recent, comment: $ sudo zdb - srv/backups/millbarge/rpool/var/log 529 Dataset srv/backups/millbarge/rpool/var/log [ZPL], ID 39694, cr_txg 23197757, 554M, 274 objects, rootbp DVA[0]=<1:1d000d42000:1000> DVA[1]=<2:24416f4b000:1000> [L0 DMU objset] sha256 uncompressed LE contiguous unique double size=800L/800P birth=23437543L/23437543P fill=274 cksum=cf5fcb2f986c768c:7cc8638aeb1a5162:9de9f236f072e16:40c6a08d90b54951 Object lvl iblk dblk dsize dnsize lsize %full type 5290 0 0 0 512 0-nan DSL permissions (K=UNKNOWN) (Z=UNKNOWN) 0 bonus UNKNOWN dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED SPILL_BLKPTR dnode maxblkid: 0 UNKNOWN OBJECT TYPE c < (1ULL << 24) >> 9 (0x7f < 0x8000) ASSERT at ../../module/zfs/zio.c:266:zio_buf_alloc()Aborted Thanks -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
OK so object 0x211 looks like the curlprit, I just can't see the pool name being dumped - do you know which pool is triggering this issue? If you can identify the poolt then we can next dump the potentially corrupt object using: sudo zdb - / 529 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
Hi Seth, I've upload a .4 - it may add a lot more debug but it will allow us to see the object type that is failing before the crash. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
On Sat, May 16, 2020 at 01:56:08AM -, Seth Arnold wrote: > Sadly, journalctl doesn't have the dmesg from the previous boot: I meant to say, journalctl's copy of dmesg from the previous boot doesn't have the new debug output. Sorry. Thanks -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
Hello Colin, trying the zfs recv operation on the .3 dkms eventually kills my system dead. There was nothing on the console. The cursor on the console stopped blinking; I couldn't switch VTs. My ssh sessions were hung. ping reported destination host unreachable. It ran for about two and a half minutes, transferred around 10 datasets or snapshots: $ journalctl -b -1 | grep -c "COMMAND=/sbin/zfs receive" 10 Sadly, journalctl doesn't have the dmesg from the previous boot: $ journalctl -b -1 -k [...] May 15 17:25:10 wopr kernel: ZFS: Loaded module v0.7.5-1ubuntu16.10~lp1861235.3, ZFS pool version 5000, ZFS filesystem version 5 May 15 17:25:15 wopr kernel: zd32: p1 May 15 17:25:20 wopr kernel: pps pps0: new PPS source ptp0 May 15 17:25:20 wopr kernel: ixgbe :05:00.0: registered PHC device on enp5s0f0 May 15 17:25:20 wopr kernel: IPv6: ADDRCONF(NETDEV_UP): enp5s0f0: link is not ready May 15 17:25:20 wopr kernel: new mount options do not match the existing superblock, will be ignored May 15 17:25:21 wopr kernel: Process accounting resumed May 15 17:25:24 wopr kernel: ixgbe :05:00.0 enp5s0f0: NIC Link is Up 1 Gbps, Flow Control: None May 15 17:25:24 wopr kernel: IPv6: ADDRCONF(NETDEV_CHANGE): enp5s0f0: link becomes ready May 15 17:25:29 wopr kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory May 15 17:25:29 wopr kernel: NFSD: starting 90-second grace period (net f0a9) May 15 17:25:32 wopr kernel: ip6_tables: (C) 2000-2006 Netfilter Core Team May 15 17:25:32 wopr kernel: Ebtables v2.0 registered May 15 17:25:33 wopr kernel: bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this. May 15 17:25:33 wopr kernel: virbr0: port 1(virbr0-nic) entered blocking state May 15 17:25:33 wopr kernel: virbr0: port 1(virbr0-nic) entered disabled state May 15 17:25:33 wopr kernel: device virbr0-nic entered promiscuous mode May 15 17:25:33 wopr kernel: nf_conntrack version 0.5.0 (65536 buckets, 262144 max) May 15 17:25:33 wopr kernel: virbr0: port 1(virbr0-nic) entered blocking state May 15 17:25:33 wopr kernel: virbr0: port 1(virbr0-nic) entered listening state May 15 17:25:33 wopr kernel: virbr0: port 1(virbr0-nic) entered disabled state Thanks -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
Thanks for the data for the .2 tests. I've updated the packages with a .3 test build. I've added a couple more lines of debug now to figure out some earlier missing information once we pop the stack. I've updated the package in the PPA, do you mind updating the zfs-dkms to 0.7.5-1ubuntu16.10~lp1861235.3 and re-testing? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
Oh, I see you are using the latest version: v0.7.5-1ubuntu16.10~lp1861235.2 I may need to figure out a .3 version to try next. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
Hi, I was expecting some extra information and a panic message in a different place, are you sure this is running the latest zfs-dkms? One can check that by using dmesg | grep ZFS and check this is the .2 version of the debug zfd-dkms package. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
Thanks Colin! [ 271.628232] dnone_free_ramge: nblks = 0, trunc = 1, len = 18446744073709551615, blkshift = 0 [ 271.628297] dnode_free_range: nblks == 0, len == 18446744073709551615 , off=0 [ 271.628298] range_tree_clear: size == 0 [ 271.628375] range_tree_find_impl: size == 0 [ 271.628393] VERIFY(size != 0) failed [ 271.628409] PANIC at range_tree.c:168:range_tree_add() [ 271.628429] Showing stack for process 7960 [ 271.628432] CPU: 26 PID: 7960 Comm: receive_writer Tainted: P OE 4.15.0-99-generic #100-Ubuntu [ 271.628433] Hardware name: Supermicro SSG-6038R-E1CR16L/X10DRH-iT, BIOS 2.0 12/17/2015 [ 271.628433] Call Trace: [ 271.628442] dump_stack+0x6d/0x8e [ 271.628458] spl_dumpstack+0x42/0x50 [spl] [ 271.628462] spl_panic+0xc8/0x110 [spl] [ 271.628468] ? vt_console_print+0x240/0x410 [ 271.628473] ? console_unlock+0x2a1/0x560 [ 271.628475] ? vprintk_emit+0x104/0x2c0 [ 271.628477] ? vprintk_default+0x29/0x50 [ 271.628479] ? printk+0x52/0x6e [ 271.628566] range_tree_add+0x29b/0x300 [zfs] [ 271.628600] ? range_tree_clear+0xa4/0xb0 [zfs] [ 271.628602] ? printk+0x52/0x6e [ 271.628635] dnode_free_range+0x178/0x720 [zfs] [ 271.628673] dmu_object_free+0x6d/0xc0 [zfs] [ 271.628698] dmu_free_long_object+0x9f/0xc0 [zfs] [ 271.628723] receive_freeobjects.isra.12+0x7a/0x100 [zfs] [ 271.628747] receive_writer_thread+0x6d2/0xa60 [zfs] [ 271.628750] ? set_curr_task_fair+0x2b/0x60 [ 271.628753] ? spl_kmem_free+0x33/0x40 [spl] [ 271.628757] ? kfree+0x162/0x180 [ 271.628781] ? receive_free.isra.13+0xc0/0xc0 [zfs] [ 271.628785] thread_generic_wrapper+0x74/0x90 [spl] [ 271.628788] kthread+0x121/0x140 [ 271.628791] ? __thread_exit+0x20/0x20 [spl] [ 271.628792] ? kthread_create_worker_on_cpu+0x70/0x70 [ 271.628796] ret_from_fork+0x35/0x40 ** Attachment added: "dmesg-0.7.5-1ubuntu16.10~lp1861235.2" https://bugs.launchpad.net/linux/+bug/1861235/+attachment/5370673/+files/dmesg-0.7.5-1ubuntu16.10~lp1861235.2 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
Thanks for the data. I've added a couple more lines of debug now to figure out some earlier missing information once we pop the stack. I've updated the package in the PPA, do you mind updating the zfs-dkms to 0.7.5-1ubuntu16.10~lp1861235.2 and re-testing? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
Here's the part that looks important; the whole dmesg is in the attachment. Thanks [ 761.730488] dnone_free_ramge: nblks = 0, trunc = 1, len = 18446744073709551615, blkshift = 0 [ 761.730542] dnode_free_range: nblks == 0, len == 18446744073709551615 , off=0 [ 761.730543] range_tree_clear: size == 0 [ 761.730605] VERIFY(size != 0) failed [ 761.730625] PANIC at range_tree.c:307:range_tree_find_impl() [ 761.730653] Showing stack for process 23913 [ 761.730656] CPU: 31 PID: 23913 Comm: receive_writer Tainted: P OE 4.15.0-99-generic #100-Ubuntu [ 761.730657] Hardware name: Supermicro SSG-6038R-E1CR16L/X10DRH-iT, BIOS 2.0 12/17/2015 [ 761.730658] Call Trace: [ 761.730668] dump_stack+0x6d/0x8e [ 761.730679] spl_dumpstack+0x42/0x50 [spl] [ 761.730684] spl_panic+0xc8/0x110 [spl] [ 761.730690] ? fbcon_cursor+0x13e/0x1c0 [ 761.730695] ? atomic_notifier_call_chain+0x1a/0x20 [ 761.730700] ? vt_console_print+0x240/0x410 [ 761.730705] ? up+0x32/0x50 [ 761.730708] ? console_unlock+0x2a1/0x560 [ 761.730767] range_tree_find_impl+0x88/0x90 [zfs] [ 761.730769] ? printk+0x52/0x6e [ 761.730810] range_tree_clear+0x54/0x70 [zfs] [ 761.730843] dnode_free_range+0x15b/0x720 [zfs] [ 761.730874] dmu_object_free+0x75/0xc0 [zfs] [ 761.730904] dmu_free_long_object+0x9f/0xc0 [zfs] [ 761.730934] receive_freeobjects.isra.12+0x7a/0x100 [zfs] [ 761.730963] receive_writer_thread+0x6d2/0xa60 [zfs] [ 761.730965] ? set_curr_task_fair+0x2b/0x60 [ 761.730970] ? spl_kmem_free+0x33/0x40 [spl] [ 761.730973] ? kfree+0x162/0x180 [ 761.731001] ? receive_free.isra.13+0xc0/0xc0 [zfs] [ 761.731006] thread_generic_wrapper+0x74/0x90 [spl] [ 761.731008] kthread+0x121/0x140 [ 761.731012] ? __thread_exit+0x20/0x20 [spl] [ 761.731014] ? kthread_create_worker_on_cpu+0x70/0x70 [ 761.731018] ret_from_fork+0x35/0x40 [ 967.430681] INFO: task txg_quiesce:4049 blocked for more than 120 seconds. [ 967.430739] Tainted: P OE4.15.0-99-generic #100-Ubuntu [ 967.430785] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 967.430836] txg_quiesce D0 4049 2 0x8000 [ 967.430840] Call Trace: [ 967.430853] __schedule+0x24e/0x880 [ 967.430858] schedule+0x2c/0x80 [ 967.430875] cv_wait_common+0x11e/0x140 [spl] [ 967.430885] ? wait_woken+0x80/0x80 [ 967.430893] __cv_wait+0x15/0x20 [spl] [ 967.431010] txg_quiesce_thread+0x2cb/0x3d0 [zfs] [ 967.431078] ? txg_delay+0x1b0/0x1b0 [zfs] [ 967.431088] thread_generic_wrapper+0x74/0x90 [spl] [ 967.431093] kthread+0x121/0x140 [ 967.431099] ? __thread_exit+0x20/0x20 [spl] [ 967.431102] ? kthread_create_worker_on_cpu+0x70/0x70 [ 967.431107] ret_from_fork+0x35/0x40 [ 967.431140] INFO: task zfs:23834 blocked for more than 120 seconds. [ 967.431184] Tainted: P OE4.15.0-99-generic #100-Ubuntu [ 967.431229] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 967.431279] zfs D0 23834 23831 0x8080 [ 967.431282] Call Trace: [ 967.431286] __schedule+0x24e/0x880 [ 967.431290] schedule+0x2c/0x80 [ 967.431298] cv_wait_common+0x11e/0x140 [spl] [ 967.431303] ? wait_woken+0x80/0x80 [ 967.431310] __cv_wait+0x15/0x20 [spl] [ 967.431367] dmu_recv_stream+0xa51/0xef0 [zfs] [ 967.431438] zfs_ioc_recv_impl+0x306/0x1100 [zfs] [ 967.431487] ? dbuf_rele+0x36/0x40 [zfs] [ 967.431555] zfs_ioc_recv_new+0x33d/0x410 [zfs] [ 967.431564] ? spl_kmem_alloc_impl+0xe5/0x1a0 [spl] [ 967.431571] ? spl_vmem_alloc+0x19/0x20 [spl] [ 967.431584] ? nv_alloc_sleep_spl+0x1f/0x30 [znvpair] [ 967.431590] ? nv_mem_zalloc.isra.0+0x2e/0x40 [znvpair] [ 967.431596] ? nvlist_xalloc.part.2+0x50/0xb0 [znvpair] [ 967.431660] zfsdev_ioctl+0x451/0x610 [zfs] [ 967.431672] do_vfs_ioctl+0xa8/0x630 [ 967.431677] ? __audit_syscall_entry+0xbc/0x110 [ 967.431682] ? syscall_trace_enter+0x1da/0x2d0 [ 967.431685] SyS_ioctl+0x79/0x90 [ 967.431688] do_syscall_64+0x73/0x130 [ 967.431693] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 967.431696] RIP: 0033:0x7f4c273385d7 [ 967.431698] RSP: 002b:7ffed780e068 EFLAGS: 0246 ORIG_RAX: 0010 [ 967.431701] RAX: ffda RBX: 5a46 RCX: 7f4c273385d7 [ 967.431702] RDX: 7ffed780e080 RSI: 5a46 RDI: 0006 [ 967.431704] RBP: 7ffed780e080 R08: 7f4c2760de20 R09: [ 967.431705] R10: 55dc5929b010 R11: 0246 R12: 7ffed78116b0 [ 967.431707] R13: 0006 R14: 55dc592a2f10 R15: 000c [ 967.431722] INFO: task receive_writer:23913 blocked for more than 120 seconds. [ 967.431771] Tainted: P OE4.15.0-99-generic #100-Ubuntu [ 967.431816] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 967.431867] receive_writer D0 23913 2 0x8080 [ 967.431870] Call Trace: [ 967.431874] __schedule+0x24e/0x880 [
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
i've uploaded a debug zfs-dkms package to https://launchpad.net/~colin- king/+archive/ubuntu/zfs-sru-1861235 for testing. This will dump out internal state of the driver to get a better idea of what is happening during this crash. Please can you test this by doing the following: sudo add-apt-repository ppa:colin-king/zfs-sru-1861235 sudo apt-get update sudo apt-get install zfs-dkms let this debug dkms zfs module build and reboot. when the failure occurs again please supply as much of the dmesg output as possible Thank you -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
** Changed in: linux Status: Unknown => New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
Hello Colin, yes, this is still an open issue: Linux wopr 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux Apr 22 19:10:03 wopr zed[12576]: eid=8352 class=history_event pool_guid=0xB3B099B638F02EEF Apr 22 19:10:03 wopr kernel: VERIFY(size != 0) failed Apr 22 19:10:03 wopr kernel: PANIC at range_tree.c:304:range_tree_find_impl() Apr 22 19:10:03 wopr kernel: Showing stack for process 12577 Apr 22 19:10:03 wopr kernel: CPU: 8 PID: 12577 Comm: receive_writer Tainted: P O 4.15.0-91-generic #92-Ubuntu Apr 22 19:10:03 wopr kernel: Hardware name: Supermicro SSG-6038R-E1CR16L/X10DRH-iT, BIOS 2.0 12/17/2015 Apr 22 19:10:03 wopr kernel: Call Trace: Apr 22 19:10:03 wopr kernel: dump_stack+0x6d/0x8e Apr 22 19:10:03 wopr kernel: spl_dumpstack+0x42/0x50 [spl] Apr 22 19:10:03 wopr kernel: spl_panic+0xc8/0x110 [spl] Apr 22 19:10:03 wopr kernel: ? __switch_to_asm+0x41/0x70 Apr 22 19:10:03 wopr kernel: ? abd_iter_map+0xa/0x90 [zfs] Apr 22 19:10:03 wopr kernel: ? dbuf_dirty+0x43d/0x850 [zfs] Apr 22 19:10:03 wopr kernel: ? getrawmonotonic64+0x43/0xd0 Apr 22 19:10:03 wopr kernel: ? getrawmonotonic64+0x43/0xd0 Apr 22 19:10:03 wopr kernel: ? dmu_zfetch+0x49a/0x500 [zfs] Apr 22 19:10:03 wopr kernel: ? getrawmonotonic64+0x43/0xd0 Apr 22 19:10:03 wopr kernel: ? dmu_zfetch+0x49a/0x500 [zfs] Apr 22 19:10:03 wopr kernel: ? mutex_lock+0x12/0x40 Apr 22 19:10:03 wopr kernel: ? dbuf_rele_and_unlock+0x1a8/0x4b0 [zfs] Apr 22 19:10:03 wopr kernel: range_tree_find_impl+0x88/0x90 [zfs] Apr 22 19:10:03 wopr kernel: ? spl_kmem_zalloc+0xdc/0x1a0 [spl] Apr 22 19:10:03 wopr kernel: range_tree_clear+0x4f/0x60 [zfs] Apr 22 19:10:03 wopr kernel: dnode_free_range+0x11f/0x5a0 [zfs] Apr 22 19:10:03 wopr kernel: dmu_object_free+0x53/0x90 [zfs] Apr 22 19:10:03 wopr kernel: dmu_free_long_object+0x9f/0xc0 [zfs] Apr 22 19:10:03 wopr kernel: receive_freeobjects.isra.12+0x7a/0x100 [zfs] Apr 22 19:10:03 wopr kernel: receive_writer_thread+0x6d2/0xa60 [zfs] Apr 22 19:10:03 wopr kernel: ? set_curr_task_fair+0x2b/0x60 Apr 22 19:10:03 wopr kernel: ? spl_kmem_free+0x33/0x40 [spl] Apr 22 19:10:03 wopr kernel: ? kfree+0x165/0x180 Apr 22 19:10:03 wopr kernel: ? receive_free.isra.13+0xc0/0xc0 [zfs] Apr 22 19:10:03 wopr kernel: thread_generic_wrapper+0x74/0x90 [spl] Apr 22 19:10:03 wopr kernel: kthread+0x121/0x140 Apr 22 19:10:03 wopr kernel: ? __thread_exit+0x20/0x20 [spl] Apr 22 19:10:03 wopr kernel: ? kthread_create_worker_on_cpu+0x70/0x70 Apr 22 19:10:03 wopr kernel: ret_from_fork+0x35/0x40 Apr 22 19:12:56 wopr kernel: INFO: task txg_quiesce:2265 blocked for more than 120 seconds. Apr 22 19:12:56 wopr kernel: Tainted: P O 4.15.0-91-generic #92-Ubuntu Apr 22 19:12:56 wopr kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 22 19:12:56 wopr kernel: txg_quiesce D0 2265 2 0x8000 Apr 22 19:12:56 wopr kernel: Call Trace: Apr 22 19:12:56 wopr kernel: __schedule+0x24e/0x880 Apr 22 19:12:56 wopr kernel: schedule+0x2c/0x80 Apr 22 19:12:56 wopr kernel: cv_wait_common+0x11e/0x140 [spl] Apr 22 19:12:56 wopr kernel: ? wait_woken+0x80/0x80 Apr 22 19:12:56 wopr kernel: __cv_wait+0x15/0x20 [spl] Apr 22 19:12:56 wopr kernel: txg_quiesce_thread+0x2cb/0x3d0 [zfs] Apr 22 19:12:56 wopr kernel: ? txg_delay+0x1b0/0x1b0 [zfs] Apr 22 19:12:56 wopr kernel: thread_generic_wrapper+0x74/0x90 [spl] Apr 22 19:12:56 wopr kernel: kthread+0x121/0x140 Apr 22 19:12:56 wopr kernel: ? __thread_exit+0x20/0x20 [spl] Apr 22 19:12:56 wopr kernel: ? kthread_create_worker_on_cpu+0x70/0x70 Apr 22 19:12:56 wopr kernel: ret_from_fork+0x35/0x40 Apr 22 19:12:56 wopr kernel: INFO: task zfs:12482 blocked for more than 120 seconds. Apr 22 19:12:56 wopr kernel: Tainted: P O 4.15.0-91-generic #92-Ubuntu Apr 22 19:12:56 wopr kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 22 19:12:56 wopr kernel: zfs D0 12482 12479 0x8080 Apr 22 19:12:56 wopr kernel: Call Trace: Apr 22 19:12:56 wopr kernel: __schedule+0x24e/0x880 Apr 22 19:12:56 wopr kernel: schedule+0x2c/0x80 Apr 22 19:12:56 wopr kernel: cv_wait_common+0x11e/0x140 [spl] Apr 22 19:12:56 wopr kernel: ? wait_woken+0x80/0x80 Apr 22 19:12:56 wopr kernel: __cv_wait+0x15/0x20 [spl] Apr 22 19:12:56 wopr kernel: dmu_recv_stream+0xa51/0xef0 [zfs] Apr 22 19:12:56 wopr kernel: zfs_ioc_recv_impl+0x306/0x1100 [zfs] Apr 22 19:12:56 wopr kernel: ? dbuf_rele+0x36/0x40 [zfs] Apr 22 19:12:56 wopr kernel: zfs_ioc_recv_new+0x33d/0x410 [zfs] Apr 22 19:12:56 wopr kernel: ? spl_kmem_alloc_impl+0xe5/0x1a0 [spl] Apr 22 19:12:56 wopr kernel: ? spl_vmem_alloc+0x19/0x20 [spl] Apr 22 19:12:56 wopr kernel: ? nv_alloc_sleep_spl+0x1f/0x30 [znvpair] Apr 22 19:12:56 wopr kernel: ? nv_mem_zalloc.isra.0+0x2e/0x40 [znvpair] Apr 22 19:12:56 wopr kernel: ? nvlist_xalloc.part.2+0x50/0xb0 [znvpair] Apr 22 19:12:56 wopr kernel:
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
*still an open issue? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
BTW, Is this still and open issue? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
In this scenario, dmu_object_free has been called and this calls
dnode_free_range using dnode_free_range(dn, 0, DMU_OBJECT_END, tx)
In this case, in dnode_free_range len is DMU_OBJECT_ENDand offset is 0,
so..:
if (len == DMU_OBJECT_END) {
len = UINT64_MAX - off;
trunc = TRUE;
}
makes len is UINT64_MAX and trunc = TRUE. If one follows the code
through we have this stanza:
blkid = off >> blkshift;
nblks = len >> blkshift;
if (trunc)
nblks += 1;
and this is the last place nblks gets updated. This implies blkshift is
zero, nblks = UINT64_MAX >> 0 is UINT64_MAX, trunc is TRUE so nblks +=1
wraps nblks back to 0, hence the assert later on.
For blkshift to be zero blksz dn->dn_datablksz is zero, so this looks
like some form of corruption.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1861235
Title:
zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
To manage notifications about this bug go to:
https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
Incidentally, dnode_free_range hasn't changed much in the focal 8.2.x zfs, so I'm not sure if upgrading to that will make much of a difference. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
Please ignore the above. Apparently the issue needs a little more digging and the workaround is insufficient. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
I've uploaded a potential fix to a PPA, do you mind testing this using the zfs-dkms kernel modules as follows: sudo add-apt-repository ppa:colin-king/zfs-sru-1861235 sudo apt-get update sudo apt-get install zfs-dkms and reboot. Then check the correct ZFS module is being used by: dmesg | grep ZFS It should be the 0.7.5-1ubuntu16.9~lp1861235 version. And see if this helps avoid this issue. ** Changed in: linux (Ubuntu) Status: Confirmed => Incomplete ** Also affects: zfs-linux (Ubuntu) Importance: Undecided Status: New ** Changed in: zfs-linux (Ubuntu) Status: New => Incomplete ** Changed in: zfs-linux (Ubuntu) Importance: Undecided => High ** Changed in: zfs-linux (Ubuntu) Assignee: (unassigned) => Colin Ian King (colin-king) ** Also affects: linux (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: zfs-linux (Ubuntu Bionic) Importance: Undecided Status: New ** No longer affects: linux (Ubuntu) ** No longer affects: linux (Ubuntu Bionic) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
What is interesting is the following commit modifies range_tree_clear()
so it performs a zero size check and returns before calling
range_tree_find_impl(). This commit is not in 18.10 and 19.04 Ubuntu ZFS
releases.
commit a1d477c
Author: Matthew Ahrens mahr...@delphix.com
Date: Thu Sep 22 09:30:13 2016 -0700
OpenZFS 7614, 9064 - zfs device evacuation/removal
OpenZFS 7614 - zfs device evacuation/removal
OpenZFS 9064 - remove_mirror should wait for device removal to complete
the specific change is:
@@ -560,6 +536,9 @@ range_tree_clear(range_tree_t *rt, uint64_t start, uint64_t
size)
{
range_seg_t *rs;
+ if (size == 0)
+ return;
+
while ((rs = range_tree_find_impl(rt, start, size)) != NULL) {
uint64_t free_start = MAX(rs->rs_start, start);
uint64_t free_end = MIN(rs->rs_end, start + size);
I'm not sure why this check was added, but I guess it handles the cases
were zero sized allocations are allowed and stops these from doing any
unnecessary clearing and avoids the assertion. But the semantics change
is not clear in the commit message.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1861235
Title:
zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
To manage notifications about this bug go to:
https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
The two machines involved are: Receiver, bionic, probably running 4.15.0-76-generic and zfsutils-linux 0.7.5-1ubuntu16.7 Sender, focal, probably running 5.4.0-12-generic and zfsutils-linux 0.8.3-1ubuntu3 I'm using sanoid and syncoid to automate snapshot management and sending and receiving. The combination worked well for about a week before I got this failure. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
These are just the zfs bookmark and zfs send commands from the sender. ** Attachment added: "zfs bookmark, zfs send commands" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1861235/+attachment/5327157/+files/sender_limited_history -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
These are all the zpool and zfs commands on the receiver except snapshots, renames, and destroys, associated with my Ubuntu archive mirror rsync. (7000-ish lines of juggling 30-snapshots. I should do something better here.) ** Attachment added: "receiver_limited_history" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1861235/+attachment/5327158/+files/receiver_limited_history -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
Can you describe the zfs environment and the command that was being actioned that triggered this issue? ** Bug watch added: Github Issue Tracker for ZFS #8637 https://github.com/zfsonlinux/zfs/issues/8637 ** Also affects: linux via https://github.com/zfsonlinux/zfs/issues/8637 Importance: Unknown Status: Unknown -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
** Changed in: linux (Ubuntu) Assignee: (unassigned) => Colin Ian King (colin-king) ** Changed in: linux (Ubuntu) Importance: Undecided => Medium ** Changed in: linux (Ubuntu) Importance: Medium => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1861235/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
