[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2021-10-08 Thread Colin Ian King
** Changed in: zfs-linux (Ubuntu) Assignee: Colin Ian King (colin-king) => Dimitri John Ledkov (xnox) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-12-30 Thread Launchpad Bug Tracker
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: zfs-linux (Ubuntu Bionic) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-12-04 Thread Andrea Righi
This time I don't see any error or panic in `journalctl -b` and apparently everything is fine according to smartctl, so it looks like we are not getting any obvious I/O error at the moment. IIUC you have done already a `zpool scrub` and that also didn't report any error, so apparently the zfs

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-12-03 Thread Seth Arnold
My guess is the next step is trying the zfs send | ssh zfs recv commands that were causing the VERIFY(size != 0) failed message with traces. I've always been uneasy with the 'main' error coming only with a recv operation, relying upon two systems to recreate a bug is iffy at best. And since it

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-12-03 Thread Seth Arnold
The scrub has finished, no new errors in zpool status -v, dmesg is filled with debugging messages, I hope something is useful. :) Thanks $ uname -a Linux wopr 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux $ modinfo zfs | grep -i version version:

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-12-02 Thread Seth Arnold
Here's smartctl output on the disks in the pool. It's a bit hard to summarize nine drives of smartctl output but nothing stood out as interesting to me. My journals only go back a few months so they wouldn't show any IO errors at the time of this problem being introduced, but this currently gives

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-12-02 Thread Seth Arnold
Picking the previous bad object (which I'm pretty sure should still be intact): $ modinfo zfs | grep ^version version:0.7.5-1ubuntu16.10+lp1861235 $ uname -a Linux wopr 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux $ sudo zdb -

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-12-02 Thread Seth Arnold
Hello Andrea, thanks for looking into my problem. # zpool status -v pool: fst state: ONLINE scan: scrub repaired 0B in 2h37m with 0 errors on Sun Nov 8 03:01:53 2020 config: NAMESTATE READ WRITE CKSUM fst ONLINE 0

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-12-02 Thread Andrea Righi
I've uploaded the latest version of zfs with debugging enabled (--enable-debug) in this ppa: https://launchpad.net/~arighi/+archive/ubuntu/zfs-linux sudo add-apt-repository ppa:arighi/zfs-linux sudo apt-get update sudo apt-get install zfs-dkms It'd be interesting to repeat the test using this

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-12-02 Thread Andrea Righi
I just noticed the bug on github that also mentions about potential data corruption. It looks like we need to recompile zfs with --enable-debug. I'll try to produce a debug package with that. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-12-02 Thread Andrea Righi
Hi @seth-arnold, adding some details to this bug: dn->dn_type = 19 = DMU_OT_PLAIN_FILE_CONTENTS dn->dn_type = 20 = DMU_OT_DIRECTORY_CONTENTS So they look correct, they seem to be just plain files and directories. However, the specific object that is causing the panic seems to have dn_type=32

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-09-24 Thread Seth Arnold
Is it interesting that I'm seeing both 19 and 20 in my dmesg? [1229822.406130] dmu_object_free: object = 0x266d2904, dn->dn_type = 20 [1229823.980888] dmu_object_free: object = 0x266d0d5a, dn->dn_type = 20 [1229823.994690] dmu_object_free: object = 0x266d0d5b, dn->dn_type = 20 [1229823.998123]

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-05-28 Thread Seth Arnold
I picked the last dataset given in the command output from an earlier, but not the most recent, comment: $ sudo zdb - srv/backups/millbarge/rpool/var/log 529 Dataset srv/backups/millbarge/rpool/var/log [ZPL], ID 39694, cr_txg 23197757, 554M, 274 objects, rootbp DVA[0]=<1:1d000d42000:1000>

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-05-28 Thread Colin Ian King
OK so object 0x211 looks like the curlprit, I just can't see the pool name being dumped - do you know which pool is triggering this issue? If you can identify the poolt then we can next dump the potentially corrupt object using: sudo zdb - / 529 -- You received this bug notification

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-05-20 Thread Colin Ian King
Hi Seth, I've upload a .4 - it may add a lot more debug but it will allow us to see the object type that is failing before the crash. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title:

Re: [Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-05-15 Thread Seth Arnold
On Sat, May 16, 2020 at 01:56:08AM -, Seth Arnold wrote: > Sadly, journalctl doesn't have the dmesg from the previous boot: I meant to say, journalctl's copy of dmesg from the previous boot doesn't have the new debug output. Sorry. Thanks -- You received this bug notification because you

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-05-15 Thread Seth Arnold
Hello Colin, trying the zfs recv operation on the .3 dkms eventually kills my system dead. There was nothing on the console. The cursor on the console stopped blinking; I couldn't switch VTs. My ssh sessions were hung. ping reported destination host unreachable. It ran for about two and a half

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-05-15 Thread Colin Ian King
Thanks for the data for the .2 tests. I've updated the packages with a .3 test build. I've added a couple more lines of debug now to figure out some earlier missing information once we pop the stack. I've updated the package in the PPA, do you mind updating the zfs-dkms to

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-05-12 Thread Colin Ian King
Oh, I see you are using the latest version: v0.7.5-1ubuntu16.10~lp1861235.2 I may need to figure out a .3 version to try next. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-05-12 Thread Colin Ian King
Hi, I was expecting some extra information and a panic message in a different place, are you sure this is running the latest zfs-dkms? One can check that by using dmesg | grep ZFS and check this is the .2 version of the debug zfd-dkms package. -- You received this bug notification because you

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-05-12 Thread Seth Arnold
Thanks Colin! [ 271.628232] dnone_free_ramge: nblks = 0, trunc = 1, len = 18446744073709551615, blkshift = 0 [ 271.628297] dnode_free_range: nblks == 0, len == 18446744073709551615 , off=0 [ 271.628298] range_tree_clear: size == 0 [ 271.628375] range_tree_find_impl: size == 0

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-05-12 Thread Colin Ian King
Thanks for the data. I've added a couple more lines of debug now to figure out some earlier missing information once we pop the stack. I've updated the package in the PPA, do you mind updating the zfs-dkms to 0.7.5-1ubuntu16.10~lp1861235.2 and re-testing? -- You received this bug notification

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-05-11 Thread Seth Arnold
Here's the part that looks important; the whole dmesg is in the attachment. Thanks [ 761.730488] dnone_free_ramge: nblks = 0, trunc = 1, len = 18446744073709551615, blkshift = 0 [ 761.730542] dnode_free_range: nblks == 0, len == 18446744073709551615 , off=0 [ 761.730543]

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-05-11 Thread Colin Ian King
i've uploaded a debug zfs-dkms package to https://launchpad.net/~colin- king/+archive/ubuntu/zfs-sru-1861235 for testing. This will dump out internal state of the driver to get a better idea of what is happening during this crash. Please can you test this by doing the following: sudo

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-04-23 Thread Bug Watch Updater
** Changed in: linux Status: Unknown => New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-04-22 Thread Seth Arnold
Hello Colin, yes, this is still an open issue: Linux wopr 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux Apr 22 19:10:03 wopr zed[12576]: eid=8352 class=history_event pool_guid=0xB3B099B638F02EEF Apr 22 19:10:03 wopr kernel: VERIFY(size != 0)

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-04-22 Thread Colin Ian King
*still an open issue? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go to:

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-04-22 Thread Colin Ian King
BTW, Is this still and open issue? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at range_tree.c:304:range_tree_find_impl() To manage notifications about this bug go

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-04-22 Thread Colin Ian King
In this scenario, dmu_object_free has been called and this calls dnode_free_range using dnode_free_range(dn, 0, DMU_OBJECT_END, tx) In this case, in dnode_free_range len is DMU_OBJECT_ENDand offset is 0, so..: if (len == DMU_OBJECT_END) { len = UINT64_MAX - off; trunc

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-04-22 Thread Colin Ian King
Incidentally, dnode_free_range hasn't changed much in the focal 8.2.x zfs, so I'm not sure if upgrading to that will make much of a difference. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-02-24 Thread Colin Ian King
Please ignore the above. Apparently the issue needs a little more digging and the workaround is insufficient. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1861235 Title: zfs recv PANIC at

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-02-24 Thread Colin Ian King
I've uploaded a potential fix to a PPA, do you mind testing this using the zfs-dkms kernel modules as follows: sudo add-apt-repository ppa:colin-king/zfs-sru-1861235 sudo apt-get update sudo apt-get install zfs-dkms and reboot. Then check the correct ZFS module is being used by: dmesg | grep

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-02-24 Thread Colin Ian King
What is interesting is the following commit modifies range_tree_clear() so it performs a zero size check and returns before calling range_tree_find_impl(). This commit is not in 18.10 and 19.04 Ubuntu ZFS releases. commit a1d477c Author: Matthew Ahrens mahr...@delphix.com Date: Thu Sep 22

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-02-10 Thread Seth Arnold
The two machines involved are: Receiver, bionic, probably running 4.15.0-76-generic and zfsutils-linux 0.7.5-1ubuntu16.7 Sender, focal, probably running 5.4.0-12-generic and zfsutils-linux 0.8.3-1ubuntu3 I'm using sanoid and syncoid to automate snapshot management and sending and receiving.

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-02-10 Thread Seth Arnold
These are just the zfs bookmark and zfs send commands from the sender. ** Attachment added: "zfs bookmark, zfs send commands" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1861235/+attachment/5327157/+files/sender_limited_history -- You received this bug notification because you are

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-02-10 Thread Seth Arnold
These are all the zpool and zfs commands on the receiver except snapshots, renames, and destroys, associated with my Ubuntu archive mirror rsync. (7000-ish lines of juggling 30-snapshots. I should do something better here.) ** Attachment added: "receiver_limited_history"

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-02-10 Thread Colin Ian King
Can you describe the zfs environment and the command that was being actioned that triggered this issue? ** Bug watch added: Github Issue Tracker for ZFS #8637 https://github.com/zfsonlinux/zfs/issues/8637 ** Also affects: linux via https://github.com/zfsonlinux/zfs/issues/8637

[Bug 1861235] Re: zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

2020-01-29 Thread Colin Ian King
** Changed in: linux (Ubuntu) Assignee: (unassigned) => Colin Ian King (colin-king) ** Changed in: linux (Ubuntu) Importance: Undecided => Medium ** Changed in: linux (Ubuntu) Importance: Medium => High -- You received this bug notification because you are a member of Ubuntu Bugs,