** Changed in: zfs-linux (Ubuntu)
Assignee: Colin Ian King (colin-king) => Dimitri John Ledkov (xnox)
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1861235
Title:
zfs recv PANIC at
Status changed to 'Confirmed' because the bug affects multiple users.
** Changed in: zfs-linux (Ubuntu Bionic)
Status: New => Confirmed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1861235
This time I don't see any error or panic in `journalctl -b` and
apparently everything is fine according to smartctl, so it looks like we
are not getting any obvious I/O error at the moment.
IIUC you have done already a `zpool scrub` and that also didn't report
any error, so apparently the zfs
My guess is the next step is trying the zfs send | ssh zfs recv commands
that were causing the VERIFY(size != 0) failed message with traces. I've
always been uneasy with the 'main' error coming only with a recv
operation, relying upon two systems to recreate a bug is iffy at best.
And since it
The scrub has finished, no new errors in zpool status -v, dmesg is
filled with debugging messages, I hope something is useful. :)
Thanks
$ uname -a
Linux wopr 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020
x86_64 x86_64 x86_64 GNU/Linux
$ modinfo zfs | grep -i version
version:
Here's smartctl output on the disks in the pool. It's a bit hard to
summarize nine drives of smartctl output but nothing stood out as
interesting to me.
My journals only go back a few months so they wouldn't show any IO
errors at the time of this problem being introduced, but this currently
gives
Picking the previous bad object (which I'm pretty sure should still be
intact):
$ modinfo zfs | grep ^version
version:0.7.5-1ubuntu16.10+lp1861235
$ uname -a
Linux wopr 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020
x86_64 x86_64 x86_64 GNU/Linux
$ sudo zdb -
Hello Andrea, thanks for looking into my problem.
# zpool status -v
pool: fst
state: ONLINE
scan: scrub repaired 0B in 2h37m with 0 errors on Sun Nov 8 03:01:53 2020
config:
NAMESTATE READ WRITE CKSUM
fst ONLINE 0
I've uploaded the latest version of zfs with debugging enabled (--enable-debug)
in this ppa:
https://launchpad.net/~arighi/+archive/ubuntu/zfs-linux
sudo add-apt-repository ppa:arighi/zfs-linux
sudo apt-get update
sudo apt-get install zfs-dkms
It'd be interesting to repeat the test using this
I just noticed the bug on github that also mentions about potential data
corruption. It looks like we need to recompile zfs with --enable-debug.
I'll try to produce a debug package with that.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to
Hi @seth-arnold,
adding some details to this bug:
dn->dn_type = 19 = DMU_OT_PLAIN_FILE_CONTENTS
dn->dn_type = 20 = DMU_OT_DIRECTORY_CONTENTS
So they look correct, they seem to be just plain files and directories.
However, the specific object that is causing the panic seems to have
dn_type=32
Is it interesting that I'm seeing both 19 and 20 in my dmesg?
[1229822.406130] dmu_object_free: object = 0x266d2904, dn->dn_type = 20
[1229823.980888] dmu_object_free: object = 0x266d0d5a, dn->dn_type = 20
[1229823.994690] dmu_object_free: object = 0x266d0d5b, dn->dn_type = 20
[1229823.998123]
I picked the last dataset given in the command output from an earlier,
but not the most recent, comment:
$ sudo zdb - srv/backups/millbarge/rpool/var/log 529
Dataset srv/backups/millbarge/rpool/var/log [ZPL], ID 39694, cr_txg 23197757,
554M, 274 objects, rootbp DVA[0]=<1:1d000d42000:1000>
OK so object 0x211 looks like the curlprit, I just can't see the pool
name being dumped - do you know which pool is triggering this issue?
If you can identify the poolt then we can next dump the potentially
corrupt object using:
sudo zdb - / 529
--
You received this bug notification
Hi Seth, I've upload a .4 - it may add a lot more debug but it will
allow us to see the object type that is failing before the crash.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1861235
Title:
On Sat, May 16, 2020 at 01:56:08AM -, Seth Arnold wrote:
> Sadly, journalctl doesn't have the dmesg from the previous boot:
I meant to say, journalctl's copy of dmesg from the previous boot doesn't
have the new debug output. Sorry.
Thanks
--
You received this bug notification because you
Hello Colin, trying the zfs recv operation on the .3 dkms eventually
kills my system dead. There was nothing on the console. The cursor on
the console stopped blinking; I couldn't switch VTs. My ssh sessions
were hung. ping reported destination host unreachable.
It ran for about two and a half
Thanks for the data for the .2 tests. I've updated the packages with a
.3 test build.
I've added a couple more lines of debug now to figure out some earlier
missing information once we pop the stack. I've updated the package in
the PPA, do you mind updating the zfs-dkms to
Oh, I see you are using the latest version:
v0.7.5-1ubuntu16.10~lp1861235.2
I may need to figure out a .3 version to try next.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1861235
Title:
zfs recv
Hi, I was expecting some extra information and a panic message in a
different place, are you sure this is running the latest zfs-dkms? One
can check that by using dmesg | grep ZFS and check this is the .2
version of the debug zfd-dkms package.
--
You received this bug notification because you
Thanks Colin!
[ 271.628232] dnone_free_ramge: nblks = 0, trunc = 1, len =
18446744073709551615, blkshift = 0
[ 271.628297] dnode_free_range: nblks == 0, len == 18446744073709551615
, off=0
[ 271.628298] range_tree_clear: size == 0
[ 271.628375] range_tree_find_impl: size == 0
Thanks for the data.
I've added a couple more lines of debug now to figure out some earlier
missing information once we pop the stack. I've updated the package in
the PPA, do you mind updating the zfs-dkms to
0.7.5-1ubuntu16.10~lp1861235.2 and re-testing?
--
You received this bug notification
Here's the part that looks important; the whole dmesg is in the
attachment.
Thanks
[ 761.730488] dnone_free_ramge: nblks = 0, trunc = 1, len =
18446744073709551615, blkshift = 0
[ 761.730542] dnode_free_range: nblks == 0, len == 18446744073709551615
, off=0
[ 761.730543]
i've uploaded a debug zfs-dkms package to https://launchpad.net/~colin-
king/+archive/ubuntu/zfs-sru-1861235 for testing. This will dump out
internal state of the driver to get a better idea of what is happening
during this crash.
Please can you test this by doing the following:
sudo
** Changed in: linux
Status: Unknown => New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1861235
Title:
zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
To manage notifications
Hello Colin, yes, this is still an open issue:
Linux wopr 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020
x86_64 x86_64 x86_64 GNU/Linux
Apr 22 19:10:03 wopr zed[12576]: eid=8352 class=history_event
pool_guid=0xB3B099B638F02EEF
Apr 22 19:10:03 wopr kernel: VERIFY(size != 0)
*still an open issue?
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1861235
Title:
zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
To manage notifications about this bug go to:
BTW, Is this still and open issue?
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1861235
Title:
zfs recv PANIC at range_tree.c:304:range_tree_find_impl()
To manage notifications about this bug go
In this scenario, dmu_object_free has been called and this calls
dnode_free_range using dnode_free_range(dn, 0, DMU_OBJECT_END, tx)
In this case, in dnode_free_range len is DMU_OBJECT_ENDand offset is 0,
so..:
if (len == DMU_OBJECT_END) {
len = UINT64_MAX - off;
trunc
Incidentally, dnode_free_range hasn't changed much in the focal 8.2.x
zfs, so I'm not sure if upgrading to that will make much of a
difference.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1861235
Please ignore the above. Apparently the issue needs a little more
digging and the workaround is insufficient.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1861235
Title:
zfs recv PANIC at
I've uploaded a potential fix to a PPA, do you mind testing this using
the zfs-dkms kernel modules as follows:
sudo add-apt-repository ppa:colin-king/zfs-sru-1861235
sudo apt-get update
sudo apt-get install zfs-dkms
and reboot.
Then check the correct ZFS module is being used by:
dmesg | grep
What is interesting is the following commit modifies range_tree_clear()
so it performs a zero size check and returns before calling
range_tree_find_impl(). This commit is not in 18.10 and 19.04 Ubuntu ZFS
releases.
commit a1d477c
Author: Matthew Ahrens mahr...@delphix.com
Date: Thu Sep 22
The two machines involved are:
Receiver, bionic, probably running 4.15.0-76-generic and zfsutils-linux
0.7.5-1ubuntu16.7
Sender, focal, probably running 5.4.0-12-generic and zfsutils-linux
0.8.3-1ubuntu3
I'm using sanoid and syncoid to automate snapshot management and sending and
receiving.
These are just the zfs bookmark and zfs send commands from the sender.
** Attachment added: "zfs bookmark, zfs send commands"
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1861235/+attachment/5327157/+files/sender_limited_history
--
You received this bug notification because you are
These are all the zpool and zfs commands on the receiver except
snapshots, renames, and destroys, associated with my Ubuntu archive
mirror rsync. (7000-ish lines of juggling 30-snapshots. I should do
something better here.)
** Attachment added: "receiver_limited_history"
Can you describe the zfs environment and the command that was being
actioned that triggered this issue?
** Bug watch added: Github Issue Tracker for ZFS #8637
https://github.com/zfsonlinux/zfs/issues/8637
** Also affects: linux via
https://github.com/zfsonlinux/zfs/issues/8637
** Changed in: linux (Ubuntu)
Assignee: (unassigned) => Colin Ian King (colin-king)
** Changed in: linux (Ubuntu)
Importance: Undecided => Medium
** Changed in: linux (Ubuntu)
Importance: Medium => High
--
You received this bug notification because you are a member of Ubuntu
Bugs,
38 matches
Mail list logo