Re: [Bug 1854982] Re: Lost compatibilty for backup between Ubuntu 19.10 and FreeBSD 12.0
I'm using Ubuntu 21.04 Beta and FreeBSD 13.0-RC5 now on my desktop and backup server. I did check it and it works without any isue, even after I changed the setting I used to force compatibility. I already added this text to the bug report. On Wed, 2021-04-07 at 13:24 +, Colin Ian King wrote: > ZFS 2.0.x is now the default for Ubuntu Hirsute 21.04. If anyone > would > like to check that this is now compatible with BSD then that would be > useful so we can close this issue. > > ** Changed in: zfs-linux (Ubuntu) >Importance: Undecided => Medium > -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1854982 Title: Lost compatibilty for backup between Ubuntu 19.10 and FreeBSD 12.0 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1854982/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1854982] Re: Lost compatibilty for backup between Ubuntu 19.10 and FreeBSD 12.0
The system uses xattr=sa, but I did not set it myself. Like you can see in the annex, it has been inherited from rpool everywhere and it has been set by the installer. I annexed the properties - Home of Ubuntu, which properties I did not touch at all (except canmount) and - my main dataset with my VMs (rpool/vms/Vbox), where I only changed its dnodesize and snapdir. The last one (rpool/vms/Vbox) I backup weekly to FreeBSD 12.1. For my last try-outs I did use the Home directory of Ubuntu and that one failed. The only significant difference, I notice between the two datasets, is dnodesize. I made the .zfs folder visible this year to restore a VM. Remark: I use the experimental Ubuntu install with Root on ZFS, but I also have an ext4 installation on the PC, so I dual boot. Kind of last resort for the experimental install. In say October and November to be able to update ZFS on the ext4 installation, I had to change canmount during the update, but I think that bug has been solved. Canmount is back at its default. On Fri, 2020-01-31 at 13:24 +, Garrett Fields wrote: > I quickly set up a VM last night and tried to recreate problem, I > have > been unsuccessful so far. I'm wondering if certain dnode data > (perhaps > the use of xattr=sa on the Linux source) might be triggering the > issue? > ** Attachment added: "properties" https://bugs.launchpad.net/bugs/1854982/+attachment/5324493/+files/properties -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1854982 Title: Lost compatibilty for backup between Ubuntu 19.10 and FreeBSD 12.0 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1854982/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1854982] Re: Lost compatibilty for backup between Ubuntu 19.10 and FreeBSD 12.0
I now also filed a bug-report Bug 243730 for FreeBSD with the following ending; Ubuntu and FreeBSD did choose different defaults for large-dnodes and dnodesizes, but to solve bugs related to feature incompatibility both groups have to communicate! The problem will not disappear completely, because you start using the same source. There will be probably months between release dates, so feature incompatibility probably will remain an issue. My problem is bypassed, but the default dnodesize incompatibility between Ubuntu 19.10 and FreeBSD 12.1 remains. On Thu, 2020-01-30 at 08:47 +, Richard Laager wrote: > ** Changed in: zfs-linux (Ubuntu) >Status: New => Incomplete > -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1854982 Title: Lost compatibilty for backup between Ubuntu 19.10 and FreeBSD 12.0 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1854982/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1854982] Re: Lost compatibilty for backup between Ubuntu 19.10 and FreeBSD 12.0
Garret Fields also specified some test and the result of those test were as specified here. I used Ubuntu 19.10 and FreeBSD 12.1. I detected the issue running FreeBSD 12.0. Both system have the large-dnode feature active! Weekly I do send the data with the param -c, there is however one uncompressed dataset, that is sent without -c. And the following combinations work, Correct Local transfers: - freeBSD 12.1 (dnodesize=legacy) to freeBSD 12.1 (dnodesize=legacy) - freeBSD 12.1 (dnodesize=legacy) to freeBSD 12.1 (dnodesize=auto) - Ubuntu to Ubuntu, with and without -c I do not remember any problem. Correct Remote transfers: - freeBSD 12.1 (dnodesize=legacy) to Ubuntu (dnodesize=auto) - Ubuntu (dnodesize=legacy) to Laptop-Ubuntu (dnodesize=legacy), with and without -c - Ubuntu (dnodesize=legacy) to FreeBSD 12.x (dnodesize=legacy), with and without -c The last two I do use weekly! Failing transfers: - Ubuntu 19.10 (dnodesize=auto) to FreeBSD 12.x (dnodesize=legacy) - Ubuntu 19.10 (dnodesize=auto) to FreeBSD 12.1 (dnodesize=auto) Note that during the transfers, I do see, that the dataset is growing in size, while using zfs list or Conky in FreeBSD (using zfs list). At the end of the transfer, sometimes after hours or minutes I get the error message and the dataset disappears :( The settings selected as default by both development teams in spendlid isolation, do not work! The one from Ubuntu 19.10 (dnodesize=auto) to FreeBSD 12.x (dnodesize=legacy) This is the relevant information you can get out of me. Time to get somebody involved, who knows the corresponding program code. In the past I solved bugs, having with less information available. Like I said: GOOD LUCK in solving the bug. On Thu, 2020-01-30 at 05:53 +, Richard Laager wrote: > In terms of a compact reproducer, does this work: > > # Create a temp pool with large_dnode enabled: > truncate -s 1G lp1854982.img > sudo zpool create -d -o feature@large_dnode=enabled lp1854982 > $(pwd)/lp1854982.img > > # Create a dataset with dnodesize=auto > sudo zfs create -o dnodesize=auto lp1854982/ldn > > # Create a send stream > sudo zfs snapshot lp1854982/ldn@snap > sudo zfs send lp1854982/ldn@snap > lp1854982-ldn.zfs > > sudo zpool export lp1854982 > > cat lp1854982-ldn.zfs | ssh 192.168.1.100 zfs receive zroot/ldn > > If that doesn't reproduce the problem, adjust it until it does. You > were > using `zfs send -c`, so maybe that's it. You may need to enable more > pool features, etc. > > But if this can be reproduced with an empty dataset on an empty pool, > the send stream file is 8.5K (and far less compressed). Attach the > script for reference and the send stream to a FreeBSD bug. > -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1854982 Title: Lost compatibilty for backup between Ubuntu 19.10 and FreeBSD 12.0 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1854982/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1854982] Re: Lost compatibilty for backup between Ubuntu 19.10 and FreeBSD 12.0
"dpool" is another datapool created with Ubuntu 19.10 and it had the same defaults with respect to "large-dnode" as rpool. My main problem has been with rpool, since it took my whole nvme-SSD. By the way the same happened in FreeBSD with zroot, during the install it also took all space on my striped HDDs :) Note that FreeBSD is a 32-bits version on an old Pentium 4 HT :) By the way dpool (Ubuntu) is also striped over two 450GB partitions on a 500GB and a 1TB HDD. The second part of the 1TB HDD still had the partition/datapool created by Ubuntu 18.04 with zfs 0.7.x release and that one had no large-dnode problems. I solved my send/receive problem by specifying on the Ubuntu system for each dataset on rpool and dpool dnodesize=legacy and reloaded the content of those datasets. See the Ubuntu dnodesize overview in the attachment. On FreeBSD zroot has "large-dnode = active" and the dnodesize is as follows: root@freebsd:~ # zfs get dnodesize NAME PROPERTY VALUE SOURCE bootpool dnodesize legacy default zroot dnodesize legacy default zroot/ROOTdnodesize legacy default zroot/ROOT@upgrade12-1dnodesize - - zroot/hp-data dnodesize legacy local zroot/hp-data/ISO dnodesize legacy inherited from zroot/hp-data - I have created a separate dataset on FreeBSD with the same attributes as rpool/USERDATA on Ubuntu zroot/USERDATAdnodesize autolocal Sending data to this datset had the following result: See the send/receive results after the dnodesize overview in the attachment. Note that at the end I tried to create a new dataset zroot/USER. --- And now the sends inside FreeBSD both with a new USER dataset and the exsisting USERDATA with dnodesize=auto. root@freebsd:~ # zfs send -c zroot/var/log@upgrade12-1 | zfs receive zroot/USER root@freebsd:~ # zfs send -c zroot/var/log@upgrade12-1 | zfs receive -F zroot/USERDATA root@freebsd:~ # zroot/USER has been created and USERDATA existed with dnodesize=auto The result has been as expected. zroot/USERdnodesize legacy default zroot/USER@upgrade12-1dnodesize - - zroot/USERDATAdnodesize autolocal zroot/USERDATA@upgrade12-1dnodesize - - - And now send from FreeBSD to Ubuntu: see for the command attachment at the end. and the result rpool/USER@upgrade12-1 0B - 888K - rpool/USER dnodesize autoinherited from rpool rpool/USER@upgrade12-1 dnodesize - - - Both system have the large-dnode feature active! And almost all combinations work, - freeBSD to freeBSD from dnodesize=legacy to a dnodesize, that is either legacy or auto - Ubuntu to Ubuntu, I do not remember any problem. - freeBSD to Ubuntu from dnodesize=legacy to dnodesize=auto - Ubuntu (dnodesize=legacy) to FreeBSD 12.x (dnodesize-legacy) works and that is what I use now. The default one selected as default by both development teams in spendlid isolation, did not work! The one from Ubuntu 19.10 (dnodesize=auto) to FreeBSD 12.x (dnodesize=legacy) Also from Ubuntu 19.10 (dnodesize=auto) to FreeBSD 12.x (dnodesize=auto) failed, see test. GOOD LUCK finding the error. On Wed, 2020-01-29 at 04:29 +, Garrett Fields wrote: > So these pools were created with the Ubuntu Ubiquity ZFS > installer? I > missed that because the pool names are hardcoded to bpool and rpool > and > your message lists 'dpool/dummy' and 'zroot/hp-data/dummy' > > Also, in the linked email thread, you stated "The ZFS manual advised > auto, if also using xattr=sa, so that is why I used auto for my own > datapools/datasets." > > Now the origin of the pool is clearer to me. Yes I do see -O > dnodesize=auto being set on (and inherited from) rpool in Ubiquity > root > zfs installation. This would impact the ease of sending to non- > large_dnode pools (or in your case FreeBSD with large_dnode > problems). > > Some simple tests to run: > Within FreeBSD, I'd be really surprised if large_dnode=active to > large_dnode=enabled/active zfs send/recv doesn't work, but I'd start > there. > > Next, I'd try to send from FreeBSD large_dnode=active to Linux > large_dnode=enabled/active. If it fails, what error is returned? > > Also, like rlaager stated, we should do the original Linux > large_dnode=active to FreeBSD large_dnode=enabled/active that gave > you > problems. This all will give us evidence for bug reports in FreeBSD > and/or ZOL
Re: [Bug 1854982] Re: Lost compatibilty for backup between Ubuntu 19.10 and FreeBSD 12.0
What do you mean with "a compact reproducer"? I can reproduce the error easily, but I have no clue, how to produce more info? I'm still relatively new to FreeBSD. On Wed, 2020-01-29 at 01:20 +, Richard Laager wrote: > So, one of two things is true: > A) ZFS on Linux is generating the stream incorrectly. > B) FreeBSD is receiving the stream incorrectly. > > I don't have a good answer as to how we might differentiate those > two. > Filing a bug report with FreeBSD might be a good next step. But like > I > said, a compact reproducer would go a long way. > -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1854982 Title: Lost compatibilty for backup between Ubuntu 19.10 and FreeBSD 12.0 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1854982/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1854982] Re: Lost compatibilty for backup between Ubuntu 19.10 and FreeBSD 12.0
According to the overview of features on the OpenZFS website (see the link provided by Richard Laager in the bug-report), FreeBSD 12 does not support "large dnode". However FreeBSD did set the large dnode feature to "active" and it is still set to "active". But FreeBSD does not handle those send/receive streams correctly. It reads the stream builds up the dataset@snapshot and at the end after an hour or so, it gives the errormessage. --- cannot receive new filesystem stream: destination 'zroot/hp-data/dummy' exists must specify -F to overwrite it That dataset did exists, I can see it grow in size during that hour with "zfs list" but, when the transfer is completed, it gives that error message. I guess FreeBSD is creating the dataset and start filling it and in the end instead of creating the snapshot, it tries to recreate the whole dataset :) And all transfered data disappears. The fun part is, that if you follow the advise in the errormessage, the system will say: cannot receive new filesystem stream: dataset does not exist -- Your remark: "the situation was created by a user explicitly running "zfs set dnodesize=auto" or a "zpool create -O dnodesize=auto", was wrong. I did not set anything, but those settings were chosen by the Ubuntu install process and I only created some own datasets on that "rpool" datapool. That is why I complained about the missing params in a future "install" or "create datapool" process. For me personally, it also could be solved by allowing me to choose the "rpool" size during install. In that case I would not store my user datasets in rpool, but create an own datapool on that nvme-SSD with the correct compatible feature settings using the OpenZFS document. I tried it again, because I updated FreeBSD to 12.1, but exactly the same error happened again. What happens and the commands and error messages were in the original bugreport. If you think it is a FreeBSD problem, please send the stuff to them. On Tue, 2020-01-28 at 20:46 +, Richard Laager wrote: > The last we heard on this, FreeBSD was apparently not receiving the > send > stream, even though it supports large_dnode: > > https://zfsonlinux.topicbox.com/groups/zfs- > discuss/T187d60c7257e2eb6-M14bb2d52d4d5c230320a4f56/feature- > incompatibility-between-ubuntu-19-10-and-freebsd-12-0 > > That's really bizarre. If it supports large_dnode, it should be able > to > receive that stream. Ideally, this needs more troubleshooting, > particularly on the receive side. "It said (dataset does not exist) > after a long transfer." is not particularly clear. I'd like to see a > copy-and-paste of the actual `zfs recv` output, at a minimum. > > @BertN45, if you want to keep troubleshooting, a good next step would > be > to boil this down to a reproducible test case. That is, create a list > of > specific commands to create dataset and send it that demonstrates the > problem. That would help. We may need to flesh out the reproducer a > bit > more, e.g. by creating a pool on sparse files with particular feature > flags. > -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1854982 Title: Lost compatibilty for backup between Ubuntu 19.10 and FreeBSD 12.0 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1854982/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1854982] Re: Lost compatibilty for backup between Ubuntu 19.10 and FreeBSD 12.0
The easy, lazy solution is to close this bugreport. However if you start to use this install option on servers in server farms, you might have this problem more frequently. The OpenZFS website has a matrix with which features are supported by which OSes. It would be relatively easy to implement that matrix in the installers and to ask during installation, whether feature compatibility is required with another OS. At least you should do something about the inconsistent and confusing error messages. Just read my bugreport again carefully! By the way, I could solve this large-dnode feature incompatibility, by putting all dataset property to "dnodesize=legacy" and reload all those datasets. I don't care anymore, but a server administrator will not be amused, when running in these type of issues with an Ubuntu Server. On Tue, 2020-01-28 at 13:45 +, fields_g wrote: > At this point, does this need a code improvement to avoid this > situation > for others? If not, this looks like this bug report is ready to be > closed/resolved. > -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1854982 Title: Lost compatibilty for backup between Ubuntu 19.10 and FreeBSD 12.0 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1854982/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
