Re: btrfs-convert missing in btrfs-tools v4.15.1
Nicholas D Steeves posted on Thu, 23 Aug 2018 14:15:18 -0400 as excerpted: >> It's in my interest to ship all tools in distros, but there's also only >> that much what the upstream community can do. If you're going to >> reconsider the status of btrfs-convert in Debian, please let me know. > > Yes, I'd be happy to advocate for its reinclusion if the answer to 4/5 > of the following questions is "yes". Does SUSE now recommend the use of > btrfs-convert to its enterprise customers? The following is a > frustrating criteria, but: Can a random desktop user run btrfs-convert > against their ext4 rootfs and expect the operation to succeed? Is > btrfs-convert now sufficiently trusted that it can be recommended with > the same degree of confidence as a backup, mkfs.btrfs, then restore to > new filesystem approach? Does the user of a btrfs volume created with > btrfs-convert have an equal or lesser probability of encountering bugs > compared to a one who used mkfs.btrfs? Just a user and list regular here, and gentoo not debian, but for what it counts... I'd personally never consider or recommend a filesystem converter over the backup, mkfs-to-new-fs, restore-to-new-fs, method, for three reasons. 1) Regardless of how stable a filesystem converter is and what two filesystems the conversion is between, "things" /do/ occasionally happen, thus making it irresponsible to use or recommend use of such a converter without a suitably current and tested backup, "just in case." (This is of course a special case of the sysadmin's first rule of backups, that the true value of data is defined not by any arbitrary claims, but by the number of backups of that data it's considered worth the time/trouble/resources to make/have. If the data value is trivial enough, sure, don't bother with the backup, but if it's of /that/ low a value, so low it's not worth a backup even when doing something as theoretically risky as a filesystem conversion, why is it worth the time and trouble to bother converting it in the first place, instead of just blowing it away and starting clean?) 2) Once a backup is considered "strongly recommended", as we've just established that it should be in 1 regardless of the stability of the converter, just using the existing filesystem as that backup and starting fresh with a mkfs for the new filesystem and copying things over is simply put the easiest, simplest and cleanest method to change filesystems. 3) (Pretty much)[1] Regardless of the filesystems in question, a fresh mkfs and clean sequential transfer of files from the old-fs/backup to the new one is pretty well guaranteed to be better optimized than conversion from an existing filesystem of a different type, particularly one that has been in normal operation for awhile and thus has operational fragmentation of both data and free-space. That's in addition to being less bug-prone, even for a "stable" converter. Restating: So(1) doing a conversion without a backup is irresponsible, (2) the easiest backup and conversion method is directly using the old fs as the backup, and copying over to the freshly mkfs-ed new filesystem, and (3) a freshly mkfs-ed filesystem and sequential copy of files to it from backup, whether that be the old filesystem or not, is going to be more efficient and less bug-prone than an in-place conversion. Given the above, why would /anyone/ /sane/ consider using a converter? It simply doesn't make sense, even if the converter were as stable as the most stable filesystems we have. So as a distro btrfs package maintainer, do what you wish in terms of the converter, but were it me, I might actually consider replacing it with an executable that simply printed out some form of the above argument, with a pointer to the sources should they still be interested after having read that argument.[2] Then, if people really are determined to unnecessarily waste their time to get a less efficient filesystem, possibly risking their data in the process of getting it, they can always build the converter from sources themselves. --- [1] I debated omitting the qualifier as I know of no exceptions, but I'm not a filesystem expert and while I'm a bit skeptical, I suppose it's possible that they might exist. [2] There's actually btrfs precedent for this in the form of the executable built as fsck.btrfs, which does nothing (successfully) but possibly print a message referring people to btrfs check, if run in interactive mode. -- Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Re: btrfs-convert missing in btrfs-tools v4.15.1
Hi Qu, On Sun, Jul 29, 2018 at 07:44:05AM +0800, Qu Wenruo wrote: > > > On 2018年07月29日 05:34, Nicholas D Steeves wrote: > > Resending because I forgot to CC list. > > > > Hi jkexcel, > > > > On 28 July 2018 at 16:50, jkexcel wrote: > >> > >> I'm an end user trying to use btrfs-convert but when I installed > >> btrfs-tools and its dependency btrfs-progs on kubuntu 18.04, the > >> installation was successful, and it shows that v4.15.1-1build1 was > >> installed. > >> > >> However, when using the command # brtfs-convert /dev/sda4 (with the > >> drive unmounted) the resulting error appears "command not found" > >> I also tried command "btrfs convert" in case this was folded into the > >> main tool, but this also failed. > >> > >> 1. Is btrfs-convert still available? > >> > >> 2. Where can I find it? > >> > >> 3. Has btrfs-convert been replaced? what is it's new name? > >> > >> 4. Is it safe to use a downgraded version of btrfs-tools ie: 4.14 or > >> older? > > > > You can blame me for that. In Debian several users had reported > > release-critical issues in btrfs-convert, so I submitted a patch to > > disable it for the forseable future, eg: > > > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864798 > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=854489 > > Both report looks pretty old (4.7~4.9). > > In fact, just in v4.10 we had a big rework for convert. > It should work much better after that. > > Furthermore, we have newer (but smaller) fixes to remove a lot of > BUG_ON(), and do graceful exit for ENOSPC case since then. > > And the design of btrfs-convert (at least for the latest btrfs-convert) > is to ensure until everything goes well, we won't touch any bytes of the > ext* fs (in fact we open the ext* fs in RO mode). > So it at least won't corrupt the ext* fs. > > > > > Also, please consider the official status "As of 4.0 kernels this feature > > is not often used or well tested anymore, and there have been some reports > > that the conversion doesn't work reliably. Feel free to try it out, but > > make sure you have backups" ( > > https://btrfs.wiki.kernel.org/index.php/Conversion_from_Ext3 ). > > The wiki page looks needs to be updated. > > Both btrfs-convert and base btrfs-progs are improving, especially after > v4.10 btrfs-convert goes through a big rework and works well so far, and > even added support for reiserfs under the same framwork. > > So IMHO it's at least worth trying. > > Thanks, > Qu Thank you for sharing the cut-off where success became more likely :-) Debian 9 could have had 4.9.1 at the newest, so it wouldn't have had btrfs-convert. So it sounds like btrfs-convert could have been enabled for the experimental suite (which is almost only used by Debian developers and not users) for 4.10. Looking at the changelog it seems we might have had to disable it again before 4.14.1 or 4.16. I'm happy we're having this conversation now, because the time to give it another try is probably sometime in the next month :-) See my long email to David for the caveats. Cheers, Nicholas signature.asc Description: PGP signature
Re: btrfs-convert missing in btrfs-tools v4.15.1
Hi everyone, Sorry for the delay replying, I've been busy with other work. Reply follows inline. P.S. sorry about the table in this email that is 99 columns wide. On Thu, Aug 09, 2018 at 01:50:46PM +0200, David Sterba wrote: > On Sat, Jul 28, 2018 at 05:34:49PM -0400, Nicholas D Steeves wrote: > > On 28 July 2018 at 16:50, jkexcel wrote: > > > I'm an end user trying to use btrfs-convert but when I installed > > > btrfs-tools and its dependency btrfs-progs on kubuntu 18.04, the > > > installation was successful, and it shows that v4.15.1-1build1 was > > > installed. > > > > > > However, when using the command # brtfs-convert /dev/sda4 (with the > > > drive unmounted) the resulting error appears "command not found" > > > I also tried command "btrfs convert" in case this was folded into the > > > main tool, but this also failed. > > > > > > 1. Is btrfs-convert still available? > > > > > > 2. Where can I find it? > > > > > > 3. Has btrfs-convert been replaced? what is it's new name? > > > > > > 4. Is it safe to use a downgraded version of btrfs-tools ie: 4.14 or > > > older? > > > > You can blame me for that. In Debian several users had reported > > release-critical issues in btrfs-convert, so I submitted a patch to > > disable it for the forseable future, eg: > > > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864798 > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=854489 > > The reports are against version 4.7.3 released one year before the time > of the bug reports. The fix for the reported bug happened in 4.10, that > was half a year before the bug. Debian stable will always have an old version, which will be in use for two to four years. Btrfs-progs 4.7.3 will be in use in Debian 9 until at least 2021, possibly longer. Btw, I strongly believe Debian 9 should have shipped with btrfs-progs 4.9.1, but alas the primary maintainer didn't upload it on time. For "buster" (Debian 10), which will probably be released in mid 2019, the newest possible btrfs-progs version that could be included is whatever is current at the end of January 2019. Exceptions are sometimes granted for an unblock of a newer version. For example, if an LTS kernel won't be released until March, and the release, technical committee, and kernel team decide that's the version we want, then a preapproved exception will be granted. At that time an argument can be made for preapproval of a newer btrfs-progs as well. That said, I try to keep a backported newer version of btrfs-progs for the stable Debian release reasonably up-to-date (backports are available to users on a per-package opt-in basis). That's the channel for feature enablement. Also, my apologies, at the moment this backport is very much out of date--it stalled while investigating which packages would be broken by the library reorganisation; although, from what I can tell that would only be snapper. > > Also, please consider the official status "As of 4.0 kernels this feature > > is not often used or well tested anymore, and there have been some reports > > that the conversion doesn't work reliably. Feel free to try it out, but > > make sure you have backups" ( > > https://btrfs.wiki.kernel.org/index.php/Conversion_from_Ext3 ). > > Sorry that this you take it as official status. The wiki is open to > edits and such claims appear there from time to time. I've removed it, > it's been there since 2015 when it possibly was accurate but it's not > anymore. Is there a more authoritative and up-to-date location for various statuses? It would be nice to have something in btrfs-progs as a table like this, or exportable in some kind of human-friendly format: +---+-++--+---+ |Feature|1st mainline version where feature |LTS kernel 1|LTS kernel 2 |LTS kernel 3 | | |appeared |eg: 4.4 |eg: 4.9 |eg: 4.14 | +---+-++--+---+ |convert|Assume -progs and kernel |exp?|mostly? |stable?| |from |vessions are strongly|| amend | | |ext3/4 |associated, for simplicity. || status with: | | | | ||4.9.z:testing | | +---+-++--+---+ |foo|3.16:danger||exp||mostly||testing||stable|exp |mostly |testing| +---+-++--+---+ Ideally something that could be trivially exported to the format the btrfs.wiki.kernel.org wiki uses. The trick is to make it convenient to maintain... Other than CSV, I'm not sure what would work well for
Re: btrfs-convert missing in btrfs-tools v4.15.1
On Sat, Jul 28, 2018 at 05:34:49PM -0400, Nicholas D Steeves wrote: > On 28 July 2018 at 16:50, jkexcel wrote: > > I'm an end user trying to use btrfs-convert but when I installed > > btrfs-tools and its dependency btrfs-progs on kubuntu 18.04, the > > installation was successful, and it shows that v4.15.1-1build1 was > > installed. > > > > However, when using the command # brtfs-convert /dev/sda4 (with the > > drive unmounted) the resulting error appears "command not found" > > I also tried command "btrfs convert" in case this was folded into the > > main tool, but this also failed. > > > > 1. Is btrfs-convert still available? > > > > 2. Where can I find it? > > > > 3. Has btrfs-convert been replaced? what is it's new name? > > > > 4. Is it safe to use a downgraded version of btrfs-tools ie: 4.14 or > > older? > > You can blame me for that. In Debian several users had reported > release-critical issues in btrfs-convert, so I submitted a patch to > disable it for the forseable future, eg: > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864798 > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=854489 The reports are against version 4.7.3 released one year before the time of the bug reports. The fix for the reported bug happened in 4.10, that was half a year before the bug. > Also, please consider the official status "As of 4.0 kernels this feature > is not often used or well tested anymore, and there have been some reports > that the conversion doesn't work reliably. Feel free to try it out, but > make sure you have backups" ( > https://btrfs.wiki.kernel.org/index.php/Conversion_from_Ext3 ). Sorry that this you take it as official status. The wiki is open to edits and such claims appear there from time to time. I've removed it, it's been there since 2015 when it possibly was accurate but it's not anymore. > I'm happy to hear it is still disabled in Ubuntu, where many more > users would be affected. IIRC OpenSUSE LEAP and SLED 15 reenabled it > (it was previously disabled there), so maybe it needs specific kernel > versions or patches to not trigger RC bugs, and/or very specific > btrfs-progs versions, and/or very specific e2fslibs, and/or specific > combinations? While I very much look forward to the day when > btrfs-convert can be relied on in Debian, I don't think we're there > yet. So if there's no way to update package to newer version or nobody who backports fixes, then it's better not to ship the tool. There's nothing close to the 'specific version of X', besides using more up to date versions. Alternatively, there could have been a separate package with the convert tool, with a warning about the known issues. It's in my interest to ship all tools in distros, but there's also only that much what the upstream community can do. If you're going to reconsider the status of btrfs-convert in Debian, please let me know. -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs-convert missing in btrfs-tools v4.15.1
On 2018年07月29日 05:34, Nicholas D Steeves wrote: > Resending because I forgot to CC list. > > Hi jkexcel, > > On 28 July 2018 at 16:50, jkexcel wrote: >> >> I'm an end user trying to use btrfs-convert but when I installed >> btrfs-tools and its dependency btrfs-progs on kubuntu 18.04, the >> installation was successful, and it shows that v4.15.1-1build1 was >> installed. >> >> However, when using the command # brtfs-convert /dev/sda4 (with the >> drive unmounted) the resulting error appears "command not found" >> I also tried command "btrfs convert" in case this was folded into the >> main tool, but this also failed. >> >> 1. Is btrfs-convert still available? >> >> 2. Where can I find it? >> >> 3. Has btrfs-convert been replaced? what is it's new name? >> >> 4. Is it safe to use a downgraded version of btrfs-tools ie: 4.14 or >> older? > > You can blame me for that. In Debian several users had reported > release-critical issues in btrfs-convert, so I submitted a patch to > disable it for the forseable future, eg: > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864798 > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=854489 Both report looks pretty old (4.7~4.9). In fact, just in v4.10 we had a big rework for convert. It should work much better after that. Furthermore, we have newer (but smaller) fixes to remove a lot of BUG_ON(), and do graceful exit for ENOSPC case since then. And the design of btrfs-convert (at least for the latest btrfs-convert) is to ensure until everything goes well, we won't touch any bytes of the ext* fs (in fact we open the ext* fs in RO mode). So it at least won't corrupt the ext* fs. > > Also, please consider the official status "As of 4.0 kernels this feature > is not often used or well tested anymore, and there have been some reports > that the conversion doesn't work reliably. Feel free to try it out, but > make sure you have backups" ( > https://btrfs.wiki.kernel.org/index.php/Conversion_from_Ext3 ). The wiki page looks needs to be updated. Both btrfs-convert and base btrfs-progs are improving, especially after v4.10 btrfs-convert goes through a big rework and works well so far, and even added support for reiserfs under the same framwork. So IMHO it's at least worth trying. Thanks, Qu > > I'm happy to hear it is still disabled in Ubuntu, where many more > users would be affected. IIRC OpenSUSE LEAP and SLED 15 reenabled it > (it was previously disabled there), so maybe it needs specific kernel > versions or patches to not trigger RC bugs, and/or very specific > btrfs-progs versions, and/or very specific e2fslibs, and/or specific > combinations? While I very much look forward to the day when > btrfs-convert can be relied on in Debian, I don't think we're there > yet. Please take this as an opportunity to test that your backups are > restorable, mkfs.btrfs, and then restore from backup. P.S. I have no > idea if Ubuntu has additional btrfs support. > > Cheers, > Nicholas > signature.asc Description: OpenPGP digital signature
Re: btrfs-convert missing in btrfs-tools v4.15.1
Resending because I forgot to CC list. Hi jkexcel, On 28 July 2018 at 16:50, jkexcel wrote: > > I'm an end user trying to use btrfs-convert but when I installed > btrfs-tools and its dependency btrfs-progs on kubuntu 18.04, the > installation was successful, and it shows that v4.15.1-1build1 was > installed. > > However, when using the command # brtfs-convert /dev/sda4 (with the > drive unmounted) the resulting error appears "command not found" > I also tried command "btrfs convert" in case this was folded into the > main tool, but this also failed. > > 1. Is btrfs-convert still available? > > 2. Where can I find it? > > 3. Has btrfs-convert been replaced? what is it's new name? > > 4. Is it safe to use a downgraded version of btrfs-tools ie: 4.14 or > older? You can blame me for that. In Debian several users had reported release-critical issues in btrfs-convert, so I submitted a patch to disable it for the forseable future, eg: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=864798 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=854489 Also, please consider the official status "As of 4.0 kernels this feature is not often used or well tested anymore, and there have been some reports that the conversion doesn't work reliably. Feel free to try it out, but make sure you have backups" ( https://btrfs.wiki.kernel.org/index.php/Conversion_from_Ext3 ). I'm happy to hear it is still disabled in Ubuntu, where many more users would be affected. IIRC OpenSUSE LEAP and SLED 15 reenabled it (it was previously disabled there), so maybe it needs specific kernel versions or patches to not trigger RC bugs, and/or very specific btrfs-progs versions, and/or very specific e2fslibs, and/or specific combinations? While I very much look forward to the day when btrfs-convert can be relied on in Debian, I don't think we're there yet. Please take this as an opportunity to test that your backups are restorable, mkfs.btrfs, and then restore from backup. P.S. I have no idea if Ubuntu has additional btrfs support. Cheers, Nicholas signature.asc Description: PGP signature
