Hello community, here is the log from the commit of package libstorage-ng for openSUSE:Factory checked in at 2019-04-28 20:00:24 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/libstorage-ng (Old) and /work/SRC/openSUSE:Factory/.libstorage-ng.new.5536 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "libstorage-ng" Sun Apr 28 20:00:24 2019 rev:53 rq:698282 version:4.1.117 Changes: -------- --- /work/SRC/openSUSE:Factory/libstorage-ng/libstorage-ng.changes 2019-04-22 12:24:08.868921631 +0200 +++ /work/SRC/openSUSE:Factory/.libstorage-ng.new.5536/libstorage-ng.changes 2019-04-28 20:00:27.406898622 +0200 @@ -1,0 +2,37 @@ +Fri Apr 26 14:52:00 UTC 2019 - [email protected] + +- merge gh#openSUSE/libstorage-ng#643 +- extended documentation +- 4.1.117 + +-------------------------------------------------------------------- +Fri Apr 26 05:37:31 UTC 2019 - [email protected] + +- merge gh#openSUSE/libstorage-ng#642 +- added documentation about multiple devices btrfs +- added example +- 4.1.116 + +-------------------------------------------------------------------- +Thu Apr 25 13:00:12 UTC 2019 - [email protected] + +- merge gh#openSUSE/libstorage-ng#641 +- handle mixed mode in output of 'btrfs filesystem df' +- 4.1.115 + +-------------------------------------------------------------------- +Thu Apr 25 08:48:39 UTC 2019 - [email protected] + +- merge gh#openSUSE/libstorage-ng#640 +- parse UUID in output of mkfs.btrfs +- fixed exception message +- 4.1.114 + +-------------------------------------------------------------------- +Tue Apr 23 06:39:03 UTC 2019 - [email protected] + +- merge gh#openSUSE/libstorage-ng#639 +- Use the Docker image from OBS +- 4.1.113 + +-------------------------------------------------------------------- Old: ---- libstorage-ng-4.1.112.tar.xz New: ---- libstorage-ng-4.1.117.tar.xz ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ libstorage-ng.spec ++++++ --- /var/tmp/diff_new_pack.79wwPI/_old 2019-04-28 20:00:28.634897882 +0200 +++ /var/tmp/diff_new_pack.79wwPI/_new 2019-04-28 20:00:28.642897877 +0200 @@ -18,7 +18,7 @@ %define libname %{name}1 Name: libstorage-ng -Version: 4.1.112 +Version: 4.1.117 Release: 0 Summary: Library for storage management License: GPL-2.0-only ++++++ libstorage-ng-4.1.112.tar.xz -> libstorage-ng-4.1.117.tar.xz ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/libstorage-ng-4.1.112/Dockerfile new/libstorage-ng-4.1.117/Dockerfile --- old/libstorage-ng-4.1.112/Dockerfile 2019-04-18 18:02:17.000000000 +0200 +++ new/libstorage-ng-4.1.117/Dockerfile 2019-04-26 16:52:00.000000000 +0200 @@ -1,2 +1,2 @@ -FROM yastdevel/libstorage-ng +FROM registry.opensuse.org/yast/head/containers/libstorage-ng:latest COPY . /usr/src/app diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/libstorage-ng-4.1.112/VERSION new/libstorage-ng-4.1.117/VERSION --- old/libstorage-ng-4.1.112/VERSION 2019-04-18 18:02:17.000000000 +0200 +++ new/libstorage-ng-4.1.117/VERSION 2019-04-26 16:52:00.000000000 +0200 @@ -1 +1 @@ -4.1.112 +4.1.117 diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/libstorage-ng-4.1.112/doc/btrfs-multiple-devices.md new/libstorage-ng-4.1.117/doc/btrfs-multiple-devices.md --- old/libstorage-ng-4.1.112/doc/btrfs-multiple-devices.md 1970-01-01 01:00:00.000000000 +0100 +++ new/libstorage-ng-4.1.117/doc/btrfs-multiple-devices.md 2019-04-26 16:52:00.000000000 +0200 @@ -0,0 +1,123 @@ + +Multiple Devices Btrfs +====================== + + +Create Btrfs +------------ + +Before creating a multiple devices btrfs the metadata and data RAID +level can be set. The value DEFAULT will not pass a RAID level to +mkfs.btrfs thus mkfs.btrfs will decide the RAID level. + +When using the "--mixed" option (via set_mkfs_options()) the metadata +and data RAID level must be identical. + + +Mounting Degraded Btrfs +----------------------- + +Mounting a degraded btrfs requires the 'degraded' mount option. + +Using the 'degraded' mount option in /etc/fstab by default was not recommended +by btrfs experts (on IRC). + +To be decided: What should be done during probing with degraded btrfses? + +The 'degraded' option is a nop if no devices are missing. + + +Querying RAID Levels +-------------------- + +The RAID levels can be queried using get_{metadata,data}_raid_level(). + +During a balance job to change the RAID level the RAID level as +reported by btrfs can be a combination of RAID levels. In that case +get_{metadata,data}_raid_level() returns only one of the reported RAID +levels. + + +Changing RAID Levels +-------------------- + +Changing the RAID levels required running btrfs balance job. + + btrfs balance start -dconvert=raid1,soft -mconvert=raid1,soft /test + +Changing the RAID levels can require copying huge amounts of data so is +potentially very time consuming. Usually such operations are not supported by +libstorage-ng. + +As a benefit the balance can be started in the background. But this fails if +there is already a balance job running (without an error message from the +btrfs command). The system does not provide a way to queue balance jobs and it +should not be the task of libstorage-ng to implement such a way. + +The balance job can also fail due to missing space. When run in the background +reporting that to the user is not possible (the balance job starts and simply +fails later). + +If the filesystem needs to be mounted temporarily for the balance job, +starting the balance job in the background would prevent unmounting. Normally +this should be a minor problem since the philosophy of btrfs is to be always +mounted. But during installation this could be problematic. + +Thus operations requiring a balance job are not supported. + + +Adding a Device +--------------- + +Adding a device to an existing btrfs is easy. + + btrfs device add /dev/sdc1 /test + +Sometimes after that the RAID levels should be changed, e.g. from DUP +to RAID1, but that is not supported (see above). + + +Removing a Device +----------------- + +Removing a device from an existing btrfs is at first sight also easy. + + btrfs device delete /dev/sdc1 /test + +"It redistributes any extents in use on the device being removed to the other +devices in the filesystem" [1]. So this also may be very slow. + +Even more problematic is to know in advance whether the operation will +succeed. It may fail due to two reasons: + +1. Minimal number of devices for RAID level. + + Should be easy to check in advance. + +2. Not enough free space. + + Using 'btrfs filesystem usage /test' it can be checked how much of a device + is used. But how likely is the case that only a few GiB are used? And what + happens when removing several devices? + +For the time being removing devices is not supported. + + +Resizing a Device +----------------- + +The btrfs command to resize a block device used by btrfs defaults to use the +devid 1. This must be extended to query the devid and pass it the btrfs (since +btrfs does not accept the device name here). + +Shrinking can again be very slow. Additionally 'btrfs inspect-internal +min-dev-size [--id 1] /test' still reports wrong values. + +For the time being shrinking is not supported. + + +References +---------- + +[1] https://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices + diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/libstorage-ng-4.1.112/doc/btrfs.md new/libstorage-ng-4.1.117/doc/btrfs.md --- old/libstorage-ng-4.1.112/doc/btrfs.md 2019-04-18 18:02:17.000000000 +0200 +++ new/libstorage-ng-4.1.117/doc/btrfs.md 2019-04-26 16:52:00.000000000 +0200 @@ -34,3 +34,11 @@ objects when the subvol option is the default subvolume. This behaviour looks more natural in the SUSE btrfs setup - but may change anytime if required. + +Modification to btrfs +--------------------- + +Modification to a btrfs filesystem need the filesystem to be +mounted. libstorage-ng will temporarily mount the filesystem if +needed. + diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/libstorage-ng-4.1.112/storage/Filesystems/BtrfsImpl.cc new/libstorage-ng-4.1.117/storage/Filesystems/BtrfsImpl.cc --- old/libstorage-ng-4.1.112/storage/Filesystems/BtrfsImpl.cc 2019-04-18 18:02:17.000000000 +0200 +++ new/libstorage-ng-4.1.117/storage/Filesystems/BtrfsImpl.cc 2019-04-26 16:52:00.000000000 +0200 @@ -410,7 +410,7 @@ { // TRANSLATORS: error message error_callback(prober.get_probe_callbacks(), sformat(_("Probing file system with UUID %s failed"), - detected_btrfs.uuid, exception)); + detected_btrfs.uuid), exception); } } } @@ -532,6 +532,26 @@ void + Btrfs::Impl::parse_mkfs_output(const vector<string>& stdout) + { + static const regex uuid_regex("UUID:[ \t]+(" UUID_REGEX ")", regex::extended); + + smatch match; + + for (const string& line : stdout) + { + if (regex_match(line, match, uuid_regex) && match.size() == 2) + { + set_uuid(match[1]); + return; + } + } + + ST_THROW(Exception("UUID not found in output of mkfs.btrfs")); + } + + + void Btrfs::Impl::do_create() { string cmd_line = MKFSBTRFSBIN " --force"; @@ -552,9 +572,7 @@ SystemCmd cmd(cmd_line, SystemCmd::DoThrow); - // TODO uuid is included in mkfs output - - probe_uuid(); + parse_mkfs_output(cmd.stdout()); // This would fit better in do_mount(), but that one is a const method // which would not allow to set the snapper_config member variable. diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/libstorage-ng-4.1.112/storage/Filesystems/BtrfsImpl.h new/libstorage-ng-4.1.117/storage/Filesystems/BtrfsImpl.h --- old/libstorage-ng-4.1.112/storage/Filesystems/BtrfsImpl.h 2019-04-18 18:02:17.000000000 +0200 +++ new/libstorage-ng-4.1.117/storage/Filesystems/BtrfsImpl.h 2019-04-26 16:52:00.000000000 +0200 @@ -141,6 +141,8 @@ virtual void do_pre_mount() const override; + void parse_mkfs_output(const vector<string>& stdout); + private: bool configure_snapper; diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/libstorage-ng-4.1.112/storage/SystemInfo/CmdBtrfs.cc new/libstorage-ng-4.1.117/storage/SystemInfo/CmdBtrfs.cc --- old/libstorage-ng-4.1.112/storage/SystemInfo/CmdBtrfs.cc 2019-04-18 18:02:17.000000000 +0200 +++ new/libstorage-ng-4.1.117/storage/SystemInfo/CmdBtrfs.cc 2019-04-26 16:52:00.000000000 +0200 @@ -294,6 +294,7 @@ { static const regex metadata_rx("Metadata, ([A-Za-z0-9]+):.*", regex::extended); static const regex data_rx("Data, ([A-Za-z0-9]+):.*", regex::extended); + static const regex mixed_rx("Data\\+Metadata, ([A-Za-z0-9]+):.*", regex::extended); smatch match; @@ -310,6 +311,12 @@ string tmp = boost::to_upper_copy(match[1].str(), locale::classic()); data_raid_level = toValueWithFallback(tmp, BtrfsRaidLevel::UNKNOWN); } + + if (regex_match(line, match, mixed_rx) && match.size() == 2) + { + string tmp = boost::to_upper_copy(match[1].str(), locale::classic()); + metadata_raid_level = data_raid_level = toValueWithFallback(tmp, BtrfsRaidLevel::UNKNOWN); + } } y2mil(*this); diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/libstorage-ng-4.1.112/testsuite/SystemInfo/btrfs-filesystem-df.cc new/libstorage-ng-4.1.117/testsuite/SystemInfo/btrfs-filesystem-df.cc --- old/libstorage-ng-4.1.112/testsuite/SystemInfo/btrfs-filesystem-df.cc 2019-04-18 18:02:17.000000000 +0200 +++ new/libstorage-ng-4.1.117/testsuite/SystemInfo/btrfs-filesystem-df.cc 2019-04-26 16:52:00.000000000 +0200 @@ -49,7 +49,6 @@ "System, DUP: total=8.00MiB, used=16.00KiB", "Metadata, DUP: total=102.25MiB, used=112.00KiB", "GlobalReserve, single: total=16.00MiB, used=0.00B" - }; vector<string> output = { @@ -57,4 +56,49 @@ }; check(input, output); +} + + +/* + * In mixed mode metadata and data are identical and reported + * together. + */ +BOOST_AUTO_TEST_CASE(parse2) +{ + vector<string> input = { + "System, single: total=4.00MiB, used=4.00KiB", + "Data+Metadata, single: total=840.00MiB, used=220.00KiB", + "GlobalReserve, single: total=16.00MiB, used=0.00B" + }; + + vector<string> output = { + "metadata-raid-level:SINGLE data-raid-level:SINGLE" + }; + + check(input, output); +} + + +/* + * During a balance job to convert the RAID level several RAID levels + * can be reported. Since this is only an interim state is not handle + * it all its beauty. Instead just the last reported RAID level is + * used. + */ +BOOST_AUTO_TEST_CASE(parse3) +{ + vector<string> input = { + "Data, RAID10: total=4.00GiB, used=3.35GiB", + "Data, RAID5: total=3.00GiB, used=509.71MiB", + "System, RAID10: total=64.00MiB, used=16.00KiB", + "Metadata, RAID10: total=512.00MiB, used=247.55MiB", + "Metadata, RAID5: total=288.00MiB, used=5.56MiB", + "GlobalReserve, single: total=23.91MiB, used=0.00B" + }; + + vector<string> output = { + "metadata-raid-level:RAID5 data-raid-level:RAID5" + }; + + check(input, output); } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/libstorage-ng-4.1.112/testsuite/actions/btrfs1-mockup.xml new/libstorage-ng-4.1.117/testsuite/actions/btrfs1-mockup.xml --- old/libstorage-ng-4.1.112/testsuite/actions/btrfs1-mockup.xml 2019-04-18 18:02:17.000000000 +0200 +++ new/libstorage-ng-4.1.117/testsuite/actions/btrfs1-mockup.xml 2019-04-26 16:52:00.000000000 +0200 @@ -5,11 +5,28 @@ <name>/sbin/udevadm settle --timeout=20</name> </Command> <Command> - <name>/sbin/mkfs.btrfs --force --metadata=RAID1 --data=RAID0 --mixed '/dev/sda' '/dev/sdb' '/dev/sdc' '/dev/sdd'</name> - </Command> - <Command> - <name>/sbin/blkid -c '/dev/null' '/dev/sda'</name> - <stdout>/dev/sda: UUID="318eac77-bbe5-40b4-a8fd-b30ec45d0003" UUID_SUB="b62c72f0-2798-4a79-8307-c07a3781fbb2" TYPE="btrfs"</stdout> + <name>/sbin/mkfs.btrfs --force --metadata=RAID1 --data=RAID0 --nodiscard '/dev/sda' '/dev/sdb' '/dev/sdc' '/dev/sdd'</name> + <stdout>btrfs-progs v4.19.1 </stdout> + <stdout>See http://btrfs.wiki.kernel.org for more information.</stdout> + <stdout></stdout> + <stdout>Label: (null)</stdout> + <stdout>UUID: 318eac77-bbe5-40b4-a8fd-b30ec45d0003</stdout> + <stdout>Node size: 16384</stdout> + <stdout>Sector size: 4096</stdout> + <stdout>Filesystem size: 31.99GiB</stdout> + <stdout>Block group profiles:</stdout> + <stdout> Data: RAID0 3.20GiB</stdout> + <stdout> Metadata: RAID1 1.00GiB</stdout> + <stdout> System: RAID1 8.00MiB</stdout> + <stdout>SSD detected: no</stdout> + <stdout>Incompat features: extref, skinny-metadata</stdout> + <stdout>Number of devices: 4</stdout> + <stdout>Devices:</stdout> + <stdout> ID SIZE PATH</stdout> + <stdout> 1 8.00GiB /dev/sda</stdout> + <stdout> 2 8.00GiB /dev/sdb</stdout> + <stdout> 3 8.00GiB /dev/sdc</stdout> + <stdout> 4 8.00GiB /dev/sdd</stdout> </Command> </Commands> </Mockup> diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/libstorage-ng-4.1.112/testsuite/actions/btrfs1-staging.xml new/libstorage-ng-4.1.117/testsuite/actions/btrfs1-staging.xml --- old/libstorage-ng-4.1.112/testsuite/actions/btrfs1-staging.xml 2019-04-18 18:02:17.000000000 +0200 +++ new/libstorage-ng-4.1.117/testsuite/actions/btrfs1-staging.xml 2019-04-26 16:52:00.000000000 +0200 @@ -81,7 +81,7 @@ <sid>46</sid> <metadata-raid-level>RAID1</metadata-raid-level> <data-raid-level>RAID0</data-raid-level> - <mkfs-options>--mixed</mkfs-options> + <mkfs-options>--nodiscard</mkfs-options> </Btrfs> <BtrfsSubvolume> <sid>47</sid>
