Re: [gentoo-dev] [PATCH] glep-0074: Remove single filesystem limitation
On Thu, Feb 08, 2018 at 06:09:22PM +0100, Michał Górny wrote: > Remove the limitation that all files covered by the Manifest must reside > on a single filesystem. This breaks valid uses of overlayfs without > providing any real advantage. Replying for the record: I approve this change, because I didn't realize how overlayfs could cause files & directories to show up in a single place with different device IDs. Keeping overlayfs working is more important than the slight security concerns of traversing device boundaries. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
Re: [gentoo-dev] [PATCH] glep-0074: Remove single filesystem limitation
W dniu czw, 08.02.2018 o godzinie 17∶25 +, użytkownik M. J. Everitt napisał: > On 08/02/18 17:09, Michał Górny wrote: > > Remove the limitation that all files covered by the Manifest must reside > > on a single filesystem. This breaks valid uses of overlayfs without > > providing any real advantage. > > > > The removal is justified further in the updated rationale section. > > --- > > glep-0074.rst | 66 > > +++ > > 1 file changed, 39 insertions(+), 27 deletions(-) > > > > RST: https://dev.gentoo.org/~mgorny/tmp/glep-0074.rst > > HTML: https://dev.gentoo.org/~mgorny/tmp/glep-0074.html > > > > diff --git a/glep-0074.rst b/glep-0074.rst > > index 3835247..2f8deb2 100644 > > --- a/glep-0074.rst > > +++ b/glep-0074.rst > > @@ -6,10 +6,10 @@ Author: Michał Górny, > > Ulrich Müller > > Type: Standards Track > > Status: Final > > -Version: 1 > > +Version: 1.1 > > Created: 2017-10-21 > > -Last-Modified: 2017-12-16 > > -Post-History: 2017-10-26, 2017-11-16 > > +Last-Modified: 2018-02-08 > > +Post-History: 2017-10-26, 2017-11-16, 2018-02-08 > > Content-Type: text/x-rst > > Requires: 59, 61 > > Replaces: 44, 58, 60 > > @@ -126,13 +126,6 @@ a different file type. If the tree contain files of > > other types > > that are not otherwise ignored, they need to be covered by an explicit > > ``IGNORE``. > > > > -All the local (non-``DIST``) files covered by a Manifest tree must > > -reside on the same filesystem. It is an error to specify entries > > -applying to files on another filesystem. If files or directories that > > -are not otherwise ignored reside on a different filesystem, or symbolic > > -links point to targets on a different filesystem, they must > > -be explicitly excluded via ``IGNORE``. > > - > > > > Path and filename encoding > > -- > > @@ -325,22 +318,18 @@ Algorithm for finding parent Manifests > > In order to find the top-level Manifest from the current directory > > the following algorithm can be used: > > > > -1. Store the current directory as *original* and the device ID > > - of the containing filesystem (``st_dev``) as *startdev*, > > - > > -2. If the device ID of the containing filesystem (``st_dev``) > > - of the current directory is different than *startdev*, stop. > > +1. Store the current directory as *original*, > > > > -3. If the current directory contains a ``Manifest`` file: > > +2. If the current directory contains a ``Manifest`` file: > > > > a. If an ``IGNORE`` entry in the ``Manifest`` file covers > >the *original* directory (or one of the parent directories), stop. > > > > b. Otherwise, store the current directory as *last_found*. > > > > -4. If the current directory is the root system directory (``/``), stop. > > +3. If the current directory is the root system directory (``/``), stop. > > > > -5. Otherwise, enter the parent directory and jump to step 2. > > +4. Otherwise, enter the parent directory and jump to step 2. > > > > Once the algorithm stops, *last_found* will contain the relevant > > top-level Manifest. If *last_found* is null, then the directory tree > > > > > > Observation: does RST not support auto-numbering like other formats? It > would make renumbering lists like this much easier (and from programming > experience, less error-prone). If not, #FEATUREREQ ... :] The whole purpose of WYWIWYG is to write *proper* text and not some magical syntax that magically gets transformed to auto-numbered list. Except when the format doesn't get it right. Also, this comment is entirely off-topic and not useful to the thread at all. Please cease those pointless off-topics. -- Best regards, Michał Górny
Re: [gentoo-dev] [PATCH] glep-0074: Remove single filesystem limitation
On 08/02/18 17:09, Michał Górny wrote: > Remove the limitation that all files covered by the Manifest must reside > on a single filesystem. This breaks valid uses of overlayfs without > providing any real advantage. > > The removal is justified further in the updated rationale section. > --- > glep-0074.rst | 66 > +++ > 1 file changed, 39 insertions(+), 27 deletions(-) > > RST: https://dev.gentoo.org/~mgorny/tmp/glep-0074.rst > HTML: https://dev.gentoo.org/~mgorny/tmp/glep-0074.html > > diff --git a/glep-0074.rst b/glep-0074.rst > index 3835247..2f8deb2 100644 > --- a/glep-0074.rst > +++ b/glep-0074.rst > @@ -6,10 +6,10 @@ Author: Michał Górny, > Ulrich Müller > Type: Standards Track > Status: Final > -Version: 1 > +Version: 1.1 > Created: 2017-10-21 > -Last-Modified: 2017-12-16 > -Post-History: 2017-10-26, 2017-11-16 > +Last-Modified: 2018-02-08 > +Post-History: 2017-10-26, 2017-11-16, 2018-02-08 > Content-Type: text/x-rst > Requires: 59, 61 > Replaces: 44, 58, 60 > @@ -126,13 +126,6 @@ a different file type. If the tree contain files of > other types > that are not otherwise ignored, they need to be covered by an explicit > ``IGNORE``. > > -All the local (non-``DIST``) files covered by a Manifest tree must > -reside on the same filesystem. It is an error to specify entries > -applying to files on another filesystem. If files or directories that > -are not otherwise ignored reside on a different filesystem, or symbolic > -links point to targets on a different filesystem, they must > -be explicitly excluded via ``IGNORE``. > - > > Path and filename encoding > -- > @@ -325,22 +318,18 @@ Algorithm for finding parent Manifests > In order to find the top-level Manifest from the current directory > the following algorithm can be used: > > -1. Store the current directory as *original* and the device ID > - of the containing filesystem (``st_dev``) as *startdev*, > - > -2. If the device ID of the containing filesystem (``st_dev``) > - of the current directory is different than *startdev*, stop. > +1. Store the current directory as *original*, > > -3. If the current directory contains a ``Manifest`` file: > +2. If the current directory contains a ``Manifest`` file: > > a. If an ``IGNORE`` entry in the ``Manifest`` file covers >the *original* directory (or one of the parent directories), stop. > > b. Otherwise, store the current directory as *last_found*. > > -4. If the current directory is the root system directory (``/``), stop. > +3. If the current directory is the root system directory (``/``), stop. > > -5. Otherwise, enter the parent directory and jump to step 2. > +4. Otherwise, enter the parent directory and jump to step 2. > > Once the algorithm stops, *last_found* will contain the relevant > top-level Manifest. If *last_found* is null, then the directory tree > Observation: does RST not support auto-numbering like other formats? It would make renumbering lists like this much easier (and from programming experience, less error-prone). If not, #FEATUREREQ ... :] signature.asc Description: OpenPGP digital signature
[gentoo-dev] [PATCH] glep-0074: Remove single filesystem limitation
Remove the limitation that all files covered by the Manifest must reside on a single filesystem. This breaks valid uses of overlayfs without providing any real advantage. The removal is justified further in the updated rationale section. --- glep-0074.rst | 66 +++ 1 file changed, 39 insertions(+), 27 deletions(-) RST: https://dev.gentoo.org/~mgorny/tmp/glep-0074.rst HTML: https://dev.gentoo.org/~mgorny/tmp/glep-0074.html diff --git a/glep-0074.rst b/glep-0074.rst index 3835247..2f8deb2 100644 --- a/glep-0074.rst +++ b/glep-0074.rst @@ -6,10 +6,10 @@ Author: Michał Górny, Ulrich Müller Type: Standards Track Status: Final -Version: 1 +Version: 1.1 Created: 2017-10-21 -Last-Modified: 2017-12-16 -Post-History: 2017-10-26, 2017-11-16 +Last-Modified: 2018-02-08 +Post-History: 2017-10-26, 2017-11-16, 2018-02-08 Content-Type: text/x-rst Requires: 59, 61 Replaces: 44, 58, 60 @@ -126,13 +126,6 @@ a different file type. If the tree contain files of other types that are not otherwise ignored, they need to be covered by an explicit ``IGNORE``. -All the local (non-``DIST``) files covered by a Manifest tree must -reside on the same filesystem. It is an error to specify entries -applying to files on another filesystem. If files or directories that -are not otherwise ignored reside on a different filesystem, or symbolic -links point to targets on a different filesystem, they must -be explicitly excluded via ``IGNORE``. - Path and filename encoding -- @@ -325,22 +318,18 @@ Algorithm for finding parent Manifests In order to find the top-level Manifest from the current directory the following algorithm can be used: -1. Store the current directory as *original* and the device ID - of the containing filesystem (``st_dev``) as *startdev*, - -2. If the device ID of the containing filesystem (``st_dev``) - of the current directory is different than *startdev*, stop. +1. Store the current directory as *original*, -3. If the current directory contains a ``Manifest`` file: +2. If the current directory contains a ``Manifest`` file: a. If an ``IGNORE`` entry in the ``Manifest`` file covers the *original* directory (or one of the parent directories), stop. b. Otherwise, store the current directory as *last_found*. -4. If the current directory is the root system directory (``/``), stop. +3. If the current directory is the root system directory (``/``), stop. -5. Otherwise, enter the parent directory and jump to step 2. +4. Otherwise, enter the parent directory and jump to step 2. Once the algorithm stops, *last_found* will contain the relevant top-level Manifest. If *last_found* is null, then the directory tree @@ -594,16 +583,39 @@ additional files in it, skipping some of the categories from syncing. This configuration can extend beyond the limits of this GLEP, e.g. by allowing wildcards or regular expressions. -The algorithm is restricted to work on a single filesystem. This is -mostly relevant when scanning for top-level Manifest -- we do not want -to cross filesystem boundaries then. However, to ensure consistent -bidirectional behavior we need to also ban them when operating downwards -the tree. -The directories and files on different filesystems need to be ignored -explicitly as implicitly skipping them would cause confusion. -In particular, tools might then claim that a file does not exist when -it clearly does because it was skipped due to filesystem boundaries. +Cross-filesystem Manifests +-- + +The first version of this specification had an additional requirement +that all files covered by the Manifest tree must reside on a single +filesystem. This requirement has been removed in version 1.1 for +the reasons outlined in this section. + +The original rationale stated that this restriction aims to prevent +crossing filesystem boundaries in the top-level Manifest lookup +algorithm. While that seemed a good idea at the time, there is no real +reason to prevent that and this particular method worked correctly only +if the files were placed in a dedicated filesystem. + +Worse than that, the original rationale did not anticipate the use +of overlayfs which combines multiple filesystems while preserving their +original metadata, including device and inode numbers. As a result, +if the repository was checked out to an overlayfs, it was quite possible +that different files had different device numbers, and the Manifest +checks failed due to crossing filesystem boundaries. + +Given no clear solution to that and no good reason to reject use +of overlayfs, the restriction was lifted. + +The only potential drawback of this is that the implementation may now +follow maliciously placed symbolic links pointing outside the tree. +If a regular file was replaced by such a symlink, the user could +be tricked into reporting the