Script 'mail_helper' called by obssrc Hello community, here is the log from the commit of package apptainer for openSUSE:Factory checked in at 2022-08-19 17:56:44 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/apptainer (Old) and /work/SRC/openSUSE:Factory/.apptainer.new.2083 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "apptainer" Fri Aug 19 17:56:44 2022 rev:5 rq:998138 version:1.1.0 Changes: -------- --- /work/SRC/openSUSE:Factory/apptainer/apptainer.changes 2022-08-05 19:51:54.573596430 +0200 +++ /work/SRC/openSUSE:Factory/.apptainer.new.2083/apptainer.changes 2022-08-19 17:59:19.980459320 +0200 @@ -1,0 +2,38 @@ +Fri Aug 19 10:07:20 UTC 2022 - Christian Goll <[email protected]> + +- Udpated to version 1.1.0-rc2 with following changes: + * Fixed longstanding bug in the underlay logic when there are nested bind + points separated by more than one path level, for example /var and + /var/lib/yum, and the path didn't exist in the container image. The bug + only caused an error when there was a directory in the container image that + didn't exist on the host. + * Improved wildcard matching in the %files directive of build definition + files by replacing usage of sh with the mvdan.cc library. + * Replaced checks for compatible filesystem types when using fuse-overlayfs + with an INFO message when an incompatible filesystem type causes it to be + unwritable by a fakeroot user. + * The --nvccli option now works without --fakeroot. In that case the option + can be used with --writable-tmpfs instead of --writable, and + --writable-tmpfs is implied if neither option is given. Note that also + /usr/bin has to be writable by the user, so without --fakeroot that + probably requires a sandbox image that was built with --fix-perms. + * The --nvccli option implies --nv. + * Configure squashfuse to always show files to be owned by the current user. + That's especially important for fakeroot to prevent most of the files from + looking like they are owned by user 65534. + * The fakeroot command can now be used even if $PATH is empty in the + environment of the apptainer command. + * Allow the newuidmap command to be missing if the current user is not listed + in /etc/subuid. + * Require the uidmap package in Debian packaging. + * Improved error handling of unsupported pass protected PEM files with + encrypted containers. + * Ensure bootstrap_history directory is populated with previous definition + files, present in source containers used in a build. + * Add additional options to the build command for testing different fakeroot + modes: --userns like the action flag and hidden options --ignore-subuid, + --ignore-fakeroot-command, and --ignore-userns. + * Require root user early when building an encrypted container. +- removed upstream incorated patch fix-32bit-compilation.patch + +------------------------------------------------------------------- Old: ---- apptainer-1.1.0-rc.1.tar.gz fix-32bit-compilation.patch New: ---- apptainer-1.1.0-rc.2.tar.gz ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ apptainer.spec ++++++ --- /var/tmp/diff_new_pack.8D1ROq/_old 2022-08-19 17:59:20.612460645 +0200 +++ /var/tmp/diff_new_pack.8D1ROq/_new 2022-08-19 17:59:20.616460653 +0200 @@ -19,7 +19,7 @@ %define apptainerpath src/github.com/apptainer/ %define _buildshell /bin/bash -%define vers_suffix -rc.1 +%define vers_suffix -rc.2 Summary: Application and environment virtualization License: BSD-3-Clause-LBNL @@ -35,7 +35,6 @@ Source3: SLE-15SP3.def Source5: %{name}-rpmlintrc Source10: vendor.tar.gz -Patch1: fix-32bit-compilation.patch BuildRequires: cryptsetup BuildRequires: fdupes BuildRequires: gcc @@ -68,7 +67,6 @@ cp %{S:1} %{S:2} %{S:3} . mv %{name}-%{version}%{?vers_suffix} %{name} cd %{_builddir}/gopath/%{apptainerpath}/apptainer -%patch1 -p1 %build cd %{name} ++++++ apptainer-1.1.0-rc.1.tar.gz -> apptainer-1.1.0-rc.2.tar.gz ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/.github/workflows/ci.yml new/apptainer-1.1.0-rc.2/.github/workflows/ci.yml --- old/apptainer-1.1.0-rc.1/.github/workflows/ci.yml 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/.github/workflows/ci.yml 2022-08-17 13:47:34.000000000 +0200 @@ -169,7 +169,7 @@ go-version: 1.18.4 - name: Fetch deps - run: sudo apt-get -q update && sudo apt-get install -y build-essential squashfs-tools squashfuse fuse-overlayfs fakeroot libseccomp-dev cryptsetup + run: sudo apt-get -q update && sudo apt-get install -y build-essential squashfs-tools squashfuse fuse-overlayfs fakeroot fuse2fs libseccomp-dev cryptsetup - name: Build and install Apptainer run: | @@ -244,7 +244,7 @@ - name: Fetch deps if: env.run_tests - run: sudo apt-get -q update && sudo apt-get install -y build-essential squashfs-tools squashfuse fuse-overlayfs fakeroot libseccomp-dev cryptsetup + run: sudo apt-get -q update && sudo apt-get install -y build-essential uidmap squashfs-tools squashfuse fuse-overlayfs fakeroot fuse2fs libseccomp-dev cryptsetup - name: Build and install Apptainer if: env.run_tests diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/CHANGELOG.md new/apptainer-1.1.0-rc.2/CHANGELOG.md --- old/apptainer-1.1.0-rc.1/CHANGELOG.md 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/CHANGELOG.md 2022-08-17 13:47:34.000000000 +0200 @@ -5,6 +5,52 @@ and re-branded as Apptainer. For older changes see the [archived Singularity change log](https://github.com/apptainer/singularity/blob/release-3.8/CHANGELOG.md). +## v1.1.0-rc.2 - \[2022-08-16\] + +### Changed defaults / behaviours + +- Fixed longstanding bug in the underlay logic when there are nested bind + points separated by more than one path level, for example `/var` and + `/var/lib/yum`, and the path didn't exist in the container image. + The bug only caused an error when there was a directory in the container + image that didn't exist on the host. +- Improved wildcard matching in the %files directive of build definition + files by replacing usage of sh with the mvdan.cc library. +- Replaced checks for compatible filesystem types when using fuse-overlayfs + with an INFO message when an incompatible filesystem type causes it to + be unwritable by a fakeroot user. +- Mount the user's home directory at `/root` when using `--fakeroot` in + the setuid flow (fixes a regression introduced in 1.1.0-rc.1 which didn't + impact non-setuid flow). +- The `--nvccli` option now works without `--fakeroot`. In that case the + option can be used with `--writable-tmpfs` instead of `--writable`, + and `--writable-tmpfs` is implied if neither option is given. + Note that also `/usr/bin` has to be writable by the user, so without + `--fakeroot` that probably requires a sandbox image that was built with + `--fix-perms`. +- The `--nvccli` option implies `--nv`. +- Configure squashfuse to always show files to be owned by the current user. + That's especially important for fakeroot to prevent most of the files + from looking like they are owned by user 65534. +- The fakeroot command can now be used even if $PATH is empty in the + environment of the apptainer command. +- Allow the ``newuidmap`` command to be missing if the current user is not + listed in ``/etc/subuid``. +- Require the ``uidmap`` package in Debian packaging. +- Improved error handling of unsupported pass protected PEM files with + encrypted containers. +- Require fuse2fs in RPM packaging. In EPEL7 the package is called fuse2fs, + otherwise it is in e2fsprogs. +- Require the fuse-overlayfs package for all RPM packages instead of just + on el7 because it is sometimes useful even with kernel support for + unprivileged overlayfs. +- Ensure bootstrap_history directory is populated with previous definition + files, present in source containers used in a build. +- Add additional options to the build command for testing different fakeroot + modes: `--userns` like the action flag and hidden options `--ignore-subuid`, + `--ignore-fakeroot-command`, and `--ignore-userns`. +- Require root user early when building an encrypted container. + ## v1.1.0-rc.1 - \[2022-08-01\] ### Changed defaults / behaviours diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/INSTALL.md new/apptainer-1.1.0-rc.2/INSTALL.md --- old/apptainer-1.1.0-rc.1/INSTALL.md 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/INSTALL.md 2022-08-17 13:47:34.000000000 +0200 @@ -20,6 +20,7 @@ build-essential \ libseccomp-dev \ pkg-config \ + uidmap \ squashfs-tools \ squashfuse \ fuse2fs \ @@ -43,6 +44,7 @@ squashfuse \ fuse-overlayfs \ fakeroot \ + /usr/*bin/fuse2fs \ cryptsetup \ wget git ``` @@ -134,7 +136,7 @@ for example: ```sh -git checkout v1.1.0-rc1 +git checkout v1.1.0-rc.2 ``` ## Compiling Apptainer @@ -207,7 +209,7 @@ <!-- markdownlint-disable MD013 --> ```sh -VERSION=1.1.0-rc.1 # this is the apptainer version, change as you need +VERSION=1.1.0-rc.2 # this is the apptainer version, change as you need # Fetch the source wget https://github.com/apptainer/apptainer/releases/download/v${VERSION}/apptainer-${VERSION}.tar.gz # Build rpms from the source tar.gz @@ -231,7 +233,7 @@ <!-- markdownlint-disable MD013 --> ```sh -VERSION=1.1.0-rc.1 # this is the latest apptainer version, change as you need +VERSION=1.1.0-rc.2 # this is the latest apptainer version, change as you need ./mconfig make -C builddir rpm sudo rpm -ivh ~/rpmbuild/RPMS/x86_64/apptainer-$(echo $VERSION|tr - \~)*.x86_64.rpm diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/RELEASE_PROCEDURE.md new/apptainer-1.1.0-rc.2/RELEASE_PROCEDURE.md --- old/apptainer-1.1.0-rc.1/RELEASE_PROCEDURE.md 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/RELEASE_PROCEDURE.md 2022-08-17 13:47:34.000000000 +0200 @@ -76,6 +76,8 @@ - `git add apptainer_source` - [Admin Docs](https://apptainer.org/docs/admin/main/) can be edited [here](https://github.com/apptainer/apptainer-admindocs) + - Look in replacements.py in both the User Docs and Admin Docs for + any needed updates to the variable_replacements - If a new branch was created, add it to the docsVersion list in the [web page](https://github.com/apptainer/apptainer.org/blob/master/src/pages/docs.js) 1. Ensure the user and admin documentation has been deployed to the diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/cmd/internal/cli/action_flags.go new/apptainer-1.1.0-rc.2/cmd/internal/cli/action_flags.go --- old/apptainer-1.1.0-rc.1/cmd/internal/cli/action_flags.go 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/cmd/internal/cli/action_flags.go 2022-08-17 13:47:34.000000000 +0200 @@ -579,7 +579,7 @@ DefaultValue: false, Name: "userns", ShortHand: "u", - Usage: "run container in a new user namespace, allowing Apptainer to run completely unprivileged on recent kernels. This disables some features of Apptainer, for example it only works with sandbox images.", + Usage: "run container in a new user namespace", EnvKeys: []string{"USERNS", "UNSHARE_USERNS"}, } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/cmd/internal/cli/actions_linux.go new/apptainer-1.1.0-rc.2/cmd/internal/cli/actions_linux.go --- old/apptainer-1.1.0-rc.1/cmd/internal/cli/actions_linux.go 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/cmd/internal/cli/actions_linux.go 2022-08-17 13:47:34.000000000 +0200 @@ -906,6 +906,11 @@ sylog.Verbosef("'always use rocm = yes' found in apptainer.conf") } + if NvCCLI && !Nvidia { + sylog.Debugf("implying --nv from --nvccli") + Nvidia = true + } + if Nvidia && Rocm { sylog.Warningf("--nv and --rocm cannot be used together. Only --nv will be applied.") } @@ -959,9 +964,6 @@ } engineConfig.SetNvCCLIEnv(nvCCLIEnv) - if UserNamespace && !IsWritable { - return fmt.Errorf("nvidia-container-cli requires --writable with user namespace/fakeroot") - } if !IsWritable && !IsWritableTmpfs { sylog.Infof("Setting --writable-tmpfs (required by nvidia-container-cli)") IsWritableTmpfs = true diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/cmd/internal/cli/build.go new/apptainer-1.1.0-rc.2/cmd/internal/cli/build.go --- old/apptainer-1.1.0-rc.1/cmd/internal/cli/build.go 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/cmd/internal/cli/build.go 2022-08-17 13:47:34.000000000 +0200 @@ -28,24 +28,28 @@ ) var buildArgs struct { - sections []string - bindPaths []string - mounts []string - libraryURL string - keyServerURL string - webURL string - encrypt bool - fakeroot bool - fixPerms bool - isJSON bool - noCleanUp bool - noTest bool - sandbox bool - update bool - nvidia bool - nvccli bool - rocm bool - writableTmpfs bool // For test section only + sections []string + bindPaths []string + mounts []string + libraryURL string + keyServerURL string + webURL string + encrypt bool + fakeroot bool + fixPerms bool + isJSON bool + noCleanUp bool + noTest bool + sandbox bool + update bool + nvidia bool + nvccli bool + rocm bool + writableTmpfs bool // For test section only + userns bool // Enable user namespaces + ignoreSubuid bool // Ignore /etc/subuid entries (hidden) + ignoreFakerootCmd bool // Ignore fakeroot command (hidden) + ignoreUserns bool // Ignore user namespace(hidden) } // -s|--sandbox @@ -227,6 +231,49 @@ EnvKeys: []string{"WRITABLE_TMPFS"}, } +// --userns +var buildUsernsFlag = cmdline.Flag{ + ID: "buildUsernsFlag", + Value: &buildArgs.userns, + DefaultValue: false, + Name: "userns", + Usage: "build without using setuid even if available", + EnvKeys: []string{"USERNS"}, +} + +// --ignore-subuid +var buildIgnoreSubuidFlag = cmdline.Flag{ + ID: "buildIgnoreSubuidFlag", + Value: &buildArgs.ignoreSubuid, + DefaultValue: false, + Name: "ignore-subuid", + Usage: "ignore entries inside /etc/subuid", + EnvKeys: []string{"IGNORE_SUBUID"}, + Hidden: true, +} + +// --ignore-fakeroot-command +var buildIgnoreFakerootCommand = cmdline.Flag{ + ID: "buildIgnoreFakerootCommandFlag", + Value: &buildArgs.ignoreFakerootCmd, + DefaultValue: false, + Name: "ignore-fakeroot-command", + Usage: "ignore fakeroot command", + EnvKeys: []string{"IGNORE_FAKEROOT_COMMAND"}, + Hidden: true, +} + +// --ignore-userns +var buildIgnoreUsernsFlag = cmdline.Flag{ + ID: "buildIgnoreUsernsFlag", + Value: &buildArgs.ignoreUserns, + DefaultValue: false, + Name: "ignore-userns", + Usage: "ignore user namespaces", + EnvKeys: []string{"IGNORE_USERNS"}, + Hidden: true, +} + func init() { addCmdInit(func(cmdManager *cmdline.CommandManager) { cmdManager.RegisterCmd(buildCmd) @@ -259,6 +306,11 @@ cmdManager.RegisterFlagForCmd(&buildBindFlag, buildCmd) cmdManager.RegisterFlagForCmd(&buildMountFlag, buildCmd) cmdManager.RegisterFlagForCmd(&buildWritableTmpfsFlag, buildCmd) + + cmdManager.RegisterFlagForCmd(&buildUsernsFlag, buildCmd) + cmdManager.RegisterFlagForCmd(&buildIgnoreSubuidFlag, buildCmd) + cmdManager.RegisterFlagForCmd(&buildIgnoreFakerootCommand, buildCmd) + cmdManager.RegisterFlagForCmd(&buildIgnoreUsernsFlag, buildCmd) }) } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/cmd/internal/cli/build_linux.go new/apptainer-1.1.0-rc.2/cmd/internal/cli/build_linux.go --- old/apptainer-1.1.0-rc.1/cmd/internal/cli/build_linux.go 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/cmd/internal/cli/build_linux.go 2022-08-17 13:47:34.000000000 +0200 @@ -12,6 +12,7 @@ import ( "context" + "errors" "fmt" "os" osExec "os/exec" @@ -33,12 +34,13 @@ "github.com/apptainer/apptainer/pkg/runtime/engine/config" "github.com/apptainer/apptainer/pkg/sylog" "github.com/apptainer/apptainer/pkg/util/cryptkey" + "github.com/apptainer/apptainer/pkg/util/namespaces" keyClient "github.com/apptainer/container-key-client/client" "github.com/spf13/cobra" ) func fakerootExec(isDeffile bool) { - useSuid := starter.IsSuidInstall() + useSuid := starter.IsSuidInstall() && !buildArgs.userns // First remove fakeroot option from args and environment if present short := "-" + buildFakerootFlag.ShortHand @@ -73,10 +75,14 @@ var err error uid := uint32(os.Getuid()) - if uid != 0 && !fakeroot.IsUIDMapped(uid) { + if uid != 0 && (!fakeroot.IsUIDMapped(uid) || buildArgs.ignoreSubuid) { sylog.Infof("User not listed in %v, trying root-mapped namespace", fakeroot.SubUIDFile) os.Setenv("_APPTAINER_FAKEFAKEROOT", "1") - err = fakeroot.UnshareRootMapped(args) + if buildArgs.ignoreUserns { + err = errors.New("could not start root-mapped namesapce because of --ignore-userns is set") + } else { + err = fakeroot.UnshareRootMapped(args) + } if err == nil { // All the work has been done by the child process os.Exit(0) @@ -138,7 +144,11 @@ os.Unsetenv("_APPTAINER_FAKEFAKEROOT") buildArgs.fakeroot = false var err error - fakerootPath, err = fakeroot.FindFake() + if buildArgs.ignoreFakerootCmd { + err = errors.New("fakeroot command is ignored because of --ignore-fakeroot-command") + } else { + fakerootPath, err = fakeroot.FindFake() + } if err != nil { sylog.Infof("fakeroot command not found") if os.Getuid() != 0 { @@ -193,7 +203,7 @@ func runBuildLocal(ctx context.Context, cmd *cobra.Command, dst, spec string, fakerootPath string) { var keyInfo *cryptkey.KeyInfo if buildArgs.encrypt || promptForPassphrase || cmd.Flags().Lookup("pem-path").Changed { - if os.Getuid() != 0 { + if namespaces.IsUnprivileged() { sylog.Fatalf("You must be root to build an encrypted container") } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/dist/debian/control new/apptainer-1.1.0-rc.2/dist/debian/control --- old/apptainer-1.1.0-rc.1/dist/debian/control 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/dist/debian/control 2022-08-17 13:47:34.000000000 +0200 @@ -22,7 +22,15 @@ Package: apptainer Architecture: any -Depends: ${misc:Depends}, ${shlibs:Depends}, squashfs-tools, squashfuse, fuse2fs, fuse-overlayfs, fakeroot +Depends: + ${misc:Depends}, + ${shlibs:Depends}, + uidmap, + squashfs-tools, + squashfuse, + fuse2fs, + fuse-overlayfs, + fakeroot Conflicts: singularity-container Description: container platform focused on supporting "Mobility of Compute" Mobility of Compute encapsulates the development to compute model diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/dist/rpm/apptainer.spec.in new/apptainer-1.1.0-rc.2/dist/rpm/apptainer.spec.in --- old/apptainer-1.1.0-rc.1/dist/rpm/apptainer.spec.in 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/dist/rpm/apptainer.spec.in 2022-08-17 13:47:34.000000000 +0200 @@ -73,8 +73,11 @@ BuildRequires: cryptsetup Requires: squashfuse Requires: fakeroot -%if 0%{?el7} Requires: fuse-overlayfs +%if 0%{?el7} +Requires: fuse2fs +%else +Requires: e2fsprogs %endif %description diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/e2e/actions/actions.go new/apptainer-1.1.0-rc.2/e2e/actions/actions.go --- old/apptainer-1.1.0-rc.1/e2e/actions/actions.go 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/e2e/actions/actions.go 2022-08-17 13:47:34.000000000 +0200 @@ -2448,6 +2448,7 @@ "issue 5631": c.issue5631, // https://github.com/apptainer/singularity/issues/5631 "issue 5690": c.issue5690, // https://github.com/apptainer/singularity/issues/5690 "issue 6165": c.issue6165, // https://github.com/apptainer/singularity/issues/6165 + "issue 619": c.issue619, // https://github.com/apptainer/apptainer/issues/619 "network": c.actionNetwork, // test basic networking "binds": c.actionBinds, // test various binds with --bind and --mount "exit and signals": c.exitSignals, // test exit and signals propagation diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/e2e/actions/regressions.go new/apptainer-1.1.0-rc.2/e2e/actions/regressions.go --- old/apptainer-1.1.0-rc.1/e2e/actions/regressions.go 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/e2e/actions/regressions.go 2022-08-17 13:47:34.000000000 +0200 @@ -701,3 +701,20 @@ ), ) } + +// Check that a nested bind separated by more than one level works with +// underlay when the lower directory does not existing in the image but +// the image has another directory that does not exist on the host +func (c actionTests) issue619(t *testing.T) { + e2e.EnsureImage(t, c.env) + + // use of user namespace to force runtime to use underlay + c.env.RunApptainer( + t, + e2e.WithProfile(e2e.UserNamespaceProfile), + e2e.WithDir(c.env.TestDir), + e2e.WithCommand("exec"), + e2e.WithArgs("--bind", "/var,/var/lib/apt", c.env.ImagePath, "true"), + e2e.ExpectExit(0), + ) +} diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/e2e/gpu/gpu.go new/apptainer-1.1.0-rc.2/e2e/gpu/gpu.go --- old/apptainer-1.1.0-rc.1/e2e/gpu/gpu.go 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/e2e/gpu/gpu.go 2022-08-17 13:47:34.000000000 +0200 @@ -140,21 +140,21 @@ { name: "User", profile: e2e.RootProfile, - args: []string{"--nv", "--nvccli", imagePath, "nvidia-smi"}, + args: []string{"--nvccli", imagePath, "nvidia-smi"}, expectExit: 0, }, { // With --contain, we should only see NVIDIA_VISIBLE_DEVICES configured GPUs name: "UserContainNoDevices", profile: e2e.RootProfile, - args: []string{"--contain", "--nv", "--nvccli", imagePath, "nvidia-smi"}, + args: []string{"--contain", "--nvccli", imagePath, "nvidia-smi"}, expectMatch: e2e.ExpectOutput(e2e.ContainMatch, "No devices were found"), expectExit: 6, }, { name: "UserContainAllDevices", profile: e2e.RootProfile, - args: []string{"--contain", "--nv", "--nvccli", imagePath, "nvidia-smi"}, + args: []string{"--contain", "--nvccli", imagePath, "nvidia-smi"}, env: []string{"NVIDIA_VISIBLE_DEVICES=all"}, expectExit: 0, }, @@ -162,7 +162,7 @@ // If we only request compute, not utility, then nvidia-smi should not be present name: "UserNoUtility", profile: e2e.RootProfile, - args: []string{"--nv", "--nvccli", imagePath, "nvidia-smi"}, + args: []string{"--nvccli", imagePath, "nvidia-smi"}, env: []string{"NVIDIA_DRIVER_CAPABILITIES=compute"}, expectMatch: e2e.ExpectError(e2e.ContainMatch, "\"nvidia-smi\": executable file not found in $PATH"), expectExit: 255, @@ -171,7 +171,7 @@ // Require CUDA version >8 should be fine! name: "UserValidRequire", profile: e2e.RootProfile, - args: []string{"--nv", "--nvccli", imagePath, "nvidia-smi"}, + args: []string{"--nvccli", imagePath, "nvidia-smi"}, env: []string{"NVIDIA_REQUIRE_CUDA=cuda>8"}, expectExit: 0, }, @@ -179,7 +179,7 @@ // Require CUDA version >999 should not be satisfied name: "UserInvalidRequire", profile: e2e.RootProfile, - args: []string{"--nv", "--nvccli", imagePath, "nvidia-smi"}, + args: []string{"--nvccli", imagePath, "nvidia-smi"}, env: []string{"NVIDIA_REQUIRE_CUDA=cuda>999"}, expectMatch: e2e.ExpectError(e2e.ContainMatch, "requirement error: unsatisfied condition: cuda>99"), expectExit: 255, @@ -187,7 +187,17 @@ { name: "UserNamespace", profile: e2e.UserNamespaceProfile, - args: []string{"--nv", "--nvccli", "--writable", imagePath, "nvidia-smi"}, + args: []string{"--nvccli", imagePath, "nvidia-smi"}, + }, + { + name: "UserNamespaceWritable", + profile: e2e.UserNamespaceProfile, + args: []string{"--nvccli", "--writable", imagePath, "nvidia-smi"}, + }, + { + name: "Fakeroot", + profile: e2e.FakerootProfile, + args: []string{"--nvccli", imagePath, "nvidia-smi"}, }, } @@ -409,7 +419,7 @@ args := []string{} if tt.setNvFlag { - args = append(args, "--nv", "--nvccli") + args = append(args, "--nvccli") } args = append(args, "-F", "--sandbox", sandboxImage, defFile) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/e2e/imgbuild/imgbuild.go new/apptainer-1.1.0-rc.2/e2e/imgbuild/imgbuild.go --- old/apptainer-1.1.0-rc.1/e2e/imgbuild/imgbuild.go 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/e2e/imgbuild/imgbuild.go 2022-08-17 13:47:34.000000000 +0200 @@ -1530,6 +1530,65 @@ ) } +func (c *imgBuildTests) testContainerBuildUnderFakerootModes(t *testing.T) { + tmpDir, cleanup := e2e.MakeTempDir(t, c.env.TestDir, "test-container-build-under-fakeroot-modes-", "") + defer cleanup(t) + + // running under the mode 1, 1a (--with-suid) (https://apptainer.org/docs/user/main/fakeroot.html) + c.env.RunApptainer( + t, + e2e.WithProfile(e2e.UserProfile), + e2e.WithCommand("build"), + e2e.WithArgs("--force", fmt.Sprintf("%s/openssh-mode1a.sif", tmpDir), "testdata/unprivileged_build.def"), + e2e.ExpectExit(0), + ) + + // running under the mode 1, 1b (--without-suid) (https://apptainer.org/docs/user/main/fakeroot.html) + c.env.RunApptainer( + t, + e2e.WithProfile(e2e.UserProfile), + e2e.WithCommand("build"), + e2e.WithArgs("--force", "--userns", fmt.Sprintf("%s/openssh-mode1b.sif", tmpDir), "testdata/unprivileged_build.def"), + e2e.ExpectExit(0), + ) + + // running under the mode 2(https://apptainer.org/docs/user/main/fakeroot.html) + c.env.RunApptainer( + t, + e2e.WithProfile(e2e.UserNamespaceProfile), + e2e.WithCommand("build"), + e2e.WithArgs("--force", "--userns", "--ignore-subuid", "--ignore-fakeroot-command", fmt.Sprintf("%s/openssh-mode2a.sif", tmpDir), "testdata/unprivileged_build.def"), + e2e.ExpectExit(255), // because chown will fail + ) + + // running under the mode 2(https://apptainer.org/docs/user/main/fakeroot.html) + c.env.RunApptainer( + t, + e2e.WithProfile(e2e.UserNamespaceProfile), + e2e.WithCommand("build"), + e2e.WithArgs("--force", "--userns", "--ignore-subuid", "--ignore-fakeroot-command", fmt.Sprintf("%s/openssh-mode2b.sif", tmpDir), "testdata/unprivileged_build_2.def"), + e2e.ExpectExit(0), // install epel-release under mode 2 should succeed + ) + + // running under the mode 3(https://apptainer.org/docs/user/main/fakeroot.html) + c.env.RunApptainer( + t, + e2e.WithProfile(e2e.FakerootProfile), + e2e.WithCommand("build"), + e2e.WithArgs("--force", "--userns", "--ignore-subuid", fmt.Sprintf("%s/openssh-mode3.sif", tmpDir), "testdata/unprivileged_build.def"), + e2e.ExpectExit(0), + ) + + // running under the mode 4(https://apptainer.org/docs/user/main/fakeroot.html) + c.env.RunApptainer( + t, + e2e.WithProfile(e2e.FakerootProfile), + e2e.WithCommand("build"), + e2e.WithArgs("--force", "--ignore-userns", "--ignore-subuid", fmt.Sprintf("%s/openssh-mode4.sif", tmpDir), "testdata/unprivileged_build.def"), + e2e.ExpectExit(0), + ) +} + // E2ETests is the main func to trigger the test suite func E2ETests(env e2e.TestEnv) testhelper.Tests { c := imgBuildTests{ @@ -1537,36 +1596,37 @@ } return testhelper.Tests{ - "bad path": c.badPath, // try to build from a non existent path - "build encrypt with PEM file": c.buildEncryptPemFile, // build encrypted images with certificate - "build encrypted with passphrase": c.buildEncryptPassphrase, // build encrypted images with passphrase - "definition": c.buildDefinition, // builds from definition template - "from local image": c.buildLocalImage, // build and image from an existing image - "from": c.buildFrom, // builds from definition file and URI - "multistage": c.buildMultiStageDefinition, // multistage build from definition templates - "non-root build": c.nonRootBuild, // build sifs from non-root - "build and update sandbox": c.buildUpdateSandbox, // build/update sandbox - "fingerprint check": c.buildWithFingerprint, // definition file includes fingerprint check - "build with bind mount": c.buildBindMount, // build image with bind mount - "library host": c.buildLibraryHost, // build image with hostname in library URI - "test with writable tmpfs": c.testWritableTmpfs, // build image, using writable tmpfs in the test step - "test build system environment": c.testBuildEnvironmentVariables, // build image with build system environment variables set in definition - "issue 3848": c.issue3848, // https://github.com/apptainer/singularity/issues/3848 - "issue 4203": c.issue4203, // https://github.com/apptainer/singularity/issues/4203 - "issue 4407": c.issue4407, // https://github.com/apptainer/singularity/issues/4407 - "issue 4524": c.issue4524, // https://github.com/apptainer/singularity/issues/4524 - "issue 4583": c.issue4583, // https://github.com/apptainer/singularity/issues/4583 - "issue 4820": c.issue4820, // https://github.com/apptainer/singularity/issues/4820 - "issue 4837": c.issue4837, // https://github.com/apptainer/singularity/issues/4837 - "issue 4943": c.issue4943, // https://github.com/apptainer/singularity/issues/4943 - "issue 4967": c.issue4967, // https://github.com/apptainer/singularity/issues/4967 - "issue 4969": c.issue4969, // https://github.com/apptainer/singularity/issues/4969 - "issue 5166": c.issue5166, // https://github.com/apptainer/singularity/issues/5166 - "issue 5172": c.issue5172, // https://github.com/apptainer/singularity/issues/5172 - "issue 5250": c.issue5250, // https://github.com/apptainer/singularity/issues/5250 - "issue 5315": c.issue5315, // https://github.com/apptainer/singularity/issues/5315 - "issue 5435": c.issue5435, // https://github.com/apptainer/singularity/issues/5435 - "issue 5668": c.issue5668, // https://github.com/apptainer/singularity/issues/5435 - "issue 5690": c.issue5690, // https://github.com/apptainer/singularity/issues/5690 + "bad path": c.badPath, // try to build from a non existent path + "build encrypt with PEM file": c.buildEncryptPemFile, // build encrypted images with certificate + "build encrypted with passphrase": c.buildEncryptPassphrase, // build encrypted images with passphrase + "definition": c.buildDefinition, // builds from definition template + "from local image": c.buildLocalImage, // build and image from an existing image + "from": c.buildFrom, // builds from definition file and URI + "multistage": c.buildMultiStageDefinition, // multistage build from definition templates + "non-root build": c.nonRootBuild, // build sifs from non-root + "build and update sandbox": c.buildUpdateSandbox, // build/update sandbox + "fingerprint check": c.buildWithFingerprint, // definition file includes fingerprint check + "build with bind mount": c.buildBindMount, // build image with bind mount + "library host": c.buildLibraryHost, // build image with hostname in library URI + "test with writable tmpfs": c.testWritableTmpfs, // build image, using writable tmpfs in the test step + "test build system environment": c.testBuildEnvironmentVariables, // build image with build system environment variables set in definition + "test build under fakeroot modes": c.testContainerBuildUnderFakerootModes, // build image under different fakeroot modes + "issue 3848": c.issue3848, // https://github.com/apptainer/singularity/issues/3848 + "issue 4203": c.issue4203, // https://github.com/apptainer/singularity/issues/4203 + "issue 4407": c.issue4407, // https://github.com/apptainer/singularity/issues/4407 + "issue 4524": c.issue4524, // https://github.com/apptainer/singularity/issues/4524 + "issue 4583": c.issue4583, // https://github.com/apptainer/singularity/issues/4583 + "issue 4820": c.issue4820, // https://github.com/apptainer/singularity/issues/4820 + "issue 4837": c.issue4837, // https://github.com/apptainer/singularity/issues/4837 + "issue 4943": c.issue4943, // https://github.com/apptainer/singularity/issues/4943 + "issue 4967": c.issue4967, // https://github.com/apptainer/singularity/issues/4967 + "issue 4969": c.issue4969, // https://github.com/apptainer/singularity/issues/4969 + "issue 5166": c.issue5166, // https://github.com/apptainer/singularity/issues/5166 + "issue 5172": c.issue5172, // https://github.com/apptainer/singularity/issues/5172 + "issue 5250": c.issue5250, // https://github.com/apptainer/singularity/issues/5250 + "issue 5315": c.issue5315, // https://github.com/apptainer/singularity/issues/5315 + "issue 5435": c.issue5435, // https://github.com/apptainer/singularity/issues/5435 + "issue 5668": c.issue5668, // https://github.com/apptainer/singularity/issues/5435 + "issue 5690": c.issue5690, // https://github.com/apptainer/singularity/issues/5690 } } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/e2e/testdata/unprivileged_build.def new/apptainer-1.1.0-rc.2/e2e/testdata/unprivileged_build.def --- old/apptainer-1.1.0-rc.1/e2e/testdata/unprivileged_build.def 1970-01-01 01:00:00.000000000 +0100 +++ new/apptainer-1.1.0-rc.2/e2e/testdata/unprivileged_build.def 2022-08-17 13:47:34.000000000 +0200 @@ -0,0 +1,5 @@ +BootStrap: docker +From: centos:centos7 + +%post + yum -y install openssh \ No newline at end of file diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/e2e/testdata/unprivileged_build_2.def new/apptainer-1.1.0-rc.2/e2e/testdata/unprivileged_build_2.def --- old/apptainer-1.1.0-rc.1/e2e/testdata/unprivileged_build_2.def 1970-01-01 01:00:00.000000000 +0100 +++ new/apptainer-1.1.0-rc.2/e2e/testdata/unprivileged_build_2.def 2022-08-17 13:47:34.000000000 +0200 @@ -0,0 +1,5 @@ +BootStrap: docker +From: centos:centos7 + +%post + yum -y install epel-release \ No newline at end of file diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/go.mod new/apptainer-1.1.0-rc.2/go.mod --- old/apptainer-1.1.0-rc.1/go.mod 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/go.mod 2022-08-17 13:47:34.000000000 +0200 @@ -13,7 +13,7 @@ github.com/blang/semver/v4 v4.0.0 github.com/buger/jsonparser v1.1.1 github.com/cenkalti/backoff/v4 v4.1.3 - github.com/containerd/containerd v1.6.6 + github.com/containerd/containerd v1.6.8 github.com/containernetworking/cni v1.1.2 github.com/containernetworking/plugins v1.1.1 github.com/containers/image/v5 v5.22.0 @@ -66,7 +66,7 @@ github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 // indirect github.com/BurntSushi/toml v1.2.0 // indirect github.com/Microsoft/go-winio v0.5.2 // indirect - github.com/Microsoft/hcsshim v0.9.3 // indirect + github.com/Microsoft/hcsshim v0.9.4 // indirect github.com/VividCortex/ewma v1.2.0 // indirect github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d // indirect github.com/alexflint/go-filemutex v1.1.0 // indirect diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/go.sum new/apptainer-1.1.0-rc.2/go.sum --- old/apptainer-1.1.0-rc.1/go.sum 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/go.sum 2022-08-17 13:47:34.000000000 +0200 @@ -123,8 +123,9 @@ github.com/Microsoft/hcsshim v0.8.21/go.mod h1:+w2gRZ5ReXQhFOrvSQeNfhrYB/dg3oDwTOcER2fw4I4= github.com/Microsoft/hcsshim v0.8.23/go.mod h1:4zegtUJth7lAvFyc6cH2gGQ5B3OFQim01nnU2M8jKDg= github.com/Microsoft/hcsshim v0.9.2/go.mod h1:7pLA8lDk46WKDWlVsENo92gC0XFa8rbKfyFRBqxEbCc= -github.com/Microsoft/hcsshim v0.9.3 h1:k371PzBuRrz2b+ebGuI2nVgVhgsVX60jMfSw80NECxo= github.com/Microsoft/hcsshim v0.9.3/go.mod h1:7pLA8lDk46WKDWlVsENo92gC0XFa8rbKfyFRBqxEbCc= +github.com/Microsoft/hcsshim v0.9.4 h1:mnUj0ivWy6UzbB1uLFqKR6F+ZyiDc7j4iGgHTpO+5+I= +github.com/Microsoft/hcsshim v0.9.4/go.mod h1:7pLA8lDk46WKDWlVsENo92gC0XFa8rbKfyFRBqxEbCc= github.com/Microsoft/hcsshim/test v0.0.0-20201218223536-d3e5debf77da/go.mod h1:5hlzMzRKMLyo42nCZ9oml8AdTlq/0cvIaBv6tK1RehU= github.com/Microsoft/hcsshim/test v0.0.0-20210227013316-43a75bb4edd3/go.mod h1:mw7qgWloBUl75W/gVH3cQszUg1+gUITj7D6NY7ywVnY= github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46/go.mod h1:3wb06e3pkSAbeQ52E9H9iFoQsEEwGN64994WTCIhntQ= @@ -312,8 +313,9 @@ github.com/containerd/containerd v1.5.7/go.mod h1:gyvv6+ugqY25TiXxcZC3L5yOeYgEw0QMhscqVp1AR9c= github.com/containerd/containerd v1.5.8/go.mod h1:YdFSv5bTFLpG2HIYmfqDpSYYTDX+mc5qtSuYx1YUb/s= github.com/containerd/containerd v1.6.1/go.mod h1:1nJz5xCZPusx6jJU8Frfct988y0NpumIq9ODB0kLtoE= -github.com/containerd/containerd v1.6.6 h1:xJNPhbrmz8xAMDNoVjHy9YHtWwEQNS+CDkcIRh7t8Y0= github.com/containerd/containerd v1.6.6/go.mod h1:ZoP1geJldzCVY3Tonoz7b1IXk8rIX0Nltt5QE4OMNk0= +github.com/containerd/containerd v1.6.8 h1:h4dOFDwzHmqFEP754PgfgTeVXFnLiRc6kiqC7tplDJs= +github.com/containerd/containerd v1.6.8/go.mod h1:By6p5KqPK0/7/CgO/A6t/Gz+CUYUu2zf1hUaaymVXB0= github.com/containerd/continuity v0.0.0-20190426062206-aaeac12a7ffc/go.mod h1:GL3xCUCBDV3CZiTSEKksMWbLE66hEyuu9qyDOOqM47Y= github.com/containerd/continuity v0.0.0-20190815185530-f2a389ac0a02/go.mod h1:GL3xCUCBDV3CZiTSEKksMWbLE66hEyuu9qyDOOqM47Y= github.com/containerd/continuity v0.0.0-20191127005431-f65d91d395eb/go.mod h1:GL3xCUCBDV3CZiTSEKksMWbLE66hEyuu9qyDOOqM47Y= diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/internal/pkg/build/files/files.go new/apptainer-1.1.0-rc.2/internal/pkg/build/files/files.go --- old/apptainer-1.1.0-rc.1/internal/pkg/build/files/files.go 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/internal/pkg/build/files/files.go 2022-08-17 13:47:34.000000000 +0200 @@ -11,34 +11,47 @@ import ( "bytes" + "context" "fmt" - "os/exec" "path/filepath" "strings" securejoin "github.com/cyphar/filepath-securejoin" + "mvdan.cc/sh/v3/interp" + "mvdan.cc/sh/v3/syntax" ) -const filenameExpansionScript = `for n in %[1]s ; do - printf "$n\0" +const filenameExpansionScript = ` +for n in %[1]s ; do + echo -n "$n\0" done ` func expandPath(path string) ([]string, error) { var output, stderr bytes.Buffer - // Escape spaces for glob pattern path = strings.Replace(path, " ", "\\ ", -1) - cmd := exec.Command("/bin/sh", "-c", fmt.Sprintf(filenameExpansionScript, path)) - cmd.Stdout = &output - cmd.Stderr = &stderr - if err := cmd.Run(); err != nil { - return nil, fmt.Errorf("%s: %s", err, stderr.String()) + cmdline := fmt.Sprintf(filenameExpansionScript, path) + parser, err := syntax.NewParser().Parse(strings.NewReader(cmdline), "") + if err != nil { + return nil, err + } + + runner, err := interp.New( + interp.StdIO(nil, &output, &stderr), + ) + if err != nil { + return nil, err + } + + err = runner.Run(context.TODO(), parser) + if err != nil { + return nil, err } // parse expanded output and ignore empty strings from consecutive null bytes var paths []string - for _, s := range strings.Split(output.String(), "\x00") { + for _, s := range strings.Split(output.String(), "\\0") { if s == "" { continue } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/internal/pkg/build/files/files_test.go new/apptainer-1.1.0-rc.2/internal/pkg/build/files/files_test.go --- old/apptainer-1.1.0-rc.1/internal/pkg/build/files/files_test.go 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/internal/pkg/build/files/files_test.go 2022-08-17 13:47:34.000000000 +0200 @@ -185,7 +185,7 @@ { name: "hiddenFileWildcards", path: "dirL1/.*", - correct: []string{"dirL1/.", "dirL1/..", "dirL1/.dirL2", "dirL1/.file"}, + correct: []string{"dirL1/.dirL2", "dirL1/.file"}, }, { name: "hiddenDirWildcards", diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/internal/pkg/build/metadata.go new/apptainer-1.1.0-rc.2/internal/pkg/build/metadata.go --- old/apptainer-1.1.0-rc.1/internal/pkg/build/metadata.go 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/internal/pkg/build/metadata.go 2022-08-17 13:47:34.000000000 +0200 @@ -2,7 +2,7 @@ // Apptainer a Series of LF Projects LLC. // For website terms of use, trademark policy, privacy policy and other // project policies see https://lfprojects.org/policies -// Copyright (c) 2018-2020, Sylabs Inc. All rights reserved. +// Copyright (c) 2018-2022, Sylabs Inc. All rights reserved. // This software is licensed under a 3-clause BSD license. Please consult the // LICENSE.md file distributed with the sources of this project regarding your // rights to use or distribute this software. @@ -174,31 +174,29 @@ } func insertDefinition(b *types.Bundle) error { - // if update, check for existing definition and move it to bootstrap history - if b.Opts.Update { - if _, err := os.Stat(filepath.Join(b.RootfsPath, "/.singularity.d/Singularity")); err == nil { - // make bootstrap_history directory if it doesn't exist - if _, err := os.Stat(filepath.Join(b.RootfsPath, "/.singularity.d/bootstrap_history")); err != nil { - err = os.Mkdir(filepath.Join(b.RootfsPath, "/.singularity.d/bootstrap_history"), 0o755) - if err != nil { - return err - } - } - - // look at number of files in bootstrap_history to give correct file name - files, err := ioutil.ReadDir(filepath.Join(b.RootfsPath, "/.singularity.d/bootstrap_history")) + // Check for existing definition and move it to bootstrap history + if _, err := os.Stat(filepath.Join(b.RootfsPath, "/.singularity.d/Singularity")); err == nil { + // make bootstrap_history directory if it doesn't exist + if _, err := os.Stat(filepath.Join(b.RootfsPath, "/.singularity.d/bootstrap_history")); err != nil { + err = os.Mkdir(filepath.Join(b.RootfsPath, "/.singularity.d/bootstrap_history"), 0o755) if err != nil { return err } + } - // name is "Apptainer" concatenated with an index based on number of other files in bootstrap_history - length := strconv.Itoa(len(files)) - histName := "Apptainer" + length - // move old definition into bootstrap_history - err = os.Rename(filepath.Join(b.RootfsPath, "/.singularity.d/Singularity"), filepath.Join(b.RootfsPath, "/.singularity.d/bootstrap_history", histName)) - if err != nil { - return err - } + // look at number of files in bootstrap_history to give correct file name + files, err := ioutil.ReadDir(filepath.Join(b.RootfsPath, "/.singularity.d/bootstrap_history")) + if err != nil { + return err + } + + // name is "Apptainer" concatenated with an index based on number of other files in bootstrap_history + len := strconv.Itoa(len(files)) + histName := "Apptainer" + len + // move old definition into bootstrap_history + err = os.Rename(filepath.Join(b.RootfsPath, "/.singularity.d/Singularity"), filepath.Join(b.RootfsPath, "/.singularity.d/bootstrap_history", histName)) + if err != nil { + return err } } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/internal/pkg/fakeroot/fakefake.go new/apptainer-1.1.0-rc.2/internal/pkg/fakeroot/fakefake.go --- old/apptainer-1.1.0-rc.1/internal/pkg/fakeroot/fakefake.go 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/internal/pkg/fakeroot/fakefake.go 2022-08-17 13:47:34.000000000 +0200 @@ -21,6 +21,7 @@ "syscall" "github.com/apptainer/apptainer/internal/pkg/util/bin" + "github.com/apptainer/apptainer/internal/pkg/util/env" "github.com/apptainer/apptainer/pkg/sylog" ) @@ -94,14 +95,21 @@ // Start by examining the environment fakeroot creates cmd := osExec.Command(fakerootPath, "env") - env := os.Environ() - for idx := range env { - if strings.HasPrefix(env[idx], "LD_LIBRARY_PATH=") { + environ := os.Environ() + hasPath := false + for idx := range environ { + if strings.HasPrefix(environ[idx], "LD_LIBRARY_PATH=") { // Remove any incoming LD_LIBRARY_PATH - env[idx] = "LD_LIBRARY_PREFIX=" + environ[idx] = "LD_LIBRARY_PREFIX=" + } else if strings.HasPrefix(environ[idx], "PATH=") && + environ[idx] != "PATH=" { + hasPath = true } } - cmd.Env = env + if !hasPath { + environ = append(environ, "PATH="+env.DefaultPath) + } + cmd.Env = environ stdout, err := cmd.StdoutPipe() if err != nil { return binds, fmt.Errorf("error make fakeroot stdout pipe: %v", err) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/internal/pkg/image/driver/imagedriver.go new/apptainer-1.1.0-rc.2/internal/pkg/image/driver/imagedriver.go --- old/apptainer-1.1.0-rc.1/internal/pkg/image/driver/imagedriver.go 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/internal/pkg/image/driver/imagedriver.go 2022-08-17 13:47:34.000000000 +0200 @@ -11,6 +11,7 @@ import ( "bytes" "fmt" + "io/ioutil" "os" "os/exec" "path" @@ -103,6 +104,7 @@ return features } +//nolint:maintidx func (d *fuseappsDriver) Mount(params *image.MountParams, mfunc image.MountFunc) error { var f *fuseappsFeature var cmd *exec.Cmd @@ -114,7 +116,10 @@ case "squashfs": f = &d.squashFeature - optsStr := "offset=" + strconv.FormatUint(params.Offset, 10) + optsStr := fmt.Sprintf("uid=%v,gid=%v", os.Getuid(), os.Getgid()) + if params.Offset > 0 { + optsStr += ",offset=" + strconv.FormatUint(params.Offset, 10) + } srcPath := params.Source if path.Dir(params.Source) == "/proc/self/fd" { // this will be passed as the first ExtraFile below, always fd 3 @@ -250,6 +255,32 @@ sylog.Infof("%v", msg) } sylog.Debugf("%v mounted in %v", params.Target, totTime) + if params.Filesystem == "overlay" && os.Getuid() == 0 { + // Look for unexpectedly readonly overlay + hasUpper := false + for _, opt := range params.FSOptions { + if strings.HasPrefix(opt, "upperdir=") { + hasUpper = true + } + } + if !hasUpper { + // No upperdir means readonly expected + return nil + } + // Using unix.Access is not sufficient here + // so have to attempt to create a file + binpath := params.Target + "/usr/bin" + tmpfile, err := ioutil.TempFile(binpath, ".tmp*") + if err != nil { + sylog.Debugf("%v not writable: %v", binpath, err) + sylog.Infof("/usr/bin not writable in container") + sylog.Infof("Consider using a different overlay upper layer filesystem type") + } else { + sylog.Debugf("successfully created %v", tmpfile.Name()) + tmpfile.Close() + os.Remove(tmpfile.Name()) + } + } return nil } } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/internal/pkg/runtime/engine/apptainer/container_linux.go new/apptainer-1.1.0-rc.2/internal/pkg/runtime/engine/apptainer/container_linux.go --- old/apptainer-1.1.0-rc.1/internal/pkg/runtime/engine/apptainer/container_linux.go 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/internal/pkg/runtime/engine/apptainer/container_linux.go 2022-08-17 13:47:34.000000000 +0200 @@ -1043,29 +1043,31 @@ } system.Points.AddRemount(mount.PreLayerTag, dst, flags) - if !img.Writable { - // check if the sandbox directory is located on a compatible - // filesystem usable overlay lower directory - if err := fsoverlay.CheckLower(img.Path, 0); err != nil { - return err + if !overlayImageDriver { + // When no overlay image driver available, + // make sure filesystems type are compatible + // with kernel overlayfs + if !img.Writable { + // check if the sandbox directory is located on a compatible + // filesystem usable overlay lower directory + if err := fsoverlay.CheckLower(img.Path); err != nil { + return err + } + } else { + // check if the sandbox directory is located on a compatible + // filesystem usable with overlay upper directory + if err := fsoverlay.CheckUpper(img.Path); err != nil { + return err + } } + } + + if !img.Writable { if fs.IsDir(filepath.Join(img.Path, "upper")) { ov.AddLowerDir(filepath.Join(dst, "upper")) } else { ov.AddLowerDir(dst) } - } else { - // check if the sandbox directory is located on a compatible - // filesystem usable with overlay upper directory - allowType := int64(0) - if overlayImageDriver { - // The imageDriver is likely to able to handle - // a fuse upper even though the kernel can't - allowType = fsoverlay.Fuse - } - if err := fsoverlay.CheckUpper(img.Path, allowType); err != nil { - return err - } } default: return fmt.Errorf("%s: overlay image with unknown format", img.Path) @@ -1642,12 +1644,12 @@ pw, err := user.CurrentOriginal() if err == nil { source = pw.Dir - if os.Getuid() == 0 { + if c.engine.EngineConfig.GetFakeroot() || os.Getuid() == 0 { // Mount user home directory onto /root for // any root-mapped namespace dest = "/root" } else { - dest = pw.Dir + dest = source } } else if os.Getuid() == 0 { // The user could not be found @@ -2370,7 +2372,9 @@ allowedNetNetwork = slice.ContainsString(c.engine.EngineConfig.File.AllowNetNetworks, n) // If any one requested network is not allowed, disallow the whole config if !allowedNetNetwork { - sylog.Errorf("Network %s is not permitted for unprivileged users.", n) + if !fakeroot { + sylog.Errorf("Network %s is not permitted for unprivileged users.", n) + } break } } @@ -2379,7 +2383,7 @@ } if (c.userNS || euid != 0) && !fakeroot && !allowedNetUnpriv { - return nil, fmt.Errorf("network requires root or --fakeroot, non-root users can only use --network=%s unless permitted by the administrator", noneNet) + return nil, fmt.Errorf("network requires root or a suid installation with /etc/subuid --fakeroot; non-root users can only use --network=%s unless permitted by the administrator", noneNet) } // we hold a reference to container network namespace diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/internal/pkg/runtime/engine/apptainer/prepare_linux.go new/apptainer-1.1.0-rc.2/internal/pkg/runtime/engine/apptainer/prepare_linux.go --- old/apptainer-1.1.0-rc.1/internal/pkg/runtime/engine/apptainer/prepare_linux.go 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/internal/pkg/runtime/engine/apptainer/prepare_linux.go 2022-08-17 13:47:34.000000000 +0200 @@ -138,6 +138,10 @@ e.EngineConfig.OciConfig.SetProcessNoNewPrivileges(true) } + userNS := !starterConfig.GetIsSUID() || e.EngineConfig.GetFakeroot() + driver.InitImageDrivers(true, userNS, e.EngineConfig.File, 0) + imageDriver = image.GetDriver(e.EngineConfig.File.ImageDriver) + if e.EngineConfig.GetInstanceJoin() { if err := e.prepareInstanceJoinConfig(starterConfig); err != nil { return err @@ -165,9 +169,6 @@ // determine if engine need to propagate signals across processes e.checkSignalPropagation() - userNS := !starterConfig.GetIsSUID() || e.EngineConfig.GetFakeroot() - driver.InitImageDrivers(false, userNS, e.EngineConfig.File, 0) - // We must call this here because at this point we haven't // spawned the master process nor the RPC server. The assumption // is that this function runs in stage 1 and that even if it's a @@ -560,7 +561,10 @@ } if e.EngineConfig.GetFakeroot() { - if !starterConfig.GetIsSUID() { + uid := uint32(os.Getuid()) + gid := uint32(os.Getgid()) + + if !starterConfig.GetIsSUID() && fakerootutil.IsUIDMapped(uid) { // no SUID workflow, check if newuidmap/newgidmap are present sylog.Verbosef("Fakeroot requested with unprivileged workflow, fallback to newuidmap/newgidmap") sylog.Debugf("Search for newuidmap binary") @@ -573,9 +577,6 @@ } } - uid := uint32(os.Getuid()) - gid := uint32(os.Getgid()) - getIDRange := fakerootutil.GetIDRange callbackType := (fakerootcallback.UserMapping)(nil) @@ -1170,8 +1171,9 @@ // C starter code will position current working directory starterConfig.SetWorkingDirectoryFd(int(img.Fd)) - if e.EngineConfig.GetSessionLayer() == apptainerConfig.OverlayLayer { - if err := overlay.CheckLower(img.Path, 0); overlay.IsIncompatible(err) { + if e.EngineConfig.GetSessionLayer() == apptainerConfig.OverlayLayer && + (imageDriver == nil || imageDriver.Features()&image.OverlayFeature == 0) { + if err := overlay.CheckLower(img.Path); overlay.IsIncompatible(err) { layer := apptainerConfig.UnderlayLayer if !e.EngineConfig.File.EnableUnderlay { sylog.Warningf("Could not fallback to underlay, disabled by configuration ('enable underlay = no')") diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/internal/pkg/runtime/engine/config/starter/starter_linux.go new/apptainer-1.1.0-rc.2/internal/pkg/runtime/engine/config/starter/starter_linux.go --- old/apptainer-1.1.0-rc.1/internal/pkg/runtime/engine/config/starter/starter_linux.go 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/internal/pkg/runtime/engine/config/starter/starter_linux.go 2022-08-17 13:47:34.000000000 +0200 @@ -272,7 +272,7 @@ path, err := bin.FindBin(command) if err != nil { return fmt.Errorf( - "%s was not found in PATH (%s), required with fakeroot and unprivileged installation", + "%s was not found in PATH (%s), required with fakeroot and unprivileged installation when user is in /etc/subuid", command, env.DefaultPath, ) } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/internal/pkg/util/fs/layout/layer/underlay/underlay_linux.go new/apptainer-1.1.0-rc.2/internal/pkg/util/fs/layout/layer/underlay/underlay_linux.go --- old/apptainer-1.1.0-rc.1/internal/pkg/util/fs/layout/layer/underlay/underlay_linux.go 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/internal/pkg/util/fs/layout/layer/underlay/underlay_linux.go 2022-08-17 13:47:34.000000000 +0200 @@ -126,6 +126,8 @@ return createdPath[i].len < createdPath[j].len }) + // Duplicate the parent directories of the new path, unless + // it has already been done by another bind for _, pl := range createdPath { splitted := strings.Split(filepath.Dir(pl.path), string(os.PathSeparator)) l := len(splitted) @@ -142,7 +144,7 @@ // if the directory is overridden by a bind mount we won't // need to duplicate the container image directory if _, ok := destinations[p]; ok { - continue + break } // directory not overridden, duplicate it if err := u.duplicateDir(p, system, pl.path); err != nil { diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/internal/pkg/util/fs/overlay/overlay_linux.go new/apptainer-1.1.0-rc.2/internal/pkg/util/fs/overlay/overlay_linux.go --- old/apptainer-1.1.0-rc.1/internal/pkg/util/fs/overlay/overlay_linux.go 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/internal/pkg/util/fs/overlay/overlay_linux.go 2022-08-17 13:47:34.000000000 +0200 @@ -69,7 +69,7 @@ }, } -func check(path string, d dir, allowType int64) error { +func check(path string, d dir) error { stfs := &unix.Statfs_t{} if err := statfs(path, stfs); err != nil { @@ -81,10 +81,6 @@ return nil } - if stfs.Type == allowType { - return nil - } - return &errIncompatibleFs{ path: path, name: fs.name, @@ -94,16 +90,14 @@ // CheckUpper checks if the underlying filesystem of the // provided path can be used as an upper overlay directory. -// allowType is an optional filesystem type to always allow. -func CheckUpper(path string, allowType int64) error { - return check(path, upperDir, allowType) +func CheckUpper(path string) error { + return check(path, upperDir) } // CheckLower checks if the underlying filesystem of the // provided path can be used as lower overlay directory. -// allowType is an optional filesystem type to always allow. -func CheckLower(path string, allowType int64) error { - return check(path, lowerDir, allowType) +func CheckLower(path string) error { + return check(path, lowerDir) } type errIncompatibleFs struct { diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/internal/pkg/util/fs/overlay/overlay_linux_test.go new/apptainer-1.1.0-rc.2/internal/pkg/util/fs/overlay/overlay_linux_test.go --- old/apptainer-1.1.0-rc.1/internal/pkg/util/fs/overlay/overlay_linux_test.go 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/internal/pkg/util/fs/overlay/overlay_linux_test.go 2022-08-17 13:47:34.000000000 +0200 @@ -170,9 +170,9 @@ switch tt.dir { case lowerDir: - err = CheckLower(tt.path, 0) + err = CheckLower(tt.path) case upperDir: - err = CheckUpper(tt.path, 0) + err = CheckUpper(tt.path) } if err != nil && tt.expectedSuccess { diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/internal/pkg/util/gpu/nvidia.go new/apptainer-1.1.0-rc.2/internal/pkg/util/gpu/nvidia.go --- old/apptainer-1.1.0-rc.1/internal/pkg/util/gpu/nvidia.go 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/internal/pkg/util/gpu/nvidia.go 2022-08-17 13:47:34.000000000 +0200 @@ -23,6 +23,7 @@ "github.com/apptainer/apptainer/pkg/sylog" "github.com/apptainer/apptainer/pkg/util/capabilities" "github.com/apptainer/apptainer/pkg/util/slice" + "golang.org/x/sys/unix" ) var ( @@ -63,6 +64,7 @@ uintptr(capabilities.Map["CAP_SYS_PTRACE"].Value), uintptr(capabilities.Map["CAP_DAC_OVERRIDE"].Value), uintptr(capabilities.Map["CAP_SETPCAP"].Value), + uintptr(capabilities.Map["CAP_NET_ADMIN"].Value), } // NVCLIConfigure calls out to the nvidia-container-cli configure operation. @@ -88,6 +90,16 @@ return errNvCCLIInsecure } + // Make sure the /usr/bin is writable in the container + binpath := rootfs + "/usr/bin" + if err := unix.Access(binpath, unix.W_OK); err != nil { + sylog.Debugf("%v not writable: %v", binpath, err) + if os.Getuid() != 0 { + sylog.Infof("Consider using either `--fakeroot` or sandbox built with `--fix-perms`") + } + return fmt.Errorf("/usr/bin not writable in the container") + } + // Translate the passed in NVIDIA_ env vars to option flags flags, err := NVCLIEnvToFlags(nvidiaEnv) if err != nil { diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/pkg/util/cryptkey/key.go new/apptainer-1.1.0-rc.2/pkg/util/cryptkey/key.go --- old/apptainer-1.1.0-rc.1/pkg/util/cryptkey/key.go 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/pkg/util/cryptkey/key.go 2022-08-17 13:47:34.000000000 +0200 @@ -22,6 +22,7 @@ "io" "io/ioutil" "os" + "strings" "github.com/apptainer/sif/v2/pkg/sif" ) @@ -155,6 +156,10 @@ return nil, fmt.Errorf("could not read %s: %v", fn, ErrNoPEMData) } + if strings.Contains(block.Headers["Proc-Type"], "ENCRYPTED") { + return nil, fmt.Errorf("passphrase protected pem files not supported") + } + return x509.ParsePKCS1PrivateKey(block.Bytes) } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/scripts/ci-deb-build-test new/apptainer-1.1.0-rc.2/scripts/ci-deb-build-test --- old/apptainer-1.1.0-rc.1/scripts/ci-deb-build-test 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/scripts/ci-deb-build-test 2022-08-17 13:47:34.000000000 +0200 @@ -12,6 +12,7 @@ build-essential \ libseccomp-dev \ pkg-config \ + uidmap \ squashfs-tools \ squashfuse \ fuse2fs \ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/apptainer-1.1.0-rc.1/scripts/ci-rpm-build-test new/apptainer-1.1.0-rc.2/scripts/ci-rpm-build-test --- old/apptainer-1.1.0-rc.1/scripts/ci-rpm-build-test 2022-08-01 23:52:07.000000000 +0200 +++ new/apptainer-1.1.0-rc.2/scripts/ci-rpm-build-test 2022-08-17 13:47:34.000000000 +0200 @@ -10,7 +10,7 @@ yum install -y rpm-build make yum-utils gcc binutils util-linux-ng which git yum install -y libseccomp-devel e2fsprogs cryptsetup yum install -y epel-release -yum install -y golang squashfuse fuse-overlayfs fakeroot +yum install -y golang squashfuse fuse-overlayfs fakeroot /usr/*bin/fuse2fs # switch to an unprivileged user with sudo privileges yum install -y sudo ++++++ vendor.tar.gz ++++++ /work/SRC/openSUSE:Factory/apptainer/vendor.tar.gz /work/SRC/openSUSE:Factory/.apptainer.new.2083/vendor.tar.gz differ: char 5, line 1
