Bug#910044: mmdebstrap: problem with sources.list-style mirror parameter handling
On Tue, Oct 02, 2018 at 10:13:42AM +0200, Johannes Schauer wrote: > Quoting Johannes Schauer (2018-10-02 04:23:54) > > > Another thing that I have stumbled upon is that providing a sources.list > > > on > > > stdin only works when no target directory is provided on the commandline, > > > i.e. when automatically creating a tarball. While > > > > > > echo -e "deb http://deb.debian.org/debian-ports/ sid main\ndeb > > > http://deb.debian.org/debian-ports/ unreleased main" | mmdebstrap > > > --architectures=riscv64 > /tmp/rv64-chroot.tar > > > > > > works, the following doesn't: > > > > > > echo -e "deb http://deb.debian.org/debian-ports/ sid main\ndeb > > > http://deb.debian.org/debian-ports/ unreleased main" | mmdebstrap > > > --architectures=riscv64 sid /tmp/rv64-chroot > > > > > > It results in > > > > > > I: riscv64 cannot be executed, falling back to qemu-user > > > I: automatically chosen mode: root > > > I: running apt-get update... > > > done > > > apt-get update didn't download anything at /usr/bin/mmdebstrap line 721. > > > > This is actually a feature. In your second invocation, you didn't specify > > the > > "MIRROR" argument. The docs say: > > > > > If no MIRROR option is provided, http://deb.debian.org/debian is used. > > > > In your first invocation, you also didn't specify the "SUITE" argument. The > > docs say: > > > > > If no SUITE was specified, then a single MIRROR "-" is added and thus the > > > information of the desired suite has to come from standard input as part > > > of a > > > valid apt sources.list file. > > > > So the implicit '-' as the mirror argument only works if there was no > > SUITE. If > > you pass a SUITE, you need to explicitly tell it that you want to read the > > mirror from stdin. This is to have compatibility with how debootstrap also > > just > > uses deb.debian.org/debian as the mirror if the mirror argument is missing. > > > > To make the cause of the problem more obvious, I'm now printing the content > > of > > /etc/apt/sources.list in the chroot in case apt-get didn't download > > anything: > > > > https://gitlab.mister-muffin.de/josch/mmdebstrap/commit/1f13d0157bc8364ac491203a6a9156a78f6228a9 > > > > But we could turn this bug into a feature request of the following form: > > > > If no MIRROR was provided *and* something was specified on standard input, > > then > > don't use deb.debian.org/debian but use whatever was given on standard input > > instead. > > > > Or even more liberal: > > > > If anything is provided on standard input and there is no '-' MIRROR > > argument, > > then just append whatever is given on standard input to the apt sources.list > > inside the chroot. > > > > What do you think? > > now that I've slept about it, I think your idea makes sense. If you don't > specify a mirror but you pass something on standard input, then the default > mirror should *not* be used. The default mirror should only be used if neither > a mirror nor anything from standard input was given. > > I pushed a fix to the upstream repository. I would be happy if you could > download the latest version of the shell script from there and test if it > works > as you had expected it to work. Many thanks! I've run a number of tests covering different options (native/foreign chroot, implicit/explicit mirror, plain-url/sources.list-entry, mirror information via stdin/on the commandline, target directory/tarball) with the current upstream git head and everything looks good: * native chroot with default mirror and with a directory as the target: # mmdebstrap --include="debian-ports-archive-keyring" sid /tmp/amd64-chroot I: chroot architecture amd64 is equal to the host's architecture I: automatically chosen mode: root I: running apt-get update... done I: downloading packages with apt... done I: extracting archives... done I: installing packages... done I: downloading apt... done I: installing apt... done I: installing remaining packages inside the chroot... done I: cleaning package lists and apt cache... done done * native chroot with a mirror URL on the commandline and a directory as the target: # mmdebstrap --include="debian-ports-archive-keyring" sid /tmp/amd64-chroot http://deb.debian.org/debian I: chroot architecture amd64 is equal to the host's architecture I: automatically chosen mode: root I: running apt-get update... done I: downloading packages with apt... done I: extracting archives... done I: installing packages... done I: downloading apt... done I: installing apt... done I: installing remaining packages inside the chroot... done I: cleaning package lists and apt cache... done done * foreign (riscv64) chroot with sources.list passed on stdin and with a directory as the target: # echo -e "deb http://deb.debian.org/debian-ports/ sid main\ndeb http://deb.debian.org/debian-ports/ unreleased main" | mmdebstrap --architectures=ri
Bug#910044: mmdebstrap: problem with sources.list-style mirror parameter handling
Hi Karsten, Quoting Johannes Schauer (2018-10-02 04:23:54) > > Another thing that I have stumbled upon is that providing a sources.list on > > stdin only works when no target directory is provided on the commandline, > > i.e. when automatically creating a tarball. While > > > > echo -e "deb http://deb.debian.org/debian-ports/ sid main\ndeb > > http://deb.debian.org/debian-ports/ unreleased main" | mmdebstrap > > --architectures=riscv64 > /tmp/rv64-chroot.tar > > > > works, the following doesn't: > > > > echo -e "deb http://deb.debian.org/debian-ports/ sid main\ndeb > > http://deb.debian.org/debian-ports/ unreleased main" | mmdebstrap > > --architectures=riscv64 sid /tmp/rv64-chroot > > > > It results in > > > > I: riscv64 cannot be executed, falling back to qemu-user > > I: automatically chosen mode: root > > I: running apt-get update... > > done > > apt-get update didn't download anything at /usr/bin/mmdebstrap line 721. > > This is actually a feature. In your second invocation, you didn't specify the > "MIRROR" argument. The docs say: > > > If no MIRROR option is provided, http://deb.debian.org/debian is used. > > In your first invocation, you also didn't specify the "SUITE" argument. The > docs say: > > > If no SUITE was specified, then a single MIRROR "-" is added and thus the > > information of the desired suite has to come from standard input as part of > > a > > valid apt sources.list file. > > So the implicit '-' as the mirror argument only works if there was no SUITE. > If > you pass a SUITE, you need to explicitly tell it that you want to read the > mirror from stdin. This is to have compatibility with how debootstrap also > just > uses deb.debian.org/debian as the mirror if the mirror argument is missing. > > To make the cause of the problem more obvious, I'm now printing the content of > /etc/apt/sources.list in the chroot in case apt-get didn't download anything: > > https://gitlab.mister-muffin.de/josch/mmdebstrap/commit/1f13d0157bc8364ac491203a6a9156a78f6228a9 > > But we could turn this bug into a feature request of the following form: > > If no MIRROR was provided *and* something was specified on standard input, > then > don't use deb.debian.org/debian but use whatever was given on standard input > instead. > > Or even more liberal: > > If anything is provided on standard input and there is no '-' MIRROR argument, > then just append whatever is given on standard input to the apt sources.list > inside the chroot. > > What do you think? now that I've slept about it, I think your idea makes sense. If you don't specify a mirror but you pass something on standard input, then the default mirror should *not* be used. The default mirror should only be used if neither a mirror nor anything from standard input was given. I pushed a fix to the upstream repository. I would be happy if you could download the latest version of the shell script from there and test if it works as you had expected it to work. Thanks! cheers, josch signature.asc Description: signature
Bug#910044: mmdebstrap: problem with sources.list-style mirror parameter handling
Hi Karsten, Quoting Karsten Merker (2018-10-01 22:43:31) > many thanks for creating mmdebstrap! I'm using it for creating > chroots for Debian-Ports architectures, where we need to pull the > packages from two suites: unstable and unreleased. While using > mmdebstrap, I've stumbled over two issues. Everything works fine > when creating a sources.list, feeding it to mmdebstrap on stdin > and having mmdebstrap automatically generate a tarball, i.e. > with something like > > echo -e "deb http://deb.debian.org/debian-ports/ sid main\ndeb > http://deb.debian.org/debian-ports/ unreleased main" | mmdebstrap > --architectures=riscv64 > /tmp/rv64-chroot.tar > > The process fails when one tries to pass the mirrors on the > commandline, though. Due to the fact that there is more than one > suite to use, the mirrors need to be passed in sources.list > style, i.e. as "deb http:///debian-ports main". > > Running mmdebstrap with > > mmdebstrap --architectures=riscv64 sid /tmp/rv64-chroot "deb > http://deb.debian.org/debian-ports/ sid main" "deb > http://deb.debian.org/debian-ports/ unreleased main" > > results in > > E: Malformed entry 1 in list file /tmp/rv64-chroot/etc/apt/sources.list > (URI parse) > E: The list of sources could not be read. > apt-get update failed at /usr/bin/mmdebstrap line 548. > # cat /tmp/rv64-chroot/etc/apt/sources.list > deb deb http://deb.debian.org/debian-ports/ sid main sid main > deb deb http://deb.debian.org/debian-ports/ unreleased main sid main > > The manpage states: > > If a MIRROR option starts with "deb " or "deb-src " then it is > used as a one-line-style format entry for apt's sources.list > inside the chroot. If a MIRROR option contains a "://" then it > is interpreted as a mirror URI and the apt line inside the chroot > is assembled as "deb [arch=A] B C D" where A is the host's native > architecture, B is the MIRROR, C is the given SUITE and D is the > components given via --components (defaults to "main"). > > It looks like the second part is applied unconditionally on > all mirror parameters that contain a "://", although that should > not happen if the mirror parameter starts with a "deb " or > "deb-src " and thereby already constitutes a complete sources.list > entry. yes, that is a bug. The code was indeed wrongly first checking for :// even though it should first check if the parameter starts with "deb " or "deb-src ". Fixed here: https://gitlab.mister-muffin.de/josch/mmdebstrap/commit/bb2aa6e9fdbbf4723eacdaf5937a8855ff41509d > Another thing that I have stumbled upon is that providing a sources.list on > stdin only works when no target directory is provided on the commandline, > i.e. when automatically creating a tarball. While > > echo -e "deb http://deb.debian.org/debian-ports/ sid main\ndeb > http://deb.debian.org/debian-ports/ unreleased main" | mmdebstrap > --architectures=riscv64 > /tmp/rv64-chroot.tar > > works, the following doesn't: > > echo -e "deb http://deb.debian.org/debian-ports/ sid main\ndeb > http://deb.debian.org/debian-ports/ unreleased main" | mmdebstrap > --architectures=riscv64 sid /tmp/rv64-chroot > > It results in > > I: riscv64 cannot be executed, falling back to qemu-user > I: automatically chosen mode: root > I: running apt-get update... > done > apt-get update didn't download anything at /usr/bin/mmdebstrap line 721. This is actually a feature. In your second invocation, you didn't specify the "MIRROR" argument. The docs say: > If no MIRROR option is provided, http://deb.debian.org/debian is used. In your first invocation, you also didn't specify the "SUITE" argument. The docs say: > If no SUITE was specified, then a single MIRROR "-" is added and thus the > information of the desired suite has to come from standard input as part of a > valid apt sources.list file. So the implicit '-' as the mirror argument only works if there was no SUITE. If you pass a SUITE, you need to explicitly tell it that you want to read the mirror from stdin. This is to have compatibility with how debootstrap also just uses deb.debian.org/debian as the mirror if the mirror argument is missing. To make the cause of the problem more obvious, I'm now printing the content of /etc/apt/sources.list in the chroot in case apt-get didn't download anything: https://gitlab.mister-muffin.de/josch/mmdebstrap/commit/1f13d0157bc8364ac491203a6a9156a78f6228a9 But we could turn this bug into a feature request of the following form: If no MIRROR was provided *and* something was specified on standard input, then don't use deb.debian.org/debian but use whatever was given on standard input instead. Or even more liberal: If anything is provided on standard input and there is no '-' MIRROR argument, then just append whatever is given on standard input to the apt sources.list inside the chroot. What do you think? Thanks! cheers, josch signature.asc Description: signature
Bug#910044: mmdebstrap: problem with sources.list-style mirror parameter handling
Package: mmdebstrap Version: 0.1.0-2 Hello, many thanks for creating mmdebstrap! I'm using it for creating chroots for Debian-Ports architectures, where we need to pull the packages from two suites: unstable and unreleased. While using mmdebstrap, I've stumbled over two issues. Everything works fine when creating a sources.list, feeding it to mmdebstrap on stdin and having mmdebstrap automatically generate a tarball, i.e. with something like echo -e "deb http://deb.debian.org/debian-ports/ sid main\ndeb http://deb.debian.org/debian-ports/ unreleased main" | mmdebstrap --architectures=riscv64 > /tmp/rv64-chroot.tar The process fails when one tries to pass the mirrors on the commandline, though. Due to the fact that there is more than one suite to use, the mirrors need to be passed in sources.list style, i.e. as "deb http:///debian-ports main". Running mmdebstrap with mmdebstrap --architectures=riscv64 sid /tmp/rv64-chroot "deb http://deb.debian.org/debian-ports/ sid main" "deb http://deb.debian.org/debian-ports/ unreleased main" results in E: Malformed entry 1 in list file /tmp/rv64-chroot/etc/apt/sources.list (URI parse) E: The list of sources could not be read. apt-get update failed at /usr/bin/mmdebstrap line 548. # cat /tmp/rv64-chroot/etc/apt/sources.list deb deb http://deb.debian.org/debian-ports/ sid main sid main deb deb http://deb.debian.org/debian-ports/ unreleased main sid main The manpage states: If a MIRROR option starts with "deb " or "deb-src " then it is used as a one-line-style format entry for apt's sources.list inside the chroot. If a MIRROR option contains a "://" then it is interpreted as a mirror URI and the apt line inside the chroot is assembled as "deb [arch=A] B C D" where A is the host's native architecture, B is the MIRROR, C is the given SUITE and D is the components given via --components (defaults to "main"). It looks like the second part is applied unconditionally on all mirror parameters that contain a "://", although that should not happen if the mirror parameter starts with a "deb " or "deb-src " and thereby already constitutes a complete sources.list entry. Another thing that I have stumbled upon is that providing a sources.list on stdin only works when no target directory is provided on the commandline, i.e. when automatically creating a tarball. While echo -e "deb http://deb.debian.org/debian-ports/ sid main\ndeb http://deb.debian.org/debian-ports/ unreleased main" | mmdebstrap --architectures=riscv64 > /tmp/rv64-chroot.tar works, the following doesn't: echo -e "deb http://deb.debian.org/debian-ports/ sid main\ndeb http://deb.debian.org/debian-ports/ unreleased main" | mmdebstrap --architectures=riscv64 sid /tmp/rv64-chroot It results in I: riscv64 cannot be executed, falling back to qemu-user I: automatically chosen mode: root I: running apt-get update... done apt-get update didn't download anything at /usr/bin/mmdebstrap line 721. Regards, Karsten -- Gem. Par. 28 Abs. 4 Bundesdatenschutzgesetz widerspreche ich der Nutzung sowie der Weitergabe meiner personenbezogenen Daten für Zwecke der Werbung sowie der Markt- oder Meinungsforschung.
