Re: [gentoo-dev] Functional portage with namespace
Il 17/03/2018 00:40, Kent Fredric ha scritto: > On Mon, 12 Mar 2018 07:55:46 +0900 > Benda Xu wrote: > >> Ha, indeed many packages hardwrites "date of build" alike. That is a >> hard question to define reproducibility. I would rather ignore the >> timestamps when comparing two binaries. > If a hard-timestamp is to be used, assuming you have portage via git, > then it might be desirable to hard-timestamp based on either: > > a) the timestamp of the specific ebuilds last change > b) the timestamp of the most-recent-of specific ebuild+eclass's last change > c) the timestamp of the specific ebuilds initial commit d) for rsync users the timestamp of the repository, kept in metadata/, the timestamp of last commit otherwise > > I'm not sure which one is more practical though. > > Sounds like it would be an "experts" tool which would become far more > practical > for people who are using custom overlays to maintain their production systems, > and those people can naturally make guarantees about their repos being in git, > and they can decide which of those 3 options ( well, the ones we > provide at least ) are most suited to what they're doing. >
Re: [gentoo-dev] Functional portage with namespace
On Mon, 12 Mar 2018 07:55:46 +0900 Benda Xu wrote: > Ha, indeed many packages hardwrites "date of build" alike. That is a > hard question to define reproducibility. I would rather ignore the > timestamps when comparing two binaries. If a hard-timestamp is to be used, assuming you have portage via git, then it might be desirable to hard-timestamp based on either: a) the timestamp of the specific ebuilds last change b) the timestamp of the most-recent-of specific ebuild+eclass's last change c) the timestamp of the specific ebuilds initial commit I'm not sure which one is more practical though. Sounds like it would be an "experts" tool which would become far more practical for people who are using custom overlays to maintain their production systems, and those people can naturally make guarantees about their repos being in git, and they can decide which of those 3 options ( well, the ones we provide at least ) are most suited to what they're doing. pgpArrr2WrJNn.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] Functional portage with namespace
Hi Xdej, X dej writes: > If you want to have reproducible binaries, you may also want to alter > the clock of the sandbox system. Ha, indeed many packages hardwrites "date of build" alike. That is a hard question to define reproducibility. I would rather ignore the timestamps when comparing two binaries. Benda
Re: [gentoo-dev] Functional portage with namespace (Was: Integrating Portage with other package managers)
2018-03-09 1:46 UTC+01:00, Benda Xu : > Just a side node, this seems to be the ultimate sandbox we (Gentoo and > portage) are after. With this, we might even be able to have portage > full functional: a build is completely determined and only determined by > the dependencies and USE flags. If you want to have reproducible binaries, you may also want to alter the clock of the sandbox system. Suggestions: At start, set the date as EPOCH (i.e., in 1970). Each time a file date seems to be set in the future (by tar, ...), set the date of the sandbox system to that date plus a milisecond. Each time a file is created or touched, advance the date of the sandbox system by one milisecond. Xdej
Re: [gentoo-dev] Functional portage with namespace (Was: Integrating Portage with other package managers)
On Thu, Mar 8, 2018 at 9:11 PM, R0b0t1 wrote: > > Sadly interest in the patches seems to have waned. The functionality > is not exactly duplicated in containers, but they do make it easier to > find changes. > Well, the idea with containers wouldn't be to monitor anything, but instead to ensure that there is nothing in the container the ebuild shouldn't need. If the package declares 3 build-time dependencies then the container would contain @system plus those three dependencies. If the build system chokes then it will choke for everybody (eg for the maintainer testing the package), and not randomly when some user doesn't have some package installed that wasn't declared. Now, a container wouldn't help with figuring out what the dependencies actually are. Besides inspection/etc a quick solution there is to just do a build without a container and check the linking, then go back and fix the deps, and then rebuild to verify that nothing else was needed. -- Rich
Re: [gentoo-dev] Functional portage with namespace (Was: Integrating Portage with other package managers)
On Thu, Mar 8, 2018 at 7:06 PM, Rich Freeman wrote: > On Thu, Mar 8, 2018 at 7:46 PM, Benda Xu wrote: >> Rich Freeman writes: >> >>> If you have util-linux installed then try running (as any user - you >>> don't have to be root): unshare -i -m -n -p -u -C -f --mount-proc -U >>> -r /bin/bash >>> >>> Congrats. You are now root in a container. You're in the same root >>> filesystem as always. You'll note that you can't actually see >>> anything that you couldn't see before. If you run ps -ea you'll see >>> that you're the only process running on the system. Devices like >>> /dev/sda aren't actually accessible. A lot of container managers >>> would mount a new /dev and just hide most of that stuff. You can >>> probably imagine how something like this could be useful for isolating >>> processes. >> >> Just a side node, this seems to be the ultimate sandbox we (Gentoo and >> portage) are after. With this, we might even be able to have portage >> full functional: a build is completely determined and only determined by >> the dependencies and USE flags. >> > > I'm not sure how well this would perform, but I had given this a bit > of thought a few years ago. There are two obvious ways to go about > this: the sandbox or containers. > It may be possible to use fanotify(7) (but this would only catch disk access). There is an extension to fanotify created as someone's thesis (http://lkml.iu.edu/hypermail/linux/kernel/1703.1/05249.html) that allows recursive watches. One of the features of fanotify is that it is possible to deny the events generated. The fanotify API may simplify the application of the generated rules. It was designed to be fast, and most processing is done in the kernel when calls are made. Sadly interest in the patches seems to have waned. The functionality is not exactly duplicated in containers, but they do make it easier to find changes.
Re: [gentoo-dev] Functional portage with namespace (Was: Integrating Portage with other package managers)
On Thu, Mar 8, 2018 at 7:46 PM, Benda Xu wrote: > Rich Freeman writes: > >> If you have util-linux installed then try running (as any user - you >> don't have to be root): unshare -i -m -n -p -u -C -f --mount-proc -U >> -r /bin/bash >> >> Congrats. You are now root in a container. You're in the same root >> filesystem as always. You'll note that you can't actually see >> anything that you couldn't see before. If you run ps -ea you'll see >> that you're the only process running on the system. Devices like >> /dev/sda aren't actually accessible. A lot of container managers >> would mount a new /dev and just hide most of that stuff. You can >> probably imagine how something like this could be useful for isolating >> processes. > > Just a side node, this seems to be the ultimate sandbox we (Gentoo and > portage) are after. With this, we might even be able to have portage > full functional: a build is completely determined and only determined by > the dependencies and USE flags. > I'm not sure how well this would perform, but I had given this a bit of thought a few years ago. There are two obvious ways to go about this: the sandbox or containers. Either way you first go through the declared dependencies (and @system under our current policies) and obtain a list of all files installed by these packages. Then you either bind mount those files into a container, or add them to the sandbox configuration as readable. Right now the sandbox rules are set to allow read access by default on everything, but this is just a one-liner to change, which would cause the sandbox to generate an error if a file is read which isn't explicitly enabled. The container solution is going to be cleaner because it also helps curb errors from automagic build systems. These build systems wouldn't even be able to detect files not owned by a declared dependency. With the sandbox approach those files would be visible, but would generate errors when accessed (so at least we get a bug report, but users still have to deal with the issue). Sure, this would to some extent hide problems, but then again the automagic build system would be defeated all the same. The real question is how long it takes to extract those file lists and create all those bind mounts. In the case of @system some of this could be cached. I'm pretty sure the file lists would be quite fast - we only need to go one level deep on the dependency tree (I think). I suspect the bind mounts would be fast, but I haven't tried to benchmark it. I also have no idea how sandbox performs with a very long list of rules. -- Rich
[gentoo-dev] Functional portage with namespace (Was: Integrating Portage with other package managers)
Rich Freeman writes: > If you have util-linux installed then try running (as any user - you > don't have to be root): unshare -i -m -n -p -u -C -f --mount-proc -U > -r /bin/bash > > Congrats. You are now root in a container. You're in the same root > filesystem as always. You'll note that you can't actually see > anything that you couldn't see before. If you run ps -ea you'll see > that you're the only process running on the system. Devices like > /dev/sda aren't actually accessible. A lot of container managers > would mount a new /dev and just hide most of that stuff. You can > probably imagine how something like this could be useful for isolating > processes. Just a side node, this seems to be the ultimate sandbox we (Gentoo and portage) are after. With this, we might even be able to have portage full functional: a build is completely determined and only determined by the dependencies and USE flags.
