Re: [gentoo-dev] New project: binhost
Yes, it seems they use that approach as the binary repos are called "grp".At the same time they provide binary builds for set of packages not only on release but as rolling too. They repository is available as Gentoo Calculate Overlay but I never tries to add it and use so I don't know how it affect system. Anyway own Gentoo binhost is great idea and will be very usefull. --Sergey Torokhov13.03.2021, 13:48, "Marco Scardovi" :Hi there,Well, actually you could take a look on GRP project as, if I understand, was similar to what you would like to do: https://en.wikipedia.org/wiki/Gentoo_Linux#Gentoo_Reference_PlatformUnfortunately this project was closed on 2011, so lots of things are changed, but could be a good startIl Sab 13 Mar 2021, 09:44 Torokhov Sergeyha scritto:Maybe the expirience of Calculate Linux will be usefull. It's Gentoo based distributive that use portage/emerge and have huge set of prebuild packages in *.xpack format.https://wiki.calculate-linux.org/calculate_vs_gentoo10.02.2021, 20:58, "Andreas K. Hüttel" :Hi all, I'm announcing a new project here - "binhost""The Gentoo Binhost project aims to provide readily installable, precompiled packages for a subset of configurations, via central binary package hosting. Currently we are still in the conceptual planning stage. "https://wiki.gentoo.org/wiki/Project:BinhostIf you're interested in helping out, feel free to add yourself on the wiki page.Note that I see actually *building* the packages not as the central point of the project (that could be e.g. a side effect of a tinderbox). I'm more concerned about* what configurations should we use* what portage features are still needed or need improvements (e.g. binpkg signing and verification)* how should hosting look like* and how we can test this on a limited scale before it goes "into production"* ...Comments, ideas, flamebaits? :DCheers, Andreas-- Andreas K. Hütteldilfri...@gentoo.orgGentoo Linux developer (council, qa, toolchain, base-system, perl, libreoffice)
Re: [gentoo-dev] New project: binhost
Le 2/10/21 à 9:04 PM, Frédéric Pierret a écrit : Le 2/10/21 à 6:57 PM, Andreas K. Hüttel a écrit : Hi all, I'm announcing a new project here - "binhost" "The Gentoo Binhost project aims to provide readily installable, precompiled packages for a subset of configurations, via central binary package hosting. Currently we are still in the conceptual planning stage. " https://wiki.gentoo.org/wiki/Project:Binhost If you're interested in helping out, feel free to add yourself on the wiki page. Note that I see actually *building* the packages not as the central point of the project (that could be e.g. a side effect of a tinderbox). I'm more concerned about * what configurations should we use * what portage features are still needed or need improvements (e.g. binpkg signing and verification) * how should hosting look like * and how we can test this on a limited scale before it goes "into production" * ... Comments, ideas, flamebaits? :D Cheers, Andreas Hi Andreas, I'm pretty interested to help for this topic. Notably, for my work on creating and maintaining Gentoo template for Qubes OS, I'm weekly constructing binpkgs mirrors in order to ease rebuilding from scratch Gentoo templates and also for CI purposes. So I would be glad to help the whole Gentoo community in such efforts. I'm also following a very interesting work here on portage: https://github.com/gentoo/portage/pull/562 Best regards, Frédéric Hi, A quick update here, I'm currently testing and validating the work done in https://github.com/gentoo/portage/pull/562 in order to give some feedback to Zac hoping we could have this new feature for gpkg available soon in Portage. If anyone wants to try, you just have to use @RinCat branch. I've simply emerged Portage by replacing in portage-.ebuild: EGIT_REPO_URI="https://github.com/RinCat/portage.git; EGIT_BRANCH="gpkg" and a quick make.conf example can be found in here: https://github.com/gentoo/portage/pull/562#issuecomment-797112962. As of today, the gpkg and signature are working properly. I'm currently building a Gentoo from scratch with this feature and then, I would use it as binhost with signature validation enabled. Best regards, Frédéric OpenPGP_signature Description: OpenPGP digital signature
Re: [gentoo-dev] New project: binhost
I'm unable to add myself to the wiki, as i am not a real dev (yet!) but i am definitely interested in being a part of this team and contributing to the discussions. El sáb., 13 de marzo de 2021 5:48 a. m., Marco Scardovi escribió: > Hi there, > > Well, actually you could take a look on GRP project as, if I understand, > was similar to what you would like to do: > https://en.wikipedia.org/wiki/Gentoo_Linux#Gentoo_Reference_Platform > > Unfortunately this project was closed on 2011, so lots of things are > changed, but could be a good start > > > Il Sab 13 Mar 2021, 09:44 Torokhov Sergey ha > scritto: > >> Maybe the expirience of Calculate Linux will be usefull. It's Gentoo >> based distributive that use portage/emerge and have huge set of prebuild >> packages in *.xpack format. >> >> https://wiki.calculate-linux.org/calculate_vs_gentoo >> >> 10.02.2021, 20:58, "Andreas K. Hüttel" : >> >> Hi all, >> >> I'm announcing a new project here - "binhost" >> >> "The Gentoo Binhost project aims to provide readily installable, >> precompiled >> packages for a subset of configurations, via central binary package >> hosting. >> Currently we are still in the conceptual planning stage. " >> >> https://wiki.gentoo.org/wiki/Project:Binhost >> >> If you're interested in helping out, feel free to add yourself on the >> wiki >> page. >> >> Note that I see actually *building* the packages not as the central point >> of >> the project (that could be e.g. a side effect of a tinderbox). I'm more >> concerned about >> * what configurations should we use >> * what portage features are still needed or need improvements (e.g. >> binpkg >> signing and verification) >> * how should hosting look like >> * and how we can test this on a limited scale before it goes "into >> production" >> * ... >> >> Comments, ideas, flamebaits? :D >> >> Cheers, >> Andreas >> >> -- >> Andreas K. Hüttel >> dilfri...@gentoo.org >> Gentoo Linux developer >> (council, qa, toolchain, base-system, perl, libreoffice) >> >>
Re: [gentoo-dev] New project: binhost
Hi there, Well, actually you could take a look on GRP project as, if I understand, was similar to what you would like to do: https://en.wikipedia.org/wiki/Gentoo_Linux#Gentoo_Reference_Platform Unfortunately this project was closed on 2011, so lots of things are changed, but could be a good start Il Sab 13 Mar 2021, 09:44 Torokhov Sergey ha scritto: > Maybe the expirience of Calculate Linux will be usefull. It's Gentoo based > distributive that use portage/emerge and have huge set of prebuild packages > in *.xpack format. > > https://wiki.calculate-linux.org/calculate_vs_gentoo > > 10.02.2021, 20:58, "Andreas K. Hüttel" : > > Hi all, > > I'm announcing a new project here - "binhost" > > "The Gentoo Binhost project aims to provide readily installable, > precompiled > packages for a subset of configurations, via central binary package > hosting. > Currently we are still in the conceptual planning stage. " > > https://wiki.gentoo.org/wiki/Project:Binhost > > If you're interested in helping out, feel free to add yourself on the wiki > page. > > Note that I see actually *building* the packages not as the central point > of > the project (that could be e.g. a side effect of a tinderbox). I'm more > concerned about > * what configurations should we use > * what portage features are still needed or need improvements (e.g. binpkg > signing and verification) > * how should hosting look like > * and how we can test this on a limited scale before it goes "into > production" > * ... > > Comments, ideas, flamebaits? :D > > Cheers, > Andreas > > -- > Andreas K. Hüttel > dilfri...@gentoo.org > Gentoo Linux developer > (council, qa, toolchain, base-system, perl, libreoffice) > >
[gentoo-dev] New project: binhost
Maybe the expirience of Calculate Linux will be usefull. It's Gentoo based distributive that use portage/emerge and have huge set of prebuild packages in *.xpack format.https://wiki.calculate-linux.org/calculate_vs_gentoo10.02.2021, 20:58, "Andreas K. Hüttel":Hi all, I'm announcing a new project here - "binhost""The Gentoo Binhost project aims to provide readily installable, precompiled packages for a subset of configurations, via central binary package hosting. Currently we are still in the conceptual planning stage. "https://wiki.gentoo.org/wiki/Project:BinhostIf you're interested in helping out, feel free to add yourself on the wiki page.Note that I see actually *building* the packages not as the central point of the project (that could be e.g. a side effect of a tinderbox). I'm more concerned about* what configurations should we use* what portage features are still needed or need improvements (e.g. binpkg signing and verification)* how should hosting look like* and how we can test this on a limited scale before it goes "into production"* ...Comments, ideas, flamebaits? :DCheers, Andreas-- Andreas K. Hütteldilfri...@gentoo.orgGentoo Linux developer (council, qa, toolchain, base-system, perl, libreoffice)
Re: [gentoo-dev] New project: binhost
On 2/24/21 2:29 AM, Zac Medico wrote: > For example, for 3 USE flags, up to 8 combinations will be indexed: > > IUSE="a b c installsources splitdebug" > SRC_URI=" > !a? !b? !c? ( mirror://binhost/24fe6bd377 ) > !a? !b? c? ( mirror://binhost/fbe14cbb02 ) > !a? b? !c? ( mirror://binhost/1dfff1f2ac ) > !a? b? c? ( mirror://binhost/ae60f2940d ) >a? !b? !c? ( mirror://binhost/2976e1acc0 ) >a? !b? c? ( mirror://binhost/f4809db70c ) >a? b? !c? ( mirror://binhost/ecd08466cf ) >a? b? c? ( mirror://binhost/0c00f33b2e ) > installsources? ( > !a? !b? !c? ( mirror://binhost/063a14d6c7 ) > !a? !b? c? ( mirror://binhost/f67c311625 ) > !a? b? !c? ( mirror://binhost/1dfff1f2ac ) > !a? b? c? ( mirror://binhost/17a673e16a ) > a? !b? !c? ( mirror://binhost/914d1cecfe ) > a? !b? c? ( mirror://binhost/ca18d86a2b ) > a? b? !c? ( mirror://binhost/6bce13471a ) > a? b? c? ( mirror://binhost/3a6bdcd228 ) > ) > splitdebug? ( > !a? !b? !c? ( mirror://binhost/29b2f38c41 ) > !a? !b? c? ( mirror://binhost/8adc9bef51 ) > !a? b? !c? ( mirror://binhost/954d2ce484 ) > !a? b? c? ( mirror://binhost/32a614aaca ) > a? !b? !c? ( mirror://binhost/3548a2302d ) > a? !b? c? ( mirror://binhost/e0c02cdc88 ) > a? b? !c? ( mirror://binhost/f9cbd3c181 ) > a? b? c? ( mirror://binhost/31d4c03474 ) > ) > " > > For installsources, we can automate deduplication, so that we can > distribute the same file content for multiple USE combinations when > appropriate. If all of the combinations have identical content, then it > will look like this: > > installsources? ( > !a? !b? !c? ( mirror://binhost/063a14d6c7 ) > !a? !b? c? ( mirror://binhost/063a14d6c7 ) > !a? b? !c? ( mirror://binhost/063a14d6c7 ) > !a? b? c? ( mirror://binhost/063a14d6c7 ) > a? !b? !c? ( mirror://binhost/063a14d6c7 ) > a? !b? c? ( mirror://binhost/063a14d6c7 ) > a? b? !c? ( mirror://binhost/063a14d6c7 ) > a? b? c? ( mirror://binhost/063a14d6c7 ) > ) > > In order to ensure that an ebin is not selected for a USE combination > that has not been built yet, combinations for existing builds will be > listed in REQUIRED_USE, like this: > > REQUIRED_USE=" > || ( > ( !a !b !c ) > ( !a !b c ) > ( !a b !c ) > ( !a b c ) > ( a !b !c ) > ( a !b c ) > ( a b !c ) > ( a b c ) > ) > " In https://bugs.gentoo.org/772380 I'm planning to implement a script that will take an existing $PKGDIR as input, and generate an "ebin" binhost as output. It will automatically split out pre-built content bundles for installsources and splitdebug as shown above. If there is more than one build for a particular package version and USE combination, then the script will choose the package instance with latest BUILD_TIME metadata (in alignment with FEATURES=binpkg-multi-instance). -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] New project: binhost
On 2/23/21 12:33 PM, Zac Medico wrote: > On 2/23/21 12:05 PM, Zac Medico wrote: >> On 2/23/21 11:46 AM, Zac Medico wrote: >>> On 2/20/21 8:17 PM, Zac Medico wrote: IUSE_RUNTIME will obviously introduce conditionals in binary package dependencies, but we should welcome these conditionals because they will provide useful flexibility. I think IUSE_RUNTIME will be a very nice feature to have for project binhost, since it will allow binary package dependencies to behave with flexibility that more closely resembles the flexibility of ebuild dependencies. >>> >>> We can borrow paludis's notion of pbins [1] to generate ebuilds which >>> install pre-built content, and the generated ebuilds could have USE >>> flags that behave similarly to IUSE_RUNTIME in the sense that changes to >>> USE flags will result in different builds of pre-built content being >>> installed. A content-hash distfiles layout [2] could serve as a >>> convenient way to store separate builds of pre-built content for >>> multiple combinations of USE flags, and a generated ebuild would index >>> the build by USE flag combination. >>> >>> Also, for the generated ebuilds, we can generate USE flags to include >>> separate SRC_URI downloads for pre-built content to support things like >>> FEATURES=split-debug and FEATURES=install-sources. >> >> Note that all of this can use existing EAPI features, since everything new >> would be implemented in an ebuild generator that generates a single >> ebuild to index pre-built content from multiple binary package builds. >> >>> [1] https://paludis.exherbo.org/overview/pbins.html >>> [2] https://bugs.gentoo.org/756778 >> > > For generated ebuilds, we'll have a choice to model things as USE flags > or sub-packages. For example, content from FEATURES=split-debug and > FEATURES=install-sources content might make more sense to model as > sub-packages than USE flags. It makes more sense to generate a > sub-package when there is a need for the sub-package to have USE flags. > For example, a split-debug sub-package can have USE flags which index > pre-built content from builds for multiple USE flag combinations. > Similar USE flags could be useful for an install-sources sub-package if > the source code has patches which are conditional on USE flags. Since the generated ebuilds are inspired by pbins, we might call them "ebins". Once we have designed an ebin generation process that we're happy with, we should consider making it part of an EAPI, so that package managers can generate "ebins" on request. The EAPI should include ways to split out files and distribute them separately based on USE flags and/or sub-packages, so that binhost users can easily filter which files are installed based on USE configuration. We can automatically map user's splitdebug and installsources FEATURES settings into USE settings, in the same way that FEATURES=test automatically maps to USE=test. Each generated ebuild (ebin) will use its SRC_URI metadata to index builds for each USE flag combination for which a build exists. For example, for 3 USE flags, up to 8 combinations will be indexed: IUSE="a b c installsources splitdebug" SRC_URI=" !a? !b? !c? ( mirror://binhost/24fe6bd377 ) !a? !b? c? ( mirror://binhost/fbe14cbb02 ) !a? b? !c? ( mirror://binhost/1dfff1f2ac ) !a? b? c? ( mirror://binhost/ae60f2940d ) a? !b? !c? ( mirror://binhost/2976e1acc0 ) a? !b? c? ( mirror://binhost/f4809db70c ) a? b? !c? ( mirror://binhost/ecd08466cf ) a? b? c? ( mirror://binhost/0c00f33b2e ) installsources? ( !a? !b? !c? ( mirror://binhost/063a14d6c7 ) !a? !b? c? ( mirror://binhost/f67c311625 ) !a? b? !c? ( mirror://binhost/1dfff1f2ac ) !a? b? c? ( mirror://binhost/17a673e16a ) a? !b? !c? ( mirror://binhost/914d1cecfe ) a? !b? c? ( mirror://binhost/ca18d86a2b ) a? b? !c? ( mirror://binhost/6bce13471a ) a? b? c? ( mirror://binhost/3a6bdcd228 ) ) splitdebug? ( !a? !b? !c? ( mirror://binhost/29b2f38c41 ) !a? !b? c? ( mirror://binhost/8adc9bef51 ) !a? b? !c? ( mirror://binhost/954d2ce484 ) !a? b? c? ( mirror://binhost/32a614aaca ) a? !b? !c? ( mirror://binhost/3548a2302d ) a? !b? c? ( mirror://binhost/e0c02cdc88 ) a? b? !c? ( mirror://binhost/f9cbd3c181 ) a? b? c? ( mirror://binhost/31d4c03474 ) ) " For installsources, we can automate deduplication, so that we can distribute the same file content for multiple USE combinations when appropriate. If all of the combinations have identical content, then it will look like this: installsources? ( !a? !b? !c? ( mirror://binhost/063a14d6c7 ) !a? !b? c? ( mirror://binhost/063a14d6c7 ) !a? b? !c? ( mirror://binhost/063a14d6c7 ) !a? b? c? ( mirror://binhost/063a14d6c7 ) a? !b? !c? ( mirror://binhost/063a14d6c7 ) a? !b? c? ( mirror://binhost/063a14d6c7 ) a? b? !c? ( mirror://binhost/063a14d6c7 ) a? b? c? (
Re: [gentoo-dev] New project: binhost
On 2/23/21 12:05 PM, Zac Medico wrote: > On 2/23/21 11:46 AM, Zac Medico wrote: >> On 2/20/21 8:17 PM, Zac Medico wrote: >>> IUSE_RUNTIME will obviously introduce conditionals in binary package >>> dependencies, but we should welcome these conditionals because they will >>> provide useful flexibility. >>> >>> I think IUSE_RUNTIME will be a very nice feature to have for project >>> binhost, since it will allow binary package dependencies to behave with >>> flexibility that more closely resembles the flexibility of ebuild >>> dependencies. >> >> We can borrow paludis's notion of pbins [1] to generate ebuilds which >> install pre-built content, and the generated ebuilds could have USE >> flags that behave similarly to IUSE_RUNTIME in the sense that changes to >> USE flags will result in different builds of pre-built content being >> installed. A content-hash distfiles layout [2] could serve as a >> convenient way to store separate builds of pre-built content for >> multiple combinations of USE flags, and a generated ebuild would index >> the build by USE flag combination. >> >> Also, for the generated ebuilds, we can generate USE flags to include >> separate SRC_URI downloads for pre-built content to support things like >> FEATURES=split-debug and FEATURES=install-sources. > > Note that all of this can use existing EAPI features, since everything new > would be implemented in an ebuild generator that generates a single > ebuild to index pre-built content from multiple binary package builds. > >> [1] https://paludis.exherbo.org/overview/pbins.html >> [2] https://bugs.gentoo.org/756778 > For generated ebuilds, we'll have a choice to model things as USE flags or sub-packages. For example, content from FEATURES=split-debug and FEATURES=install-sources content might make more sense to model as sub-packages than USE flags. It makes more sense to generate a sub-package when there is a need for the sub-package to have USE flags. For example, a split-debug sub-package can have USE flags which index pre-built content from builds for multiple USE flag combinations. Similar USE flags could be useful for an install-sources sub-package if the source code has patches which are conditional on USE flags. -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] New project: binhost
On 2/23/21 11:46 AM, Zac Medico wrote: > On 2/20/21 8:17 PM, Zac Medico wrote: >> On 2/13/21 4:53 PM, Zac Medico wrote: >>> On 2/13/21 4:37 PM, Zac Medico wrote: On 2/11/21 1:17 AM, Michał Górny wrote: > On Wed, 2021-02-10 at 19:51 +0100, Lars Wendler wrote: >> On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote: >> >>> Hi all, >>> >>> I'm announcing a new project here - "binhost" >>> >>> "The Gentoo Binhost project aims to provide readily installable, >>> precompiled packages for a subset of configurations, via central >>> binary package hosting. Currently we are still in the conceptual >>> planning stage. " >>> >>> https://wiki.gentoo.org/wiki/Project:Binhost >>> >>> If you're interested in helping out, feel free to add yourself on the >>> wiki page. >>> >>> Note that I see actually *building* the packages not as the central >>> point of the project (that could be e.g. a side effect of a >>> tinderbox). I'm more concerned about >>> * what configurations should we use >>> * what portage features are still needed or need improvements (e.g. >>> binpkg signing and verification) >>> * how should hosting look like >>> * and how we can test this on a limited scale before it goes "into >>> production" >>> * ... >>> >>> Comments, ideas, flamebaits? :D >>> >>> Cheers, >>> Andreas >>> >> >> It would be great to improve portage speed with handling binpkgs. I >> already have my own binhost for a couple of Gentoo systems and even >> though these systems don't have to compile anything themselves, >> installing ~100 to ~200 binpkgs takes way more than an hour of >> installation time. Arch Linux' pacman only takes a fraction of this >> time for the very same task. >> I know that I compare apples with pears here but even reducing the >> current portage time by 50% would be a huge improvement. > > Is that really a problem? For me, Portage takes about an hour just to > do the dependency processing these days. In fact, building from sources > is now faster than dependency calculations. The ratio of these times is dependent on the complexity of the dependencies involved, and so is the answer to your question. >>> >>> Also, in the context of binary packages, dependencies calculations are >>> generally simpler for binary packages because their USE conditionals and >>> slot-operator := dependencies are frozen in a particular state. This >>> dramatically reduces the search space involved in dependency calculation. >> >> IUSE_RUNTIME will obviously introduce conditionals in binary package >> dependencies, but we should welcome these conditionals because they will >> provide useful flexibility. >> >> I think IUSE_RUNTIME will be a very nice feature to have for project >> binhost, since it will allow binary package dependencies to behave with >> flexibility that more closely resembles the flexibility of ebuild >> dependencies. > > We can borrow paludis's notion of pbins [1] to generate ebuilds which > install pre-built content, and the generated ebuilds could have USE > flags that behave similarly to IUSE_RUNTIME in the sense that changes to > USE flags will result in different builds of pre-built content being > installed. A content-hash distfiles layout [2] could serve as a > convenient way to store separate builds of pre-built content for > multiple combinations of USE flags, and a generated ebuild would index > the build by USE flag combination. > > Also, for the generated ebuilds, we can generate USE flags to include > separate SRC_URI downloads for pre-built content to support things like > FEATURES=split-debug and FEATURES=install-sources. Note that all of this can existing EAPI features, since everything new would be implemented in an ebuild generator that generates a single ebuild to index pre-built content from multiple binary package builds. > [1] https://paludis.exherbo.org/overview/pbins.html > [2] https://bugs.gentoo.org/756778 -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] New project: binhost
On 2/13/21 5:51 PM, Zac Medico wrote: > On 2/10/21 11:11 AM, Rich Freeman wrote: >> On Wed, Feb 10, 2021 at 12:57 PM Andreas K. Hüttel >> wrote: >>> >>> * what portage features are still needed or need improvements (e.g. binpkg >>> signing and verification) >>> * how should hosting look like >> >> Some ideas for portage enhancements: >> >> 1. Ability to fetch binary packages from some kind of repo. > > The old PORTAGE_BINHOST functionality has been replaced with a > binrepos.conf file that's very similar to repos.conf: > > https://bugs.gentoo.org/668334 > > It doesn't have explicit support for multiple local binary package > repositories yet, but somebody got it working with src-uri set to a > file:/ uri as described in comments on this bug: > > https://bugs.gentoo.org/768957 > >> 2. Ability to have multiple binary packages co-exist in a repo (local >> or remote) with different build attributes (arch, USE, CFLAGS, >> DEPENDS, whatever). > > We can now enable FEATURES=binpkg-multi-instance by default now that > this bug is fixed: > > https://bugs.gentoo.org/571126 > >> 3. Ability to pick the most appropriate binary packages to use based >> on user preferences (with a mix of hard and soft preferences). > > Current package selection logic for binary packages is basically the > same as for ebuilds. These are the notable differences: > > 1) Binary packages are sorted in descending order by (version, mtime), > so then most recent builds are preferred when the versions are identical. > > 2) The --binpkg-respect-use option rejects binary packages what would > need to be rebuilt in order to match local USE settings. > >> One idea I've had around how #2-3 might be implemented is: >> 1. Binary packages already contain data on how they were built (USE >> flags, dependencies, etc). Place this in a file using a deterministic >> sorting/etc order so that two builds with the same settings will have >> the same results. > > This would only be needed to multi-profile binhosts that provide a > variety of configurations for the same package. > > Features like this are not necessary if the binhost only intends to > provide packages for a single profile. > >> 2. Generate a hash of the file contents - this can go in the filename >> so that the file can co-exist with other files, and be located >> assuming you have a full matching set of metadata. > > For FEATURES=binpkg-multi-instance we currently use an integer BUILD_ID > ensure that file names are unique. > >> 3. Start dropping attributes from the file based on a list of >> priorities and generate additional hashes. Create symlinked files to >> the original file using these hashes (overwriting or not existing >> symlinks based on policy). This allows the binary package to be found >> using either an exact set of attributes or a subset of higher-priority >> attributes. This is analogous to shared object symlinking. >> 4. The package manager will look for a binary package first using the >> user's full config, and then by dropping optional elements of the >> config (so maybe it does the search without CFLAGs, then without USE >> flags). Eventually it aborts based on user prefs (maybe the user only >> wants an exact match, or is willing to accept alternate CFLAGs but not >> USE flags, or maybe anything for the arch is selected> 5. As always the >> final selected binary package still gets evaluated >> like any other binary package to ensure it is usable. >> >> Such a system can identify whether a potentially usable file exists >> using only filename, cutting down on fetching. In the interests of >> avoiding useless fetches we would only carry step 3 reasonably far - >> packages would have to match based on architecture and any dynamic >> linking requirements. So we wouldn't generate hashes that didn't >> include at least those minimums, and the package manager wouldn't >> search for them. >> >> Obviously you could do more (if you have 5 combinations of use flags, >> look for the set that matches most closely). That couldn't be done >> using hashes alone in an efficient way. You could have a small >> manifest file alongside the binary package that could be fetched >> separately if the package manager wants to narrow things down and >> fetch a few of those to narrow it down further. > > All of the above is oriented toward multi-profile binhosts, so we'll > have to do a cost/benefit analysis to determine whether it's worth the > effort to introduce the complexity that multi-profile binhosts add. This idea to borrow paludis's notion of pbins could work well for a multi-profile binhost: https://archives.gentoo.org/gentoo-dev/message/1aea0de0ff588c67bf652f4c1f8ef304 -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] New project: binhost
On 2/20/21 8:17 PM, Zac Medico wrote: > On 2/13/21 4:53 PM, Zac Medico wrote: >> On 2/13/21 4:37 PM, Zac Medico wrote: >>> On 2/11/21 1:17 AM, Michał Górny wrote: On Wed, 2021-02-10 at 19:51 +0100, Lars Wendler wrote: > On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote: > >> Hi all, >> >> I'm announcing a new project here - "binhost" >> >> "The Gentoo Binhost project aims to provide readily installable, >> precompiled packages for a subset of configurations, via central >> binary package hosting. Currently we are still in the conceptual >> planning stage. " >> >> https://wiki.gentoo.org/wiki/Project:Binhost >> >> If you're interested in helping out, feel free to add yourself on the >> wiki page. >> >> Note that I see actually *building* the packages not as the central >> point of the project (that could be e.g. a side effect of a >> tinderbox). I'm more concerned about >> * what configurations should we use >> * what portage features are still needed or need improvements (e.g. >> binpkg signing and verification) >> * how should hosting look like >> * and how we can test this on a limited scale before it goes "into >> production" >> * ... >> >> Comments, ideas, flamebaits? :D >> >> Cheers, >> Andreas >> > > It would be great to improve portage speed with handling binpkgs. I > already have my own binhost for a couple of Gentoo systems and even > though these systems don't have to compile anything themselves, > installing ~100 to ~200 binpkgs takes way more than an hour of > installation time. Arch Linux' pacman only takes a fraction of this > time for the very same task. > I know that I compare apples with pears here but even reducing the > current portage time by 50% would be a huge improvement. Is that really a problem? For me, Portage takes about an hour just to do the dependency processing these days. In fact, building from sources is now faster than dependency calculations. >>> >>> The ratio of these times is dependent on the complexity of the >>> dependencies involved, and so is the answer to your question. >> >> Also, in the context of binary packages, dependencies calculations are >> generally simpler for binary packages because their USE conditionals and >> slot-operator := dependencies are frozen in a particular state. This >> dramatically reduces the search space involved in dependency calculation. > > IUSE_RUNTIME will obviously introduce conditionals in binary package > dependencies, but we should welcome these conditionals because they will > provide useful flexibility. > > I think IUSE_RUNTIME will be a very nice feature to have for project > binhost, since it will allow binary package dependencies to behave with > flexibility that more closely resembles the flexibility of ebuild > dependencies. We can borrow paludis's notion of pbins [1] to generate ebuilds which install pre-built content, and the generated ebuilds could have USE flags that behave similarly to IUSE_RUNTIME in the sense that changes to USE flags will result in different builds of pre-built content being installed. A content-hash distfiles layout [2] could serve as a convenient way to store separate builds of pre-built content for multiple combinations of USE flags, and a generated ebuild would index the build by USE flag combination. Also, for the generated ebuilds, we can generate USE flags to include separate SRC_URI downloads for pre-built content to support things like FEATURES=split-debug and FEATURES=install-sources. [1] https://paludis.exherbo.org/overview/pbins.html [2] https://bugs.gentoo.org/756778 -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] New project: binhost
Replying to a random post here- I'm sorry, briefly after I started this thread real life got in the way and a lot of other things became more urgent. As soon as I have time (next weekend?) I'll reply to the many e-mails here. Thanks -A -- Andreas K. Hüttel dilfri...@gentoo.org Gentoo Linux developer (council, toolchain, base-system, perl, libreoffice) signature.asc Description: This is a digitally signed message part.
Re: [gentoo-dev] New project: binhost
On 2/13/21 4:53 PM, Zac Medico wrote: > On 2/13/21 4:37 PM, Zac Medico wrote: >> On 2/11/21 1:17 AM, Michał Górny wrote: >>> On Wed, 2021-02-10 at 19:51 +0100, Lars Wendler wrote: On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote: > Hi all, > > I'm announcing a new project here - "binhost" > > "The Gentoo Binhost project aims to provide readily installable, > precompiled packages for a subset of configurations, via central > binary package hosting. Currently we are still in the conceptual > planning stage. " > > https://wiki.gentoo.org/wiki/Project:Binhost > > If you're interested in helping out, feel free to add yourself on the > wiki page. > > Note that I see actually *building* the packages not as the central > point of the project (that could be e.g. a side effect of a > tinderbox). I'm more concerned about > * what configurations should we use > * what portage features are still needed or need improvements (e.g. > binpkg signing and verification) > * how should hosting look like > * and how we can test this on a limited scale before it goes "into > production" > * ... > > Comments, ideas, flamebaits? :D > > Cheers, > Andreas > It would be great to improve portage speed with handling binpkgs. I already have my own binhost for a couple of Gentoo systems and even though these systems don't have to compile anything themselves, installing ~100 to ~200 binpkgs takes way more than an hour of installation time. Arch Linux' pacman only takes a fraction of this time for the very same task. I know that I compare apples with pears here but even reducing the current portage time by 50% would be a huge improvement. >>> >>> Is that really a problem? For me, Portage takes about an hour just to >>> do the dependency processing these days. In fact, building from sources >>> is now faster than dependency calculations. >> >> The ratio of these times is dependent on the complexity of the >> dependencies involved, and so is the answer to your question. > > Also, in the context of binary packages, dependencies calculations are > generally simpler for binary packages because their USE conditionals and > slot-operator := dependencies are frozen in a particular state. This > dramatically reduces the search space involved in dependency calculation. IUSE_RUNTIME will obviously introduce conditionals in binary package dependencies, but we should welcome these conditionals because they will provide useful flexibility. I think IUSE_RUNTIME will be a very nice feature to have for project binhost, since it will allow binary package dependencies to behave with flexibility that more closely resembles the flexibility of ebuild dependencies. -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] New project: binhost
Hi, > Binpkg performance is acceptable albeit not blazing fast for machines > with 500-800 packages (usually server) while for desktops which easily > have 2000 packages the time to update can be hours. I take from this that the binpkg process really should be optimized somehow. Without looking at the code it looks like binpkg is significantly less efficient than without. I think Michał also hinted at this. Essentially, from the sounds of the above I suspect we're at "we're probably burning more CPU on calculating the dependency tree than merely recompiling". Which to me defeats the purpose. This may be a single-core use vs multiple cores though, so whilst it may take similar amount of time it's possible that we end up consuming less energy overall. Kind Regards, Jaco
Re: [gentoo-dev] New project: binhost
Il giorno mer 10 feb 2021 alle ore 19:51 Lars Wendler < polynomia...@gentoo.org> ha scritto: > On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote: > > >Hi all, > > > >I'm announcing a new project here - "binhost" > > > >"The Gentoo Binhost project aims to provide readily installable, > >precompiled packages for a subset of configurations, via central > >binary package hosting. Currently we are still in the conceptual > >planning stage. " > > > >https://wiki.gentoo.org/wiki/Project:Binhost > > > >If you're interested in helping out, feel free to add yourself on the > >wiki page. > > > >Note that I see actually *building* the packages not as the central > >point of the project (that could be e.g. a side effect of a > >tinderbox). I'm more concerned about > >* what configurations should we use > >* what portage features are still needed or need improvements (e.g. > >binpkg signing and verification) > >* how should hosting look like > >* and how we can test this on a limited scale before it goes "into > >production" > >* ... > > > >Comments, ideas, flamebaits? :D > > > >Cheers, > >Andreas > > > > It would be great to improve portage speed with handling binpkgs. I > already have my own binhost for a couple of Gentoo systems and even > though these systems don't have to compile anything themselves, > installing ~100 to ~200 binpkgs takes way more than an hour of > installation time. Arch Linux' pacman only takes a fraction of this > time for the very same task. > I know that I compare apples with pears here but even reducing the > current portage time by 50% would be a huge improvement. > > Agreed, nowadays I do use Gentoo in two ways: - From binpkgs usually for baremetal, server or desktop - condensing part of the system in a squashfs image, usually for containers Binpkg performance is acceptable albeit not blazing fast for machines with 500-800 packages (usually server) while for desktops which easily have 2000 packages the time to update can be hours. While we are here the squashfs images way to distribute is wonderful and handy, except that it's read-only and managing /etc is more challenging without the commodity of etc-update/dispatch-conf, would be nicer to have a comparable tool to be used for this. Suggestions about the implementation well accepted Cheers, Francesco (vivo) Riosa
Re: [gentoo-dev] New project: binhost
Il giorno mer 10 feb 2021 alle ore 20:15 Andreas K. Hüttel < dilfri...@gentoo.org> ha scritto: > > Some ideas for portage enhancements: > > > > 1. Ability to fetch binary packages from some kind of repo. > > 2. Ability to have multiple binary packages co-exist in a repo (local > > or remote) with different build attributes (arch, USE, CFLAGS, > > DEPENDS, whatever). > > 3. Ability to pick the most appropriate binary packages to use based > > on user preferences (with a mix of hard and soft preferences). > > The more definite answer should come from Zac, but I think a good part of > this > is already implemented. :) > > kind of, from make.conf manpage: binpkg-multi-instance Enable support for multiple binary package in‐ stances per ebuild. Having multiple instances is useful for a number of purposes, such as retaining builds that were built with different USE flags or linked against different versions of libraries. The location of any particular package within PKGDIR can be expressed as follows: ${PKGDIR}/${CATEGORY}/${PN}/${PF}-${BUILD_ID}.xpak The build-id starts at 1 for the first build of a particular ebuild, and is incremented by 1 for each new build. It is possible to share a writable PKGDIR over NFS, and locking ensures that each package added to PKGDIR will have a unique build-id. It is not necessary to migrate an exist‐ ing PKGDIR to the new layout, since portage is ca‐ pable of working with a mixed PKGDIR layout, where packages using the old layout are allowed to re‐ main in place. The new PKGDIR layout is backward-compatible with binhost clients running older portage, since the file format is identical, the per-package PATH at‐ tribute in the 'Packages' index directs them to download the file from the correct URI, and they automatically use BUILD_TIME metadata to select the latest builds. There is currently no automated way to prune old builds from PKGDIR, although it is possible to re‐ move packages manually, and then run 'emaint --fix binhost' to update the ${PKGDIR}/Packages index.
Re: [gentoo-dev] New project: binhost
On Sat, Feb 13, 2021 at 8:51 PM Zac Medico wrote: > > > 2. Generate a hash of the file contents - this can go in the filename > > so that the file can co-exist with other files, and be located > > assuming you have a full matching set of metadata. > > For FEATURES=binpkg-multi-instance we currently use an integer BUILD_ID > ensure that file names are unique. > > > 3. Start dropping attributes from the file based on a list of > > priorities and generate additional hashes. Create symlinked files to > > the original file using these hashes (overwriting or not existing > > symlinks based on policy). This allows the binary package to be found > > using either an exact set of attributes or a subset of higher-priority > > attributes. This is analogous to shared object symlinking. > > 4. The package manager will look for a binary package first using the > > user's full config, and then by dropping optional elements of the > > config (so maybe it does the search without CFLAGs, then without USE > > flags). Eventually it aborts based on user prefs (maybe the user only > > wants an exact match, or is willing to accept alternate CFLAGs but not > > USE flags, or maybe anything for the arch is selected> 5. As always the > > final selected binary package still gets evaluated > > like any other binary package to ensure it is usable. > > > > Such a system can identify whether a potentially usable file exists > > using only filename, cutting down on fetching. In the interests of > > avoiding useless fetches we would only carry step 3 reasonably far - > > packages would have to match based on architecture and any dynamic > > linking requirements. So we wouldn't generate hashes that didn't > > include at least those minimums, and the package manager wouldn't > > search for them. > > > > Obviously you could do more (if you have 5 combinations of use flags, > > look for the set that matches most closely). That couldn't be done > > using hashes alone in an efficient way. You could have a small > > manifest file alongside the binary package that could be fetched > > separately if the package manager wants to narrow things down and > > fetch a few of those to narrow it down further. > > All of the above is oriented toward multi-profile binhosts, so we'll > have to do a cost/benefit analysis to determine whether it's worth the > effort to introduce the complexity that multi-profile binhosts add. The hash label on the filenames was also considered around multi-profiles. I figured that if you're going to be building variants of packages you'd want to parallelize and hashes work better for that. Plus at least in concept you could potentially identify and fetch files by hash using info already in the local repo without having to sync additional metadata from the binhost. User-contributed binaries would also work better in such a world though for obvious security issues that might just take the form of local user-generated repos (allowing users to supplement the upstream repo with local builds for a cluster, without having to mirror/reporoduce the entire upstream. I do get that multi-profiles aren't entirely an essential feature, but when you consider stuff like X11 support or stable/unstable it seems like we're probably going to have to provide at least a few variants on packages for this to be practical. You could just put each profile in a separate repo, but then anything that doesn't actually change across profiles gets built multiple times. The hash-based solution is also a form of deduping. But, hey, it is great to see anything like this being done at all. Walking before running isn't a bad thing! -- Rich
Re: [gentoo-dev] New project: binhost
On 2/10/21 11:11 AM, Rich Freeman wrote: > On Wed, Feb 10, 2021 at 12:57 PM Andreas K. Hüttel > wrote: >> >> * what portage features are still needed or need improvements (e.g. binpkg >> signing and verification) >> * how should hosting look like > > Some ideas for portage enhancements: > > 1. Ability to fetch binary packages from some kind of repo. The old PORTAGE_BINHOST functionality has been replaced with a binrepos.conf file that's very similar to repos.conf: https://bugs.gentoo.org/668334 It doesn't have explicit support for multiple local binary package repositories yet, but somebody got it working with src-uri set to a file:/ uri as described in comments on this bug: https://bugs.gentoo.org/768957 > 2. Ability to have multiple binary packages co-exist in a repo (local > or remote) with different build attributes (arch, USE, CFLAGS, > DEPENDS, whatever). We can now enable FEATURES=binpkg-multi-instance by default now that this bug is fixed: https://bugs.gentoo.org/571126 > 3. Ability to pick the most appropriate binary packages to use based > on user preferences (with a mix of hard and soft preferences). Current package selection logic for binary packages is basically the same as for ebuilds. These are the notable differences: 1) Binary packages are sorted in descending order by (version, mtime), so then most recent builds are preferred when the versions are identical. 2) The --binpkg-respect-use option rejects binary packages what would need to be rebuilt in order to match local USE settings. > One idea I've had around how #2-3 might be implemented is: > 1. Binary packages already contain data on how they were built (USE > flags, dependencies, etc). Place this in a file using a deterministic > sorting/etc order so that two builds with the same settings will have > the same results. This would only be needed to multi-profile binhosts that provide a variety of configurations for the same package. Features like this are not necessary if the binhost only intends to provide packages for a single profile. > 2. Generate a hash of the file contents - this can go in the filename > so that the file can co-exist with other files, and be located > assuming you have a full matching set of metadata. For FEATURES=binpkg-multi-instance we currently use an integer BUILD_ID ensure that file names are unique. > 3. Start dropping attributes from the file based on a list of > priorities and generate additional hashes. Create symlinked files to > the original file using these hashes (overwriting or not existing > symlinks based on policy). This allows the binary package to be found > using either an exact set of attributes or a subset of higher-priority > attributes. This is analogous to shared object symlinking. > 4. The package manager will look for a binary package first using the > user's full config, and then by dropping optional elements of the > config (so maybe it does the search without CFLAGs, then without USE > flags). Eventually it aborts based on user prefs (maybe the user only > wants an exact match, or is willing to accept alternate CFLAGs but not > USE flags, or maybe anything for the arch is selected> 5. As always the > final selected binary package still gets evaluated > like any other binary package to ensure it is usable. > > Such a system can identify whether a potentially usable file exists > using only filename, cutting down on fetching. In the interests of > avoiding useless fetches we would only carry step 3 reasonably far - > packages would have to match based on architecture and any dynamic > linking requirements. So we wouldn't generate hashes that didn't > include at least those minimums, and the package manager wouldn't > search for them. > > Obviously you could do more (if you have 5 combinations of use flags, > look for the set that matches most closely). That couldn't be done > using hashes alone in an efficient way. You could have a small > manifest file alongside the binary package that could be fetched > separately if the package manager wants to narrow things down and > fetch a few of those to narrow it down further. All of the above is oriented toward multi-profile binhosts, so we'll have to do a cost/benefit analysis to determine whether it's worth the effort to introduce the complexity that multi-profile binhosts add. > Or you could skip the hash searching and just fetch all the manifests > for a particular package/arch and just search all of those, but that > is more data to transfer just to do a query. A metadata cache of some > kind of might be another solution. Content hashes would probably > still be useful just to allow co-existence of alternate builds. This also relates to the centralized Packages file that's currently used to distribute the package metadata for all packages in a binhost. We can make it scale better if we split out a separate index per package, not unlike a pypi simple index: https://pypi.org/simple/ --
Re: [gentoo-dev] New project: binhost
On 2/13/21 4:37 PM, Zac Medico wrote: > On 2/11/21 1:17 AM, Michał Górny wrote: >> On Wed, 2021-02-10 at 19:51 +0100, Lars Wendler wrote: >>> On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote: >>> Hi all, I'm announcing a new project here - "binhost" "The Gentoo Binhost project aims to provide readily installable, precompiled packages for a subset of configurations, via central binary package hosting. Currently we are still in the conceptual planning stage. " https://wiki.gentoo.org/wiki/Project:Binhost If you're interested in helping out, feel free to add yourself on the wiki page. Note that I see actually *building* the packages not as the central point of the project (that could be e.g. a side effect of a tinderbox). I'm more concerned about * what configurations should we use * what portage features are still needed or need improvements (e.g. binpkg signing and verification) * how should hosting look like * and how we can test this on a limited scale before it goes "into production" * ... Comments, ideas, flamebaits? :D Cheers, Andreas >>> >>> It would be great to improve portage speed with handling binpkgs. I >>> already have my own binhost for a couple of Gentoo systems and even >>> though these systems don't have to compile anything themselves, >>> installing ~100 to ~200 binpkgs takes way more than an hour of >>> installation time. Arch Linux' pacman only takes a fraction of this >>> time for the very same task. >>> I know that I compare apples with pears here but even reducing the >>> current portage time by 50% would be a huge improvement. >> >> Is that really a problem? For me, Portage takes about an hour just to >> do the dependency processing these days. In fact, building from sources >> is now faster than dependency calculations. > > The ratio of these times is dependent on the complexity of the > dependencies involved, and so is the answer to your question. Also, in the context of binary packages, dependencies calculations are generally simpler for binary packages because their USE conditionals and slot-operator := dependencies are frozen in a particular state. This dramatically reduces the search space involved in dependency calculation. -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] New project: binhost
On 2/11/21 1:17 AM, Michał Górny wrote: > On Wed, 2021-02-10 at 19:51 +0100, Lars Wendler wrote: >> On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote: >> >>> Hi all, >>> >>> I'm announcing a new project here - "binhost" >>> >>> "The Gentoo Binhost project aims to provide readily installable, >>> precompiled packages for a subset of configurations, via central >>> binary package hosting. Currently we are still in the conceptual >>> planning stage. " >>> >>> https://wiki.gentoo.org/wiki/Project:Binhost >>> >>> If you're interested in helping out, feel free to add yourself on the >>> wiki page. >>> >>> Note that I see actually *building* the packages not as the central >>> point of the project (that could be e.g. a side effect of a >>> tinderbox). I'm more concerned about >>> * what configurations should we use >>> * what portage features are still needed or need improvements (e.g. >>> binpkg signing and verification) >>> * how should hosting look like >>> * and how we can test this on a limited scale before it goes "into >>> production" >>> * ... >>> >>> Comments, ideas, flamebaits? :D >>> >>> Cheers, >>> Andreas >>> >> >> It would be great to improve portage speed with handling binpkgs. I >> already have my own binhost for a couple of Gentoo systems and even >> though these systems don't have to compile anything themselves, >> installing ~100 to ~200 binpkgs takes way more than an hour of >> installation time. Arch Linux' pacman only takes a fraction of this >> time for the very same task. >> I know that I compare apples with pears here but even reducing the >> current portage time by 50% would be a huge improvement. > > Is that really a problem? For me, Portage takes about an hour just to > do the dependency processing these days. In fact, building from sources > is now faster than dependency calculations. The ratio of these times is dependent on the complexity of the dependencies involved, and so is the answer to your question. -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] New project: binhost
On 2/10/21 10:51 AM, Lars Wendler wrote: > On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote: > >> Hi all, >> >> I'm announcing a new project here - "binhost" >> >> "The Gentoo Binhost project aims to provide readily installable, >> precompiled packages for a subset of configurations, via central >> binary package hosting. Currently we are still in the conceptual >> planning stage. " >> >> https://wiki.gentoo.org/wiki/Project:Binhost >> >> If you're interested in helping out, feel free to add yourself on the >> wiki page. >> >> Note that I see actually *building* the packages not as the central >> point of the project (that could be e.g. a side effect of a >> tinderbox). I'm more concerned about >> * what configurations should we use >> * what portage features are still needed or need improvements (e.g. >> binpkg signing and verification) >> * how should hosting look like >> * and how we can test this on a limited scale before it goes "into >> production" >> * ... >> >> Comments, ideas, flamebaits? :D >> >> Cheers, >> Andreas >> > > It would be great to improve portage speed with handling binpkgs. I > already have my own binhost for a couple of Gentoo systems and even > though these systems don't have to compile anything themselves, > installing ~100 to ~200 binpkgs takes way more than an hour of > installation time. Arch Linux' pacman only takes a fraction of this > time for the very same task. > I know that I compare apples with pears here but even reducing the > current portage time by 50% would be a huge improvement. In order to maximize throughput, we have FEATURES="parallel-install" that for example allows one package's files to be merged while a pkg_{pre,post}{inst,rm} phase is running for an unrelated package that is in the merge queue. There's also FEATURES="-ebuild-locks" which allows privileged pkg_{pre,post}{inst,rm} phases to run concurrently with privileged phases of unrelated packages in the merge queue. Ultimately, pkg_{pre,post}{inst,rm} phases could be the limiting factor here. I portage, we should eliminate calls to these phases when DEFINED_PHASES metadata indicated that they're not defined. Also, FEATURES=preserve-libs introduces considerable overhead because it regenerates LinkageMapELF data structures for every merge. It would be much more efficient if it could incrementally update these data structures. -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] New project: binhost
dilfridge: > Hi all, > > I'm announcing a new project here - "binhost" > > "The Gentoo Binhost project aims to provide readily installable, precompiled > packages for a subset of configurations, via central binary package hosting. > Currently we are still in the conceptual planning stage. " > > https://wiki.gentoo.org/wiki/Project:Binhost > > If you're interested in helping out, feel free to add yourself on the wiki > page. > > Note that I see actually *building* the packages not as the central point of > the project (that could be e.g. a side effect of a tinderbox). I'm more > concerned about > * what configurations should we use > * what portage features are still needed or need improvements (e.g. binpkg > signing and verification) > * how should hosting look like > * and how we can test this on a limited scale before it goes "into production" > * ... > > Comments, ideas, flamebaits? :D What's with CFLAGS? I haven't been working with pre-built binary packages yet. But I imagine something flexible, between distant compiler and a server keeping the binaries. For my cases, from Raspberry Pis up to AMD Ryzens, I love to have this server to just receive the requests what and how to compile and keep exactly the packages which are requested so it can be distributed if requested again. Thanks
Re: [gentoo-dev] New project: binhost
On Wed, Feb 10, 2021 at 11:58 AM Andreas K. Hüttel wrote: > Hi all, > > I'm announcing a new project here - "binhost" > > "The Gentoo Binhost project aims to provide readily installable, > precompiled > packages for a subset of configurations, via central binary package > hosting. > Currently we are still in the conceptual planning stage. " > > https://wiki.gentoo.org/wiki/Project:Binhost > > If you're interested in helping out, feel free to add yourself on the wiki > page. > > Note that I see actually *building* the packages not as the central point > of > the project (that could be e.g. a side effect of a tinderbox). I'm more > concerned about > * what configurations should we use > * what portage features are still needed or need improvements (e.g. binpkg > signing and verification) > * how should hosting look like > * and how we can test this on a limited scale before it goes "into > production" > * ... > > Comments, ideas, flamebaits? :D > > Cheers, > Andreas > > -- > Andreas K. Hüttel > dilfri...@gentoo.org > Gentoo Linux developer > (council, qa, toolchain, base-system, perl, libreoffice) What level of collaboration can be anticipated with non-official overlays? E.g. Embedded systems such as Raspberry Pi, are particularly well positioned to benefit from binhosting. There is a project (which I am contributing to) which aims to provide an installable image and limited binhosting for the 64bit Raspberry Pi boards See this forum post for more details: https://forums.gentoo.org/viewtopic-t-1098232-postdays-0-postorder-asc-start-325.html
Re: [gentoo-dev] New project: binhost
Maybe as starter, if some people have any public binhosts available, some volunteers can try using those to build a (test) system to check how it fares? I have two public binhosts available (with very limited set of packages and very specific CPUs and very weak servers, so please don't crash them) https://bsd.ac/gentoo-binpkgs/ivybridge/Packages <--- a bit more up to date https://bsd.ac/gentoo-binpkgs/znver1/Packages <--- not that regularly updated The packages might be about week or two old, am going to setup weekly builds and uploads. Might be better to start making something, and testing things out. If and when things break, we can see how to fix. I'm sure others have more extensive infrastructure and setups than my tiny VPS. If some other people want to give out information about their binhosts, at least some form of trial and error can be started. Thoughts? Aisha On 2/10/21 12:57 PM, Andreas K. Hüttel wrote: Hi all, I'm announcing a new project here - "binhost" "The Gentoo Binhost project aims to provide readily installable, precompiled packages for a subset of configurations, via central binary package hosting. Currently we are still in the conceptual planning stage. " https://wiki.gentoo.org/wiki/Project:Binhost If you're interested in helping out, feel free to add yourself on the wiki page. Note that I see actually *building* the packages not as the central point of the project (that could be e.g. a side effect of a tinderbox). I'm more concerned about * what configurations should we use * what portage features are still needed or need improvements (e.g. binpkg signing and verification) * how should hosting look like * and how we can test this on a limited scale before it goes "into production" * ... Comments, ideas, flamebaits? :D Cheers, Andreas
Re: [gentoo-dev] New project: binhost
* Andreas K. Hüttel schrieb am 10.02.21 um 12:57 Uhr: > Hi all, > > I'm announcing a new project here - "binhost" > > "The Gentoo Binhost project aims to provide readily installable, precompiled > packages for a subset of configurations, via central binary package hosting. > Currently we are still in the conceptual planning stage. " > > https://wiki.gentoo.org/wiki/Project:Binhost > > If you're interested in helping out, feel free to add yourself on the wiki > page. > > Note that I see actually *building* the packages not as the central point of > the project (that could be e.g. a side effect of a tinderbox). I'm more > concerned about > * what configurations should we use > * what portage features are still needed or need improvements (e.g. binpkg > signing and verification) > * how should hosting look like > * and how we can test this on a limited scale before it goes "into production" > * ... > > Comments, ideas, flamebaits? :D I like the idea very much. Lets make Gentoo suck less energy ;-) I have thought of some aspects in the past. My idea was to maybe have an additional term: 'flavours'. A flavour could be something like "most", "slim", "desktop" or "server" and each flavour may specify a subset of USE-flags to be enabled for packages or not. That way people could propably make most out of a public binhost if the choose one of the available flavours. That way you would not need to set USE flags for any single package to match a packge available on a binhost and a binhost does not need to have any combination of USE flags for any package in order to be most effective. With a flavour you could just express something like "I want the Plasma-Desktop profile with most features enabled". Or "I want the pasma-desktop as slim as possible". And we'd have some presets for that. But before that, I think we'd need some features in portage that make binpkgs trustworthy by adding signatures to the packages or at least to the Metadata. And Metadata should also be compressed or possibly be implemented in some incremental kind so that you will not have to download all the metadata everytime just because some random package changed that you do not even have installed or want to install. If I had more time I would definitely join the project. Maybe this is the case in 2-3 years. Cheers -Marc -- 0xCA3E7BF67F979BE5 - F7FB 78F7 7CC3 79F6 DF07 6E9E CA3E 7BF6 7F97 9BE5 signature.asc Description: PGP signature
Re: [gentoo-dev] New project: binhost
On 10/02/2021 18:57, Andreas K. Hüttel wrote: Hi all, I'm announcing a new project here - "binhost" "The Gentoo Binhost project aims to provide readily installable, precompiled packages for a subset of configurations, via central binary package hosting. Currently we are still in the conceptual planning stage. " https://wiki.gentoo.org/wiki/Project:Binhost If you're interested in helping out, feel free to add yourself on the wiki page. Note that I see actually *building* the packages not as the central point of the project (that could be e.g. a side effect of a tinderbox). I'm more concerned about * what configurations should we use Others have already suggested starting with a minimal set of flags or starting with the profiles, and then adding flags at request. I would like to suggest the opposite approach, start with binpkgs of packages which have all or most of the flags enabled, and then add more specific/minimal configurations of that package later. Because from an user perspective it is less of a problem to install a binpkg which has more features than you need, than it is to install a binpkg which is lacking a certain feature you need. Therefore, I would start with configurations of packages that have most/all things enabled, and thus are usable for the largest amount of people. This would pull in more dependencies, but for binpkgs this is less of a problem since they don't add compile time. * what portage features are still needed or need improvements (e.g. binpkg signing and verification) I think a bugtracker for this might be a good idea at some point. In general, I think that portage's binpkg support is very good already, there are however some things that could be improved. Bug https://bugs.gentoo.org/687668 comes to mind (and some other things that were already mentioned by others). The wiki guide on binpkgs[1] also mentions that: """ The support for multiple binary package servers is somewhat incomplete. If several servers serve a binary package for the same package version, then only the first one will be considered. This can be problematic when these binary packages differ in their USE variable configuration and the USE variable configuration of a later binary package would match the systems configuration. """ I don't know if this is still accurate, but if it is that would definitely be something that could use some improvement. [1] https://wiki.gentoo.org/wiki/Binary_package_guide * how should hosting look like * and how we can test this on a limited scale before it goes "into production" * ... Comments, ideas, flamebaits? :D Cheers, Andreas
Re: [gentoo-dev] New project: binhost
Hi, +1 - love the idea, def joining. However, I suspect the op was aiming at a publicly hosted binpkg server. Given the below it becomes one server/host, we care only about the build parameters, and I suspect profile then has less of an effect on the built packages. For publicly *available* infrastructure I do however not recommend that arbitrary people be able to upload. We can however from attempted download logs try to determine which combinations of USE flags are required (with hashes of stuff this becomes tricky as even only 16 USE flags are already 65536 potential hashes, and 20 clocks in at over 16 million, still perfectly reversable, but once we start getting to 30+ USE flags ...). So perhaps a way to feedback "Hey, I looked for this combo/hash" so we don't need to reverse hashes. Would definitely join such a project, and it would be greatly beneficial if we can improve the infra to have a form of distributed build ... ie, for private infrastucture, improve the mechanisms used to "check for binpkg availability, i not available, build it and submit back to binary host", obviously all such nodes would need to be considered "trusted". Kind Regards, Jaco On 2021/02/10 21:11, Rich Freeman wrote: > On Wed, Feb 10, 2021 at 12:57 PM Andreas K. Hüttel > wrote: >> * what portage features are still needed or need improvements (e.g. binpkg >> signing and verification) >> * how should hosting look like > Some ideas for portage enhancements: > > 1. Ability to fetch binary packages from some kind of repo. > 2. Ability to have multiple binary packages co-exist in a repo (local > or remote) with different build attributes (arch, USE, CFLAGS, > DEPENDS, whatever). > 3. Ability to pick the most appropriate binary packages to use based > on user preferences (with a mix of hard and soft preferences). > > One idea I've had around how #2-3 might be implemented is: > 1. Binary packages already contain data on how they were built (USE > flags, dependencies, etc). Place this in a file using a deterministic > sorting/etc order so that two builds with the same settings will have > the same results. > 2. Generate a hash of the file contents - this can go in the filename > so that the file can co-exist with other files, and be located > assuming you have a full matching set of metadata. > 3. Start dropping attributes from the file based on a list of > priorities and generate additional hashes. Create symlinked files to > the original file using these hashes (overwriting or not existing > symlinks based on policy). This allows the binary package to be found > using either an exact set of attributes or a subset of higher-priority > attributes. This is analogous to shared object symlinking. > 4. The package manager will look for a binary package first using the > user's full config, and then by dropping optional elements of the > config (so maybe it does the search without CFLAGs, then without USE > flags). Eventually it aborts based on user prefs (maybe the user only > wants an exact match, or is willing to accept alternate CFLAGs but not > USE flags, or maybe anything for the arch is selected. > 5. As always the final selected binary package still gets evaluated > like any other binary package to ensure it is usable. > > Such a system can identify whether a potentially usable file exists > using only filename, cutting down on fetching. In the interests of > avoiding useless fetches we would only carry step 3 reasonably far - > packages would have to match based on architecture and any dynamic > linking requirements. So we wouldn't generate hashes that didn't > include at least those minimums, and the package manager wouldn't > search for them. > > Obviously you could do more (if you have 5 combinations of use flags, > look for the set that matches most closely). That couldn't be done > using hashes alone in an efficient way. You could have a small > manifest file alongside the binary package that could be fetched > separately if the package manager wants to narrow things down and > fetch a few of those to narrow it down further. > > Or you could skip the hash searching and just fetch all the manifests > for a particular package/arch and just search all of those, but that > is more data to transfer just to do a query. A metadata cache of some > kind of might be another solution. Content hashes would probably > still be useful just to allow co-existence of alternate builds. >
Re: [gentoo-dev] New project: binhost
On Wed, 2021-02-10 at 19:51 +0100, Lars Wendler wrote: > On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote: > > > Hi all, > > > > I'm announcing a new project here - "binhost" > > > > "The Gentoo Binhost project aims to provide readily installable, > > precompiled packages for a subset of configurations, via central > > binary package hosting. Currently we are still in the conceptual > > planning stage. " > > > > https://wiki.gentoo.org/wiki/Project:Binhost > > > > If you're interested in helping out, feel free to add yourself on the > > wiki page. > > > > Note that I see actually *building* the packages not as the central > > point of the project (that could be e.g. a side effect of a > > tinderbox). I'm more concerned about > > * what configurations should we use > > * what portage features are still needed or need improvements (e.g. > > binpkg signing and verification) > > * how should hosting look like > > * and how we can test this on a limited scale before it goes "into > > production" > > * ... > > > > Comments, ideas, flamebaits? :D > > > > Cheers, > > Andreas > > > > It would be great to improve portage speed with handling binpkgs. I > already have my own binhost for a couple of Gentoo systems and even > though these systems don't have to compile anything themselves, > installing ~100 to ~200 binpkgs takes way more than an hour of > installation time. Arch Linux' pacman only takes a fraction of this > time for the very same task. > I know that I compare apples with pears here but even reducing the > current portage time by 50% would be a huge improvement. Is that really a problem? For me, Portage takes about an hour just to do the dependency processing these days. In fact, building from sources is now faster than dependency calculations. -- Best regards, Michał Górny
Re: [gentoo-dev] New project: binhost
ср, 10 февр. 2021 г. в 19:12, Rich Freeman : > > On Wed, Feb 10, 2021 at 12:57 PM Andreas K. Hüttel > wrote: > > > > * what portage features are still needed or need improvements (e.g. binpkg > > signing and verification) > > * how should hosting look like > > Some ideas for portage enhancements: > > 1. Ability to fetch binary packages from some kind of repo. > 2. Ability to have multiple binary packages co-exist in a repo (local > or remote) with different build attributes (arch, USE, CFLAGS, > DEPENDS, whatever). > 3. Ability to pick the most appropriate binary packages to use based > on user preferences (with a mix of hard and soft preferences). > > One idea I've had around how #2-3 might be implemented is: > 1. Binary packages already contain data on how they were built (USE > flags, dependencies, etc). Place this in a file using a deterministic > sorting/etc order so that two builds with the same settings will have > the same results. > 2. Generate a hash of the file contents - this can go in the filename > so that the file can co-exist with other files, and be located > assuming you have a full matching set of metadata. > 3. Start dropping attributes from the file based on a list of > priorities and generate additional hashes. Create symlinked files to > the original file using these hashes (overwriting or not existing > symlinks based on policy). This allows the binary package to be found > using either an exact set of attributes or a subset of higher-priority > attributes. This is analogous to shared object symlinking. > 4. The package manager will look for a binary package first using the > user's full config, and then by dropping optional elements of the > config (so maybe it does the search without CFLAGs, then without USE > flags). Eventually it aborts based on user prefs (maybe the user only > wants an exact match, or is willing to accept alternate CFLAGs but not > USE flags, or maybe anything for the arch is selected. A related idea: if user could specify which USE-flags are mandatory to be set, which USE flags are mandatory to be not set, and which can be either way, it's easier to find the matching binary package with less constraints, where only some flags match. > 5. As always the final selected binary package still gets evaluated > like any other binary package to ensure it is usable. > > Such a system can identify whether a potentially usable file exists > using only filename, cutting down on fetching. In the interests of > avoiding useless fetches we would only carry step 3 reasonably far - > packages would have to match based on architecture and any dynamic > linking requirements. So we wouldn't generate hashes that didn't > include at least those minimums, and the package manager wouldn't > search for them. > > Obviously you could do more (if you have 5 combinations of use flags, > look for the set that matches most closely). That couldn't be done > using hashes alone in an efficient way. You could have a small > manifest file alongside the binary package that could be fetched > separately if the package manager wants to narrow things down and > fetch a few of those to narrow it down further. > > Or you could skip the hash searching and just fetch all the manifests > for a particular package/arch and just search all of those, but that > is more data to transfer just to do a query. A metadata cache of some > kind of might be another solution. Content hashes would probably > still be useful just to allow co-existence of alternate builds. > > -- > Rich >
Re: [gentoo-dev] New project: binhost
On 2/10/21 2:11 PM, Rich Freeman wrote: On Wed, Feb 10, 2021 at 12:57 PM Andreas K. Hüttel wrote: * what portage features are still needed or need improvements (e.g. binpkg signing and verification) * how should hosting look like Some ideas for portage enhancements: 1. Ability to fetch binary packages from some kind of repo. 2. Ability to have multiple binary packages co-exist in a repo (local or remote) with different build attributes (arch, USE, CFLAGS, DEPENDS, whatever). 3. Ability to pick the most appropriate binary packages to use based on user preferences (with a mix of hard and soft preferences). One idea I've had around how #2-3 might be implemented is: 1. Binary packages already contain data on how they were built (USE flags, dependencies, etc). Place this in a file using a deterministic sorting/etc order so that two builds with the same settings will have the same results. this is provided by FEATURES="binpkg-multi-instance" (or maybe i misread) 2. Generate a hash of the file contents - this can go in the filename so that the file can co-exist with other files, and be located assuming you have a full matching set of metadata. 3. Start dropping attributes from the file based on a list of priorities and generate additional hashes. Create symlinked files to the original file using these hashes (overwriting or not existing symlinks based on policy). This allows the binary package to be found using either an exact set of attributes or a subset of higher-priority attributes. This is analogous to shared object symlinking. 4. The package manager will look for a binary package first using the user's full config, and then by dropping optional elements of the config (so maybe it does the search without CFLAGs, then without USE flags). Eventually it aborts based on user prefs (maybe the user only wants an exact match, or is willing to accept alternate CFLAGs but not USE flags, or maybe anything for the arch is selected. 5. As always the final selected binary package still gets evaluated like any other binary package to ensure it is usable. Such a system can identify whether a potentially usable file exists using only filename, cutting down on fetching. In the interests of avoiding useless fetches we would only carry step 3 reasonably far - packages would have to match based on architecture and any dynamic linking requirements. So we wouldn't generate hashes that didn't include at least those minimums, and the package manager wouldn't search for them. Obviously you could do more (if you have 5 combinations of use flags, look for the set that matches most closely). That couldn't be done using hashes alone in an efficient way. You could have a small manifest file alongside the binary package that could be fetched separately if the package manager wants to narrow things down and fetch a few of those to narrow it down further. Or you could skip the hash searching and just fetch all the manifests for a particular package/arch and just search all of those, but that is more data to transfer just to do a query. A metadata cache of some kind of might be another solution. Content hashes would probably still be useful just to allow co-existence of alternate builds.
Re: [gentoo-dev] New project: binhost
On 2/10/21 12:57 PM, Andreas K. Hüttel wrote: Hi all, I'm announcing a new project here - "binhost" "The Gentoo Binhost project aims to provide readily installable, precompiled packages for a subset of configurations, via central binary package hosting. Currently we are still in the conceptual planning stage. " https://wiki.gentoo.org/wiki/Project:Binhost If you're interested in helping out, feel free to add yourself on the wiki page. Note that I see actually *building* the packages not as the central point of the project (that could be e.g. a side effect of a tinderbox). I'm more concerned about * what configurations should we use maybe to start with, desktop profiles (gnome/qt/none + systemd/openrc) are a good idea, these are the people who would benefit most. This may not be enough, most people also enable extra flags polkit/elogind/ttf/otf/wayland/pulseaudio We can start with a default and gather feedback as to what new flags should be enabled. * what portage features are still needed or need improvements (e.g. binpkg signing and verification) somehow checking valid CFLAG/etc. compatibility, not sure how. Some packages fail when built with differing flags * how should hosting look like Something like the list of profiles in eselect profile (replace https by protocol of choice): https://binpkg.gentoo.org/amd64/17.1/desktop/ https://binpkg.gentoo.org/amd64/17.1/desktop/gnome/ could potentially also be mirrored on rsync mirrors. Cheers, Aisha * and how we can test this on a limited scale before it goes "into production" * ... Comments, ideas, flamebaits? :D Cheers, Andreas
Re: [gentoo-dev] New project: binhost
On 2/10/21 6:57 PM, Andreas K. Hüttel wrote: Comments, ideas, flamebaits? :D I'd appreciate bin packages from a user perspective too. With -j1 I do have compile times of about 1 day and more for few packages. But the topic "repoducible builds" is lurking around the corner! -- Toralf PGP 23217DA7 9B888F45 OpenPGP_signature Description: OpenPGP digital signature
Re: [gentoo-dev] New project: binhost
Le 2/10/21 à 6:57 PM, Andreas K. Hüttel a écrit : Hi all, I'm announcing a new project here - "binhost" "The Gentoo Binhost project aims to provide readily installable, precompiled packages for a subset of configurations, via central binary package hosting. Currently we are still in the conceptual planning stage. " https://wiki.gentoo.org/wiki/Project:Binhost If you're interested in helping out, feel free to add yourself on the wiki page. Note that I see actually *building* the packages not as the central point of the project (that could be e.g. a side effect of a tinderbox). I'm more concerned about * what configurations should we use * what portage features are still needed or need improvements (e.g. binpkg signing and verification) * how should hosting look like * and how we can test this on a limited scale before it goes "into production" * ... Comments, ideas, flamebaits? :D Cheers, Andreas Hi Andreas, I'm pretty interested to help for this topic. Notably, for my work on creating and maintaining Gentoo template for Qubes OS, I'm weekly constructing binpkgs mirrors in order to ease rebuilding from scratch Gentoo templates and also for CI purposes. So I would be glad to help the whole Gentoo community in such efforts. I'm also following a very interesting work here on portage: https://github.com/gentoo/portage/pull/562 Best regards, Frédéric OpenPGP_signature Description: OpenPGP digital signature
Re: [gentoo-dev] New project: binhost
> Some ideas for portage enhancements: > > 1. Ability to fetch binary packages from some kind of repo. > 2. Ability to have multiple binary packages co-exist in a repo (local > or remote) with different build attributes (arch, USE, CFLAGS, > DEPENDS, whatever). > 3. Ability to pick the most appropriate binary packages to use based > on user preferences (with a mix of hard and soft preferences). The more definite answer should come from Zac, but I think a good part of this is already implemented. :) -- Andreas K. Hüttel dilfri...@gentoo.org Gentoo Linux developer (council, qa, toolchain, base-system, perl, libreoffice) signature.asc Description: This is a digitally signed message part.
Re: [gentoo-dev] New project: binhost
On Wed, Feb 10, 2021 at 12:57 PM Andreas K. Hüttel wrote: > > * what portage features are still needed or need improvements (e.g. binpkg > signing and verification) > * how should hosting look like Some ideas for portage enhancements: 1. Ability to fetch binary packages from some kind of repo. 2. Ability to have multiple binary packages co-exist in a repo (local or remote) with different build attributes (arch, USE, CFLAGS, DEPENDS, whatever). 3. Ability to pick the most appropriate binary packages to use based on user preferences (with a mix of hard and soft preferences). One idea I've had around how #2-3 might be implemented is: 1. Binary packages already contain data on how they were built (USE flags, dependencies, etc). Place this in a file using a deterministic sorting/etc order so that two builds with the same settings will have the same results. 2. Generate a hash of the file contents - this can go in the filename so that the file can co-exist with other files, and be located assuming you have a full matching set of metadata. 3. Start dropping attributes from the file based on a list of priorities and generate additional hashes. Create symlinked files to the original file using these hashes (overwriting or not existing symlinks based on policy). This allows the binary package to be found using either an exact set of attributes or a subset of higher-priority attributes. This is analogous to shared object symlinking. 4. The package manager will look for a binary package first using the user's full config, and then by dropping optional elements of the config (so maybe it does the search without CFLAGs, then without USE flags). Eventually it aborts based on user prefs (maybe the user only wants an exact match, or is willing to accept alternate CFLAGs but not USE flags, or maybe anything for the arch is selected. 5. As always the final selected binary package still gets evaluated like any other binary package to ensure it is usable. Such a system can identify whether a potentially usable file exists using only filename, cutting down on fetching. In the interests of avoiding useless fetches we would only carry step 3 reasonably far - packages would have to match based on architecture and any dynamic linking requirements. So we wouldn't generate hashes that didn't include at least those minimums, and the package manager wouldn't search for them. Obviously you could do more (if you have 5 combinations of use flags, look for the set that matches most closely). That couldn't be done using hashes alone in an efficient way. You could have a small manifest file alongside the binary package that could be fetched separately if the package manager wants to narrow things down and fetch a few of those to narrow it down further. Or you could skip the hash searching and just fetch all the manifests for a particular package/arch and just search all of those, but that is more data to transfer just to do a query. A metadata cache of some kind of might be another solution. Content hashes would probably still be useful just to allow co-existence of alternate builds. -- Rich
Re: [gentoo-dev] New project: binhost
On Wed, 10 Feb 2021 19:57:48 +0200 Andreas K. Hüttel wrote: >Hi all, > >I'm announcing a new project here - "binhost" > >"The Gentoo Binhost project aims to provide readily installable, >precompiled packages for a subset of configurations, via central >binary package hosting. Currently we are still in the conceptual >planning stage. " > >https://wiki.gentoo.org/wiki/Project:Binhost > >If you're interested in helping out, feel free to add yourself on the >wiki page. > >Note that I see actually *building* the packages not as the central >point of the project (that could be e.g. a side effect of a >tinderbox). I'm more concerned about >* what configurations should we use >* what portage features are still needed or need improvements (e.g. >binpkg signing and verification) >* how should hosting look like >* and how we can test this on a limited scale before it goes "into >production" >* ... > >Comments, ideas, flamebaits? :D > >Cheers, >Andreas > It would be great to improve portage speed with handling binpkgs. I already have my own binhost for a couple of Gentoo systems and even though these systems don't have to compile anything themselves, installing ~100 to ~200 binpkgs takes way more than an hour of installation time. Arch Linux' pacman only takes a fraction of this time for the very same task. I know that I compare apples with pears here but even reducing the current portage time by 50% would be a huge improvement. Cheers -- Lars Wendler Gentoo package maintainer GPG: 21CC CF02 4586 0A07 ED93 9F68 498F E765 960E 9B39 pgpjVA9jqqLPQ.pgp Description: Digitale Signatur von OpenPGP
[gentoo-dev] New project: binhost
Hi all, I'm announcing a new project here - "binhost" "The Gentoo Binhost project aims to provide readily installable, precompiled packages for a subset of configurations, via central binary package hosting. Currently we are still in the conceptual planning stage. " https://wiki.gentoo.org/wiki/Project:Binhost If you're interested in helping out, feel free to add yourself on the wiki page. Note that I see actually *building* the packages not as the central point of the project (that could be e.g. a side effect of a tinderbox). I'm more concerned about * what configurations should we use * what portage features are still needed or need improvements (e.g. binpkg signing and verification) * how should hosting look like * and how we can test this on a limited scale before it goes "into production" * ... Comments, ideas, flamebaits? :D Cheers, Andreas -- Andreas K. Hüttel dilfri...@gentoo.org Gentoo Linux developer (council, qa, toolchain, base-system, perl, libreoffice) signature.asc Description: This is a digitally signed message part.