[gentoo-portage-dev] Announcing New wiki Project pages
I have mostly migrated the Portage project page to the wiki. Please look it over, edit any errors you see (only gentoo devs have edit capabilities). I have started a couple sub-pages: Ongoing-TODO and Proposals links: https://wiki.gentoo.org/wiki/Project:Portage https://wiki.gentoo.org/wiki/Project:Portage/Ongoing-TODO https://wiki.gentoo.org/wiki/Project:Portage/Proposals -- Brian Dolbec dol...@gentoo.org signature.asc Description: This is a digitally signed message part
[gentoo-portage-dev] New proposed modular sync system
First off, I know many of you think portage needs to do everything on it's own. NO outside dependencies. BUT! Please hear me out. While working on many of gentoo's tools gentoolkit's equery, eclean, enalyze, Layman, mirrorselect, etc.. I have found that there was a lot of needless code duplication. Both within the apps themselves and copying/re-inventing the nearly identical code in other apps. I want to move many of gentoo's core applications to using some more shared libraries. But that in itself is not a topic to discuss in this thread. Please, do not sidetrack this thread. The above is only for background info leading to this proposed change. This proposal follows along that thinking. Many years ago after re-writing ecleans code I began re-writing portage's emaint code so that it could be imported and used internally in eclean to re-generate the Packages binpkg index file. In that code I found a todo which said make a plug-in system so we don't have to keep modifying this code or something along those lines. So I did. It took 4 years+, rebasing many times and migrating vcs's, but Zac finally merged it. It has been used in portage for more than a year. That plug-in system I designed to be very flexible and basic. It can be used for many areas of portage to simplify it's code and the maintenance required to keep current with changes in repository types, and transfer methods. I have started a Proposals sub-page under the Portage project page in the wiki. It has a link to a diagram I made showing how the plug-in system is laid out. This thread will be used to discuss the proposal and the details needed for the module_spec dictionary that is used to provide the manager with the details of what a module provides, options, etc.. For a working example of the system in action, just try out emaint -h, and some other actions it provides. Then move one of it's modules out of the the emaint/modules/ directory and re-try emaint -h. You will see the options for that module and it's actions are no longer available. No code editing was required, no errors occurred,... Put it back, retry... The modular sync system I propose would re-use that plug-in manager to interface the sync manager code with the modules that perform the sync actions. This sytem is easily extended with new modules for different transfer methods or VCS specific modules. This also makes it possible for other gentoo applications like layman to install a layman specific module which syncs all layman overlays that it installs. All this done with out effort on the part of the portage team either for maintenance or code changes to support it's addition. The same can be said for Funtoo, if our git module is not suiotable, they just install their own custom sync module. Done :) These add-on sync modules can also be installed via their own ebuild. For example: Zac has for years had a custom script that syncs the portage tree and makes a squashfs version of it which he uses for his PORTDIR. it is faster than a normal one. He, or someone else could make a app-portage/squashfs-sync ebuild that installs a module for that purpose. The user then just needs to edit his repos.conf file and change the sync-type to squashfs. Then emerge --sync will do it automatically without user intervention and auxiliary scripts run manually or cron job. This system would also make it possible to modify emerge-s cli to accept target repos to sync. and not any others also installed. Similarly to layman -s some-overlay, emerge --sync some-overlay P.S. sorry for such a long email -- Brian Dolbec dol...@gentoo.org signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] New proposed modular sync system
Am 09.01.2014 18:33, schrieb Brian Dolbec: introduction history of the plugin system I fully agree with idea behind the plugin system. That is, to keep things that are separate, separate. It probably wouldn't have occurred to me to implement it that way (i.e. with auto-detection) but that's just fine. I don't see a problem with reusing that system here. I have started a Proposals sub-page under the Portage project page in the wiki. It has a link to a diagram I made showing how the plug-in system is laid out. This thread will be used to discuss the proposal and the details needed for the module_spec dictionary that is used to provide the manager with the details of what a module provides, options, etc.. The layout makes sense. Except the problems I see with where the modules are installed (see later). Not sure about module_spec yet. [...] stuff about ebuilds installing plugins While I see a valid use case here (especially with your squashfs example), I wonder how that is supposed to work. 1) Where should the ebuild install the plugin? 2) How does portage find those plugins. 3) How does portage's behavior to run with the currently selected python version, mix with the fact that ebuilds usually install only for some set of python versions? (especially python 2 vs 3). This system would also make it possible to modify emerge-s cli to accept target repos to sync. and not any others also installed. Similarly to layman -s some-overlay, emerge --sync some-overlay emerge --sync some-overlay already works. P.S. sorry for such a long email
Re: [gentoo-portage-dev] New proposed modular sync system
On Thu, 2014-01-09 at 09:33 -0800, Brian Dolbec wrote: I have started a Proposals sub-page under the Portage project page in the wiki. It has a link to a diagram I made showing how the plug-in system is laid out. This thread will be used to discuss the proposal and the details needed for the module_spec dictionary that is used to provide the manager with the details of what a module provides, options, etc.. Forgot to add the link: https://wiki.gentoo.org/wiki/Project:Portage/Proposals It has the link to the diagram I created on that page signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] New proposed modular sync system
On Thu, Jan 9, 2014 at 9:33 AM, Brian Dolbec dol...@gentoo.org wrote: First off, I know many of you think portage needs to do everything on it's own. NO outside dependencies. BUT! Please hear me out. While working on many of gentoo's tools gentoolkit's equery, eclean, enalyze, Layman, mirrorselect, etc.. I have found that there was a lot of needless code duplication. Both within the apps themselves and copying/re-inventing the nearly identical code in other apps. I want to move many of gentoo's core applications to using some more shared libraries. But that in itself is not a topic to discuss in this thread. Please, do not sidetrack this thread. The above is only for background info leading to this proposed change. This proposal follows along that thinking. Many years ago after re-writing ecleans code I began re-writing portage's emaint code so that it could be imported and used internally in eclean to re-generate the Packages binpkg index file. In that code I found a todo which said make a plug-in system so we don't have to keep modifying this code or something along those lines. So I did. It took 4 years+, rebasing many times and migrating vcs's, but Zac finally merged it. It has been used in portage for more than a year. That plug-in system I designed to be very flexible and basic. It can be used for many areas of portage to simplify it's code and the maintenance required to keep current with changes in repository types, and transfer methods. I have started a Proposals sub-page under the Portage project page in the wiki. It has a link to a diagram I made showing how the plug-in system is laid out. This thread will be used to discuss the proposal and the details needed for the module_spec dictionary that is used to provide the manager with the details of what a module provides, options, etc.. For a working example of the system in action, just try out emaint -h, and some other actions it provides. Then move one of it's modules out of the the emaint/modules/ directory and re-try emaint -h. You will see the options for that module and it's actions are no longer available. No code editing was required, no errors occurred,... Put it back, retry... The modular sync system I propose would re-use that plug-in manager to interface the sync manager code with the modules that perform the sync actions. This sytem is easily extended with new modules for different transfer methods or VCS specific modules. This also makes it possible for other gentoo applications like layman to install a layman specific module which syncs all layman overlays that it installs. All this done with out effort on the part of the portage team either for maintenance or code changes to support it's addition. The same can be said for Funtoo, if our git module is not suiotable, they just install their own custom sync module. Done :) These add-on sync modules can also be installed via their own ebuild. For example: Zac has for years had a custom script that syncs the portage tree and makes a squashfs version of it which he uses for his PORTDIR. it is faster than a normal one. He, or someone else could make a app-portage/squashfs-sync ebuild that installs a module for that purpose. The user then just needs to edit his repos.conf file and change the sync-type to squashfs. Then emerge --sync will do it automatically without user intervention and auxiliary scripts run manually or cron job. This system would also make it possible to modify emerge-s cli to accept target repos to sync. and not any others also installed. Similarly to layman -s some-overlay, emerge --sync some-overlay I think the opposition to this idea primarily falls on reliability. There have been a number of hacks to portage over the years to keep it functioning during upgrades of itself, even down to the non-standard place where its .py files are stored (/usr/lib/portage/pym?) So the real question is when we move to a plug-in architecture, how do we add, remove, upgrade the plugins without breaking the actual package manager? -A P.S. sorry for such a long email -- Brian Dolbec dol...@gentoo.org
Re: [gentoo-portage-dev] New proposed modular sync system
On Thu, 2014-01-09 at 19:46 +0100, Sebastian Luther wrote: The layout makes sense. Except the problems I see with where the modules are installed (see later). Not sure about module_spec yet. [...] The module_spec is a means to make available to the operational manager what the module supplies and any options it provides. The second thing it does is not import the files and modules it supplies. This reduces the memory requirements by not loading modules, files that are not needed. It also speeds up run time as a result. stuff about ebuilds installing plugins While I see a valid use case here (especially with your squashfs example), I wonder how that is supposed to work. 1) Where should the ebuild install the plugin? 2) How does portage find those plugins. That is to be determined. It doesn't matter where to the plugin manager. It needs a path to the plug-in directory on initialization. It does not need to be in a subdirectory where it is located. 3) How does portage's behavior to run with the currently selected python version, mix with the fact that ebuilds usually install only for some set of python versions? (especially python 2 vs 3). The plug-in must be able to work with all supported python versions in one form or another. In esearch, it supports python 3, but layman does not (yet), so when you use the -l switch to sync layman overlays as well as the main tree. It first tries to import the layman modules, upon failure it falls back to calling layman using a subprocess call. Just like portage currently calls rsync and git in a subprocess now. see working code: https://github.com/fuzzyray/esearch/blob/master/esearch/sync.py layman_sync() line # 149 If a proposed module is to be added to the tree, it should be capable of working in all configurations a user may have withing portage's capability. If not, if it isn't shipped by and with portage, then it is a user issue. Not ours. In such a case we (the portage-tools team) would insist on ewarns in the ebuild stating such constraints, etc.. This system would also make it possible to modify emerge-s cli to accept target repos to sync. and not any others also installed. Similarly to layman -s some-overlay, emerge --sync some-overlay emerge --sync some-overlay already works. P.S. sorry for such a long email signature.asc Description: This is a digitally signed message part
Re: [gentoo-portage-dev] New proposed modular sync system
On Thu, 2014-01-09 at 11:15 -0800, Alec Warner wrote: I think the opposition to this idea primarily falls on reliability. There have been a number of hacks to portage over the years to keep it functioning during upgrades of itself, even down to the non-standard place where its .py files are stored (/usr/lib/portage/pym?) So the real question is when we move to a plug-in architecture, how do we add, remove, upgrade the plugins without breaking the actual package manager? -A Well, a good point. But, this is the sync module, not some of the critical pkg installation code. Another thing, once a module is loaded, the source file could be removed and/or replaced without adverse effects depending what the code does. That is not always the case though. Portage/emerge does not sync and install ebuilds simultaneously, so that problem is not a concern. Also a lock file could be used to prevent another emerge process form interfering with an ongoing sync run. Just like it does now for merging completed builds. -- Brian Dolbec dol...@gentoo.org signature.asc Description: This is a digitally signed message part
