Re: [gentoo-dev] The problem of unmaintained packages in Gentoo
On Thu, 21 Dec 2017 12:25:11 -0600 William Hubbs wrote: > I think there's some confusion here. I'm not trying to change the bar > for ~arch, just trying to understand what that bar is supposed to be. The bar for ~arch is "end users should have reasonable expectations that it mostly works for most purposes, especially those that can be trivially checked for by its maintainer" ~arch will however be imperfect, and there will be issues from time to time. However, the goal is that those issues are not the "easy to find and obvious" kind, but the subtle and hard to stumble into. And I see that as why we have the time interval: Because it gives a good opportunity for actual real world usage patterns to discover the sorts of problems that you can't discover by actively looking for them, the rumsfeldian "unknown unknowns". Because realistically speaking, ~arch is the stable of tomorrow. The goal is for it to be stable today. And when it proves itself ready to be the stable of today, it becomes the stable of today. pgpLBmGG4qGK7.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] The problem of unmaintained packages in Gentoo
On Thu, Dec 21, 2017 at 03:03:21PM +, Roy Bamford wrote: > > So, I guess this means that the quality of the ~arch tree is supposed > > to > > be somewhat lower than the quality of the stable tree. > > > > William > > > > > > William, > > I've been running ~arch everywhere since May 2002 and had exactly > two major issues. They were :- > Xorg going modular ... which I was aware of before it happened and > expat which came as a surprise while I was dealing with modular > Xorg. > > There have been some minor inconviences along the way too but > problems running ~arch have reduced over the years. > > Nobody should run Gentoo at all in production unless they build > and test packages offline before pushing the binaries to production. > Then they can run whatever they want. > Every Gentoo install is different and very few possible > combinations are actually tested. > > By all means lower the bar for ~arch. Say, to "builds and works for > me, needs more testing". The down side is that it will create more > bug reports and more work, so it may only exchange one problem > for another. I think there's some confusion here. I'm not trying to change the bar for ~arch, just trying to understand what that bar is supposed to be. William signature.asc Description: Digital signature
Re: [gentoo-dev] The problem of unmaintained packages in Gentoo
On 2017.12.21 00:35, William Hubbs wrote: > On Wed, Dec 20, 2017 at 07:12:45PM -0500, Michael Orlitzky wrote: > > On 12/20/2017 06:58 PM, William Hubbs wrote: > > > > > > There already is an overlay for dying packages, it is called > graveyard, > > > but no one is putting things there. > > > > > > This email conflates old dying packages with new versions, which > are a > > > completely separate issue. > > > > > > > Lack of new versions *is* dying. But I can make a package not-dying > in a > > few seconds by merging a PR, so long as you don't expect me to do > the > > (relatively high) level of QA required for ~arch. > > > > > > > If a new version of a package is known to cause wide scale > breakage, it > > > goes in package.mask until the breakage is resolved. Otherwise, > putting > > > it in ~ is fine. I don't see the need for another level of > keywords. > > > > The quality of ~arch is its own worst enemy; these days, stable > packages > > are just old ~arch packages. Users and developers expect ~arch to > work, > > and we have no real policy or documentation stating that it won't. > Many > > people will tell you that ~arch works better than stable, because it > > gets fixed faster. > > ~arch *will* have breakages from time to time, sometimes major > breakages, until they are masked or fixed. We are not supposed to > leave > major breakages there, but ~arch is definitely not for the faint of > heart. If you are using ~arch, you are expected to be a power user at > leasst and be able to recover if your system breaks. Production > servers > should not be running ~arch at all. That's the whole reason ~arch > exists. > > Yes, ~arch gets fixed faster than stable, but that is to be expected. > However, it is definitely not a good reason to put your production > system on > full ~arch. > > So, I guess this means that the quality of the ~arch tree is supposed > to > be somewhat lower than the quality of the stable tree. > > William > > William, I've been running ~arch everywhere since May 2002 and had exactly two major issues. They were :- Xorg going modular ... which I was aware of before it happened and expat which came as a surprise while I was dealing with modular Xorg. There have been some minor inconviences along the way too but problems running ~arch have reduced over the years. Nobody should run Gentoo at all in production unless they build and test packages offline before pushing the binaries to production. Then they can run whatever they want. Every Gentoo install is different and very few possible combinations are actually tested. By all means lower the bar for ~arch. Say, to "builds and works for me, needs more testing". The down side is that it will create more bug reports and more work, so it may only exchange one problem for another. -- Regards, Roy Bamford (Neddyseagoon) a member of elections gentoo-ops forum-mods pgp1acaAGrvTc.pgp Description: PGP signature
Re: [gentoo-dev] The problem of unmaintained packages in Gentoo
On 12/21/17 15:11, Rich Freeman wrote: > Part of me wonders if issues with stable are causing issues with > ~arch. If stable is regarded as stale that is going to push people > into ~arch who really intend to have stable systems. That said you do > want testing systems to have a reasonably low bug count because it is > kind of hard to test the latest chromium beta when X11 isn't working. > Don't wonder anymore, this already happened, however it has to be noted that different part of the tree are different. Most famous server packages (apache, nginx, postfix, mysql posgrees) are in good shape, stable here is pretty usable and working well. When the target is a desktop system however things change and you are pushed to a partially ~arch system and soon thereafter to a full unstable install. This is only partially imputable to the (lack of) manpower, increased number of installed packages, infinitely bigger use flag permutation and functionality, in short complexity make very difficult to have a stable system that stay stable. Stable has to be stable in different point in times, and it's not enough to wait a month of user test, that's a good start, but it should be constantly monitored in a changing environment. This is very difficult when thare are 2000pkgs installed instead of 500.
Re: [gentoo-dev] The problem of unmaintained packages in Gentoo
On Thu, Dec 21, 2017 at 6:01 AM, Thomas Deutschmann wrote: > On 2017-12-21 01:35, William Hubbs wrote: >> ~arch *will* have breakages from time to time, sometimes major >> breakages, until they are masked or fixed. We are not supposed to leave >> major breakages there, but ~arch is definitely not for the faint of >> heart. If you are using ~arch, you are expected to be a power user at >> leasst and be able to recover if your system breaks. Production servers >> should not be running ~arch at all. That's the whole reason ~arch >> exists. > > If you add something to > the repository which is keyworded you should at least know that it > builds and works in the default USE flag configuration. FWIW, this is a higher standard of quality than what is being proposed right now for stable keywords (roughly). Granted, stable would benefit from being in ~arch for 30 days, and would also benefit from any testing it took to get the package into ~arch. However, the current proposal is that there will be no runtime testing at all of many packages with stable dependencies. Ultimately we can define the quality standards however we want and users can decide what quality level they wish to run. However, the obvious goal would be to choose reasonable standards that are useful to as many as possible. Do we want more packages just dumped into ~arch and let the users running ~arch report runtime failures? Or do we want packages in ~arch to be more stale? Part of me wonders if issues with stable are causing issues with ~arch. If stable is regarded as stale that is going to push people into ~arch who really intend to have stable systems. That said you do want testing systems to have a reasonably low bug count because it is kind of hard to test the latest chromium beta when X11 isn't working. -- Rich
Re: [gentoo-dev] The problem of unmaintained packages in Gentoo
On 2017-12-21 01:35, William Hubbs wrote: > ~arch *will* have breakages from time to time, sometimes major > breakages, until they are masked or fixed. We are not supposed to leave > major breakages there, but ~arch is definitely not for the faint of > heart. If you are using ~arch, you are expected to be a power user at > leasst and be able to recover if your system breaks. Production servers > should not be running ~arch at all. That's the whole reason ~arch > exists. I don't agree with this. Yes, ~arch can break sometimes so you should be familiar with masking. But on the other hand: ~arch isn't a minefield. If you add something to the repository which is keyworded you should at least know that it builds and works in the default USE flag configuration. If you don't know and there's a chance to break something badly, drop keywords or add it with a mask. Breaking ~arch because a dev was unable to test for some reason is IMHO unacceptable -- even for ~arch. That's the whole reason why keywords/p.mask exist ;-) -- Regards, Thomas Deutschmann / Gentoo Linux Developer C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5 signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] The problem of unmaintained packages in Gentoo
On Wed, Dec 20, 2017 at 07:12:45PM -0500, Michael Orlitzky wrote: > On 12/20/2017 06:58 PM, William Hubbs wrote: > > > > There already is an overlay for dying packages, it is called graveyard, > > but no one is putting things there. > > > > This email conflates old dying packages with new versions, which are a > > completely separate issue. > > > > Lack of new versions *is* dying. But I can make a package not-dying in a > few seconds by merging a PR, so long as you don't expect me to do the > (relatively high) level of QA required for ~arch. > > > > If a new version of a package is known to cause wide scale breakage, it > > goes in package.mask until the breakage is resolved. Otherwise, putting > > it in ~ is fine. I don't see the need for another level of keywords. > > The quality of ~arch is its own worst enemy; these days, stable packages > are just old ~arch packages. Users and developers expect ~arch to work, > and we have no real policy or documentation stating that it won't. Many > people will tell you that ~arch works better than stable, because it > gets fixed faster. ~arch *will* have breakages from time to time, sometimes major breakages, until they are masked or fixed. We are not supposed to leave major breakages there, but ~arch is definitely not for the faint of heart. If you are using ~arch, you are expected to be a power user at leasst and be able to recover if your system breaks. Production servers should not be running ~arch at all. That's the whole reason ~arch exists. Yes, ~arch gets fixed faster than stable, but that is to be expected. However, it is definitely not a good reason to put your production system on full ~arch. So, I guess this means that the quality of the ~arch tree is supposed to be somewhat lower than the quality of the stable tree. William signature.asc Description: Digital signature
Re: [gentoo-dev] The problem of unmaintained packages in Gentoo
On 12/20/2017 06:58 PM, William Hubbs wrote: > > There already is an overlay for dying packages, it is called graveyard, > but no one is putting things there. > > This email conflates old dying packages with new versions, which are a > completely separate issue. > Lack of new versions *is* dying. But I can make a package not-dying in a few seconds by merging a PR, so long as you don't expect me to do the (relatively high) level of QA required for ~arch. > If a new version of a package is known to cause wide scale breakage, it > goes in package.mask until the breakage is resolved. Otherwise, putting > it in ~ is fine. I don't see the need for another level of keywords. The quality of ~arch is its own worst enemy; these days, stable packages are just old ~arch packages. Users and developers expect ~arch to work, and we have no real policy or documentation stating that it won't. Many people will tell you that ~arch works better than stable, because it gets fixed faster. The new level of keyword would avoid screwing all of those ~arch users at once, by not suddenly murdering the quality of their tree. From the outset, the new level of keyword would have to have a description like "only use this if you are stupid" to fulfill its intended role.
Re: [gentoo-dev] The problem of unmaintained packages in Gentoo
On Wed, Dec 20, 2017 at 06:33:21PM -0500, Michael Orlitzky wrote: > On 12/20/2017 02:41 PM, Virgil Dupras wrote: > > > > Maybe some kind of official overlay for packages needing love? We > > could send outdated packages there to die or to be born again if the > > right person picks it up. > > > > The overlay could have more relaxed rules (not malicious and looking > > good? no need to test this, merge!) for PR merging. If the package > > degrades through bad PRs, fine, let's let it die. If it improves, > > good, it can be born again. > > > > ... > > > > (my apologies if this idea is not new, I haven't been following the > > ML for very long.) > > It's not a bad idea, but personally I'd like to see it as a third level > of stability in the tree for users to opt into with ACCEPT_KEYWORDS. There already is an overlay for dying packages, it is called graveyard, but no one is putting things there. This email conflates old dying packages with new versions, which are a completely separate issue. If a new version of a package is known to cause wide scale breakage, it goes in package.mask until the breakage is resolved. Otherwise, putting it in ~ is fine. I don't see the need for another level of keywords. William signature.asc Description: Digital signature
Re: [gentoo-dev] The problem of unmaintained packages in Gentoo
On 12/20/2017 02:41 PM, Virgil Dupras wrote: > > Maybe some kind of official overlay for packages needing love? We > could send outdated packages there to die or to be born again if the > right person picks it up. > > The overlay could have more relaxed rules (not malicious and looking > good? no need to test this, merge!) for PR merging. If the package > degrades through bad PRs, fine, let's let it die. If it improves, > good, it can be born again. > > ... > > (my apologies if this idea is not new, I haven't been following the > ML for very long.) It's not a bad idea, but personally I'd like to see it as a third level of stability in the tree for users to opt into with ACCEPT_KEYWORDS. Right now ~arch means, * I've built and tested this and, because of how our package manager works, * I think this is more reliable than the previous ~arch version Often that's a pretty high bar to meet. For example, we have a pull request open right now[0] that adds an alpha version of dev-php/xdebug-client to get php-7.2 support. It's probably fine. The ebuild contains some minor, expected changes. Whatever, it probably works. But, I don't use xdebug-client, and I don't think the other member of the PHP team does either. Building php-7.2 takes a while, and I'd have to do that and figure out how to do a basic "does this work" test on the package to make sure it doesn't die immediately, because otherwise the alpha version is going to ruin life for the people who are using php-7.1 with the existing ~arch version. But it's probably fine! It would be great if there was a third level of stability called "whatever, it probably works." Then I could just merge that PR and let the crazy people who have that in their ACCEPT_KEYWORDS tell me if it breaks. Instead, it has to wait until one of two people has the time to test/review it. [0] https://github.com/gentoo/gentoo/pull/6586
Re: [gentoo-dev] The problem of unmaintained packages in Gentoo
qlist -Iv $(portageq --repo gentoo --orphaned) On Wednesday, 20 December 2017 23:54:27 MSK Christopher Head wrote: > On December 20, 2017 8:49:03 AM PST, "Michał Górny" wrote: > >Ad. 1. We currently have over 1650 m-n packages [1] and the list keeps > >growing. The advantage of this type is that we have an explicit list > >and everyone clearly sees that the packages need a new maintainer. We > >also have some dedicated people who try to help fixing worst issues > >but they're obviously not capable of doing all the work themselves. > > Is there an easy way to check whether any packages I have installed on my > system are m-n? I checked “man q” and “man equery” but neither seemed to > support searching by maintainer. The closest I found was “equery meta”, but > that requires a specific package name on the command line. > > I read the last rites and it’s quite easy to notice if a package I actively > use is shown there, but that doesn’t help with the hundreds of libraries and > other dependencies whose names I don’t even really recognize but I have > installed. > -- Best regards. Ilya Tumaykin.
Re: [gentoo-dev] The problem of unmaintained packages in Gentoo
On December 20, 2017 8:49:03 AM PST, "Michał Górny" wrote: >Ad. 1. We currently have over 1650 m-n packages [1] and the list keeps >growing. The advantage of this type is that we have an explicit list >and everyone clearly sees that the packages need a new maintainer. We >also have some dedicated people who try to help fixing worst issues >but they're obviously not capable of doing all the work themselves. Is there an easy way to check whether any packages I have installed on my system are m-n? I checked “man q” and “man equery” but neither seemed to support searching by maintainer. The closest I found was “equery meta”, but that requires a specific package name on the command line. I read the last rites and it’s quite easy to notice if a package I actively use is shown there, but that doesn’t help with the hundreds of libraries and other dependencies whose names I don’t even really recognize but I have installed. -- Christopher Head signature.asc Description: PGP signature
Re: [gentoo-dev] The problem of unmaintained packages in Gentoo
On Wed, 20 Dec 2017 17:49:03 +0100 Michał Górny wrote: > So does anyone have any ideas on what we could realistically do right > now to improve things? Maybe some kind of official overlay for packages needing love? We could send outdated packages there to die or to be born again if the right person picks it up. The overlay could have more relaxed rules (not malicious and looking good? no need to test this, merge!) for PR merging. If the package degrades through bad PRs, fine, let's let it die. If it improves, good, it can be born again. I'm under the impression that such an overlay could release pressure on proxy-maint, allow treecleaners to clean more aggressively while keeping users happy. As a bonus, it could be an interesting path to becoming a gentoo developer. More relaxed rules could mean that anyone could assume maintainership of a dying package without having to wait for someone from proxy-maint to review every little change. This allows the would-be developer to be bolder with changes and prove herself better. (my apologies if this idea is not new, I haven't been following the ML for very long.) Regards, Virgil Dupras
Re: [gentoo-dev] The problem of unmaintained packages in Gentoo
Hi, December 20, 2017 5:46 PM, "Michał Górny" wrote: > E. Some of the unmaintained packages are dependencies of other > maintained packages in Gentoo. However, developers usually don't want > to take them, even if their package is the only revdep. > > F. We are usually treecleaning packages as they become severely outdated > and broken. However, that takes serious amount of work too and usually > results in a lot of hostility from other developers (who don't want to > maintain the package in question) and users. > > G. In the past, I've attempted to evaluate the status of projects and to > clean some dead up. However, it's a lot of manual labor and it meets > with hostility from some of the Gentoo developers. > > H. For most things related to determining developer inactivity, we have > little to no automation. It's easy to tell when a developer stops > committing altogether but we have no special help in e.g. determining > that some packages are effectively unmaintained (and that would of > course meet with hostility). I believe there was some work in progress about automating check for new upstream version (via repology api I think). Couldn't this be used to check for maintainer abandon? Some bugs can't always be solved easily, so you can't take that into account to check for inactivity, but not adding new versions is, IMO, a sign that the maintainer doesn’t check often enough for updates. We could also send automatic mail a month (arbitrary choice) after a new version has been released upstream to the maintainer for the related ebuild with such a system, so that maintainers don't have to bother about that part anymore. I can't remember what was called the project but what's its current status? I don't know if a solution like that would change much to the situation, but I believe it should give us better insights about the state of the tree. Best regards, -- Corentin “Nado” Pazdera
[gentoo-dev] The problem of unmaintained packages in Gentoo
Hello, everyone.
Jalus Bilieyich has submitted the following for the last Council
meeting:
| Discuss the lack of enough package maintainers to update ebuilds. Many
| ebuilds in the Portage tree can be easily marked outdated.
Given that the item didn't see any real discussion in the mailing lists,
and that it is a non-trivial problem, the Council has decided to bounce
it back to the mailing lists in a separate discussion thread.
The problem
===
The problem could be summarized shortly: the ebuilds need much more love
than they are given right now. This results in some packages being
outdated, half-broken and/or using obsolete constructs. In the end, some
of the packages start effectively blocking other developers from doing
their job (e.g. they can't solve one problem without fixing some other
pile of existing problems first).
I think we can list a few kinds of packages that need help in Gentoo:
1. Packages explicitly listed as not having a maintainer
(maintainer-needed) [1].
2. Packages whose maintainer is MIA (including 'dead' projects).
3. Packages whose maintainer is not interested in maintaining them
(or in some cases even unaware that he is the maintainer).
4. Packages whose maintainer is not capable of maintaining them
properly (or unwilling to, in some cases).
Now, for some details:
Ad. 1. We currently have over 1650 m-n packages [1] and the list keeps
growing. The advantage of this type is that we have an explicit list
and everyone clearly sees that the packages need a new maintainer. We
also have some dedicated people who try to help fixing worst issues
but they're obviously not capable of doing all the work themselves.
Ad. 2. This is somewhat problematic, developers usually notice
inactivity after some time and sometimes help out but things are rather
suboptimal right now. It can take 3-6 months from developer's
disappearance before his packages are announced 'up for grabs'
and someone actually interested can take them.
Things are even harder when the packages are maintained by projects
whose developers are no longer active.
Ad. 3. Sometimes developers lose interest or otherwise forget that they
maintain some package. This may result in some degradation but the
developer usually notices that when a new bug is reported and abandons
the package. This is the easier part.
The harder part is when packages are maintained by projects who aren't
really interested in them. This is especially a problem in herd-style
projects that collect large number of packages by a specific topic but
the individual project members are really interested in only a subset of
them.
Ad. 4. This is the hardest part. We have some packages which are
maintained but whose maintainers can't really keep up with work. In this
case it's usually hard to determine that the maintainer needs help with
a specific package, and/or whether he wishes to accept it.
What's even worse, there were historical cases when maintainers
'claimed' packages but were rather interested in prevented others
from the maintenance work ('breaking them') or removing them.
At the same time weren't really interested in fixing bugs or updating
the packages.
Solutions?
==
So does anyone have any ideas on what we could realistically do right
now to improve things? Few notes from what I've seen:
A. Recruiters process new recruits quite smoothly but we don't have many
candidates interested in package management. Even then, I don't see
every new recruit taking 50+ packages...
B. Proxy-maint constantly lacks manpower to process contributions.
However, once again we can't expect a lot of packages per maintainer,
and we need to account for doubled manpower cost.
C. We have a lot of overlays. Some of their maintainers are quite
capable. Can we do something about that?
D. All above considered, we still haven't dealt with the copyright
issues. The work on last draft was stalled one year ago [2].
E. Some of the unmaintained packages are dependencies of other
maintained packages in Gentoo. However, developers usually don't want
to take them, even if their package is the only revdep.
F. We are usually treecleaning packages as they become severely outdated
and broken. However, that takes serious amount of work too and usually
results in a lot of hostility from other developers (who don't want to
maintain the package in question) and users.
G. In the past, I've attempted to evaluate the status of projects and to
clean some dead up. However, it's a lot of manual labor and it meets
with hostility from some of the Gentoo developers.
H. For most things related to determining developer inactivity, we have
little to no automation. It's easy to tell when a developer stops
committing altogether but we have no special help in e.g. determining
that some packages are effectively unmaintained (and that would of
course meet with hostility).
[1]:https://qa-reports.gentoo.org/output/maintainer-needed.html
[2]:https://wiki.gentoo.org/wiki/U
